JP7800525B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Interacting with robotic devices using natural language is a promising means for non-experts to enable robotic devices to solve complex and diverse tasks. In recent years, research has been attracting attention on using large language models (LLMs) to output sequences of robotic device actions in a workspace from linguistic instructions.

For example, Non-Patent Document 1 proposes a system that generates an action sequence for a robot device from linguistic instructions. The system proposed in Non-Patent Document 1 accepts input of instructions in natural language (linguistic instructions). The system uses an LLM (Say module) to infer from the linguistic instructions the next action of the robot device that is likely to be executed. The system also uses a value function (Can module) to infer from observation data which actions are likely to be executed. The system then integrates the results of the two inferences and, depending on the integration result, determines the action sequence to be given to the robot device (an action sequence that is likely to be executed and is feasible).

Michael Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, [online], [Retrieved October 24, 2020], Internet <URL: https://arxiv.org/abs/2204.01691>

The inventors of this invention have discovered that the above-mentioned conventional systems have the following problems. In other words, the action sequences generated by conventional systems are specific to the trial environment and are not necessarily interpretable by humans. Because the action sequences are output as is, there is a problem in that the interpretability of the output obtained in terms of controlling the robot device is low.

In one aspect, the present invention was made in consideration of these circumstances, and its purpose is to provide technology that can obtain highly interpretable output regarding the control of a robot device.

The present invention employs the following features to solve the above-mentioned problems. Note that the following features of the invention can be combined as appropriate.

That is, an information processing device according to one aspect of the present invention includes a control unit configured to execute the steps of acquiring observation data of an environment in which a robot device operates and instruction information regarding a task goal to be given to the robot device, generating a problem description of the task from the acquired observation data and instruction information using an inference module, and outputting the generated problem description. The problem description includes descriptions of the initial state and goal state of objects present in the environment.

The problem description, together with the domain description, is used by the planner to generate an action plan (i.e., obtain an action sequence). This problem description describes the initial and goal states of objects in the environment to accomplish a task, and is highly descriptive (human-interpretable). Therefore, this configuration can obtain a highly descriptive output for obtaining an action sequence (command) to be given to the robotic device.

In the information processing device according to the above aspect, the generated problem description may conform to a predetermined format. With this configuration, the generated problem description conforms to the predetermined format, making it easier for the planner to generate an action plan from the problem description.

In the information processing device according to the above aspect, the generated problem description may further include a description of the objects present in the environment. According to this configuration, the object description makes it easier to identify objects present in the environment, making it easier for the planner to generate an action plan from the problem description.

In the information processing device according to the above aspect, the observation data may be composed of sensing data from a sensor, and the instruction information may be composed of linguistic information that specifies the goal in natural language. With this configuration, when the environment is observed by a sensor and task goals are given in linguistic instructions, highly descriptive output can be obtained regarding the control of the robot device.

In the information processing device according to the above aspect, the inference module may be configured to include a trained model for in-context learning. When the planner outputs an error message during the process of providing the generated problem description to a planner and generating a behavior plan for the robotic device, the control unit may further be configured to: acquire the output error message; and use the inference module to generate a new problem description from the problem description and the error message. With this configuration, if an appropriate problem description cannot be obtained, the problem description can be automatically corrected. The trained model is a trained machine learning model.

The control unit of the information processing device according to the above aspect may be configured to further acquire environmental information related to the environment in the acquiring step. Generating the problem description from the acquired observation data and instruction information may be configured by generating the problem description from the acquired observation data, instruction information, and environmental information. With this configuration, by further using environmental information as an explanatory variable in addition to the observation data and instruction information, it becomes possible to more accurately identify the environment in which a task is to be performed, which is expected to improve the accuracy of generating the problem description.

In the information processing device according to the above aspect, the generated problem description may further include a description of the object present in the environment. The inference module may include an object estimator. Generating the problem description using the inference module may include using the object estimator to generate a description of the object present in the environment from the acquired observation data. This configuration makes it possible to appropriately generate the object description portion of the problem description.

In the information processing device according to the above aspect, the object estimator may include a trained model for in-context learning. With this configuration, the object estimator includes a trained model for in-context learning, enabling general-purpose estimation of objects present in the environment, which is expected to enable the generation of problem descriptions for a variety of tasks.

The control unit of the information processing device according to the above aspect may be configured to further acquire attribute information of the objects present in the environment in the acquiring step. Generating a description of the object from the acquired observation data may be configured by generating a description of the object from the acquired observation data and the attribute information. With this configuration, attribute information makes it possible to more accurately identify objects present in the environment, which can be expected to improve the accuracy of generating the description portion of the object in the problem description.

In the information processing device according to the above aspect, the inference module may include an initial state estimator. Generating the problem description using the inference module may include generating a description of the initial state of the object present in the environment using the initial state estimator. This configuration makes it possible to appropriately generate the description portion of the initial state in the problem description.

In the information processing device according to the above aspect, the initial state estimator may include a trained model for in-context learning. With this configuration, the initial state estimator includes a trained model for in-context learning, enabling general-purpose inference of the initial states of objects present in the environment, which is expected to enable the generation of problem descriptions for a variety of tasks.

In the information processing device according to the above aspect, the inference module may include a goal estimator. Generating the problem description using the inference module may include generating a description of the goal state of the object present in the environment using the goal estimator. This configuration makes it possible to appropriately generate the description portion of the goal state in the problem description.

In the information processing device according to the above aspect, the goal estimator may include a trained model for in-context learning. With this configuration, the goal estimator includes a trained model for in-context learning, enabling general inference of the goal state of an object, which is expected to enable the generation of problem descriptions for a variety of tasks.

Note that the present invention is not limited to the information processing device described above. As another form of the information processing device according to each of the above aspects, one aspect of the present invention may be an information processing method that realizes all or part of each of the above configurations, or may be a program, or may be a storage medium that stores such a program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as a program through electrical, magnetic, optical, mechanical, or chemical action.

For example, an information processing method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of acquiring observation data of an environment in which a robotic device operates and instruction information regarding a task goal to be given to the robotic device, using an inference module to generate a problem description for the task from the acquired observation data and instruction information, and outputting the generated problem description.

Also, for example, a program according to one aspect of the present invention may be a program for causing a computer to execute the steps of acquiring observation data of the environment in which a robotic device operates and instruction information regarding the goal of a task to be given to the robotic device, using an inference module to generate a problem description for the task from the acquired observation data and instruction information, and outputting the generated problem description.

This invention makes it possible to obtain highly descriptive output regarding the control of a robotic device.

FIG. 1 is a diagram showing an example of a situation in which the present invention is applied. FIG. 2 is a diagram illustrating an example of input and output of an inference module according to an embodiment. FIG. 3 is a diagram illustrating an example of a process for modifying a problem description according to an embodiment. FIG. 4 is a diagram illustrating an example of the configuration of an inference module according to an embodiment. FIG. 5A illustrates a schematic diagram of an example object estimator according to an embodiment. FIG. 5B illustrates an example of an initial state estimator according to an embodiment. FIG. 5C illustrates a schematic diagram of an example target estimator according to an embodiment. FIG. 6 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. FIG. 7 is a diagram illustrating an example of a software configuration of the information processing device according to the embodiment. FIG. 8 is a flowchart illustrating an example of a processing procedure of the information processing device according to the embodiment. FIG. 9 is a diagram illustrating an example of the configuration of an inference module according to another embodiment. FIG. 10 shows each description included in the domain description for each prepared domain. FIG. 11A shows an example of observation data and instruction information given in the domain (Cooking). FIG. 11B shows an example of observation data and instruction information given in the domain (Blocksworld). FIG. 11C shows an example of observation data and instruction information given in the domain (Hanoi). FIG. 12 shows the results of the first experimental example. FIG. 13 shows the results of the second experimental example. FIG. 14 shows the results of the third experimental example.

An embodiment of one aspect of the present invention (hereinafter also referred to as "this embodiment") will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present invention in all respects. Various improvements and modifications may be made without departing from the scope of the present invention. In implementing the present invention, specific configurations according to the embodiment may be adopted as appropriate. Note that while the data appearing in this embodiment is described in natural language, more specifically, it is specified using pseudo-language, commands, parameters, machine language, etc. that can be understood by a computer.

§1 Application Example FIG. 1 schematically illustrates an example of a scenario in which the present invention is applied. An information processing device 1 according to this embodiment is one or more computers configured to generate a problem description 5 used by a planner to generate an action plan. Specifically, the information processing device 1 acquires observation data 20 of the environment in which the robot device R operates and instruction information 21 regarding the goal of a task to be given to the robot device R. The information processing device 1 uses an inference module 3 to generate a problem description 5 for the task from the acquired observation data 20 and instruction information 21. The problem description 5 includes descriptions (51, 52) of initial states and goal states of one or more objects present in the environment. The information processing device 1 outputs the generated problem description 5.

The problem description 5 includes descriptions (51, 52) of the initial state and the goal state of each of one or more objects in the environment, so that the planner generates an action plan that reaches the goal state from the initial state of the task. The initial state is the state before the task is performed. The initial state may also be referred to as the starting state. The goal state is the state after the task is performed and the objective is achieved. The description 51 of the initial state describes the initial state of each of one or more objects in the environment. On the other hand, the description 52 of the goal state describes the goal state of each of the one or more objects that is reached by completing the task. In other words, each description (51, 52) indicates the respective states before and after the task is performed and is interpretable by humans. In one example, each description (51, 52) describes the state of an object rather than the behavior of the robot device R, making it easy for humans to interpret even without a special understanding of the robot device R (e.g., without deciphering the program code). Therefore, the problem description 5 has high interpretability. Therefore, according to this embodiment, a highly interpretable output can be obtained to obtain an action sequence (control command) to be given to the robot device R.

[Input/output data]
In this embodiment, the observation data 20 and the instruction information 21 are given to the inference module 3 as input data, and as a result of the calculation process of the inference module 3, the problem description 5 is obtained as output data.

The observation data 20 may be any data representing the initial state of an object before a task is performed. The type of observation data 20 is not particularly limited and may be selected appropriately depending on the embodiment. The observation data 20 may be composed of data from one or more modalities. The environment may be observed using any method. The environment may include all events related to the situation in which the robot device R performs a task. The environment may be composed of at least a real environment and a virtual environment. For example, the environment may be composed of both a real environment and a virtual environment, such as AR (Augmented Reality) or MR (Mixed Reality). The environment may include VR (Virtual Reality), etc. The number of objects present in the environment may be arbitrary. The observation data 20 may be acquired at any time before the start of performance of a task. When multiple tasks are given, the initial state may be the state before the multiple tasks are performed, or the state before one of the multiple tasks is performed (including an intermediate state between the performance of the multiple tasks). That is, "initial" does not necessarily mean "beginning." A task may be any work (job) for transitioning from an initial state to a goal state. The state before the task is performed may be the initial state, and the state after the task is properly performed may be the goal state. The interval between tasks may be determined appropriately depending on the embodiment. Because the initial state of the object appears in the observation data 20, the inference module 3 can infer the description 51 of the initial state in the problem description 5 from the observation data 20.

The instruction information 21 may be any information that can identify the goal of the task. The type of instruction information 21 is not particularly limited and may be selected appropriately depending on the embodiment. The instruction information 21 may be composed of data of one or more modalities. The goal may be given as appropriate. The instruction information 21 may be acquired at any timing before the start of performance of the task. The instruction information 21 may be acquired before or after the observation data 20. The instruction information 21 may also be acquired at least partially in parallel with the observation data 20. Because the goal of the task is expressed in the instruction information 21, the inference module 3 can infer the description 52 of the goal state in the problem description 5 from the instruction information 21.

Problem description 5 includes descriptions (51, 52) of the initial state and the goal state, which enables the planner to generate an action plan for robot device R, including the feasibility (whether or not the robot device R is feasible) of the robot device R, given the skills of the robot device R. As long as it is of this nature, the configuration of problem description 5 is not particularly limited and may be selected appropriately depending on the embodiment. Because problem description 5 is robot device independent, it may be used to generate an action plan for any robot device. The planner may be configured appropriately to generate an action plan from problem description 5.

1, in one example, the skills of the robot device R may be given by the domain description 23. Also, in one example, the planner may include a symbolic planner 61 and a motion planner 65. The symbolic planner 61 may be configured to generate an action sequence 63, which is a sequence of abstract actions, from the problem description 5 and the domain description 23. The abstract action may be defined as any collection of actions that includes one or more actions of the robot device R and can be expressed as desired (e.g., in words, etc.). The abstract action may be defined as a collection of meaningful (i.e., human-understandable) actions, such as grasping, carrying, and positioning an object. The symbolic planner 61 may be a known symbolic planner such as Fast Downward in Reference 1 ("The Fast Downward Planning System", [online], [searched October 24, 2023], Internet <URL: https://planning.wiki/ref/planners/fd>) or the LAMA planner in Reference 2 (Silvia Richter et al., "The LAMA planner: guiding cost-based anytime planning with landmarks", [online], [searched October 24, 2023], Internet <URL: https://dl.acm.org/doi/10.5555/1946417.1946420>).

When the symbolic planner 61 provides the motion sequence 63, the motion planner 65 may be configured to generate a sequence of control commands 67 for causing the robot device R to execute each action included in the motion sequence 63. The motion planner 65 may also be configured to determine whether the robot device R can execute each action included in the motion sequence 63. The motion planner 65 may be, for example, the RRT-connect technique described in Reference 3 (James J. Kuffner, Jr., Steven M. LaValle, “RRT-Connect: An Efficient Approach to Single-Query Path Planning”, [online], [searched October 24, 2023], Internet URL: https://www.cs.cmu.edu/afs/cs/academic/class/15494-s12/readings/kuffner_icra2000.pdf>), or the CHOMP (Covariant Hamiltonian Optimization) technique described in Reference 4 (Nathan Ratliff et al., “CHOMP: Gradient Optimization Techniques for Efficient Motion Planning”, [online], [searched October 24, 2023], Internet URL: https://www.ri.cmu.edu/pub_files/2009/5/icra09-chomp.pdf>). A known motion planner may be used, such as STOMP, which is described in Reference 5 (Mrinal Kalakrishnan et al., "STOMP: Stochastic Trajectory Optimization for Motion Planning", [online], [searched October 24, 2023], Internet <URL: http://ros.fei.edu.br/roswiki/attachments/Papers(2f)ICRA2011_Kalakrishnan/kalakrishnan_icra2011.pdf>). The generated control command 67 may be provided to the robot device R as appropriate. The robot device R may start performing a task at any timing in accordance with the provided control command 67. Note that the configuration of the planner is not limited to the example shown in FIG. 1 and may be modified as appropriate depending on the embodiment. In another example, the planner may be configured to generate a behavior plan by directly inferring the control command 67 from the problem description 5 and the domain description 23.

As long as the problem description 5 includes descriptions of the initial state and the goal state (51, 52) and can be used to generate an action plan, the format of the problem description 5 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the generated problem description 5 may be configured to follow a predetermined format. The predetermined format may be provided by a language for planners (planning description languages) such as PDDL (Planning Domain Definition Language) or PDDLStream (Reference 6: "pddlstream", [online], [searched October 24, 2023], Internet <URL: https://github.com/caelan/pddlstream>). In one example of this embodiment, the generated problem description 5 follows a predetermined format, making it easier for a planner (e.g., symbolic planner 61) to generate an action plan from the problem description 5.

In another example, the generated problem statement 5 may not conform to a predetermined format. In this case, the generated problem statement 5 may be converted to conform to a predetermined format by an intermediate process such as a conversion program. Alternatively, the generated problem statement 5 may be used to generate an action plan without conforming to a predetermined format by a planner that can accept input in any format, such as a planner configured with a trained model for in-context learning.

The input and output data of the inference module 3 may be modified as appropriate depending on the embodiment. The inference module 3 may further accept input of any data other than the observation data 20 and the instruction information 21. The inference module 3 may also output any data other than the problem description 5. The problem description 5 may further include any information other than the descriptions of the initial state and the goal state (51, 52).

(Example of input/output data)
2 is a schematic diagram illustrating an example of input and output of the inference module 3 according to this embodiment. In the example of FIG. 2, the input data includes observation data 20, instruction information 21, and environment information 22. The problem description 5 (output data) includes descriptions (50, 51, 52) of the object, the initial state, and the goal state.

In one example, the environment may be observed by one or more sensors S, and the observation data 20 may accordingly be composed of sensing data from one or more sensors S. The type of sensor S is not particularly limited and may be selected appropriately depending on the embodiment. The sensor S may include, for example, a camera, depth sensor, infrared sensor, optical sensor, radar, LiDAR (Light Detection and Ranging), microphone, position sensor, other measurement sensor, etc. The position sensor may be, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. The sensing data may include, for example, image data, depth data, infrared data, measurement data from an optical sensor (such as marker detection results), radar data, LiDAR data, sound data, position data, other measurement data, etc. The sensor S may also include one or more measurement sensors for measuring the state of the robot device R. The measurement sensor may include, for example, an encoder, motion capture, a tactile sensor, a force sensor, an inertial measurement unit, etc. Accordingly, the sensing data may include measurement data of the robot device R (e.g., joint angles, hand positions, tactile data at the hand, force data at the hand, posture measurement data, etc.). The sensor S may be located outside the robot device R or within the robot device R.

In one example, the instruction information 21 may be composed of linguistic information that indicates a target in natural language. The natural language instruction may be acquired by any method. The natural language instruction may be acquired by, for example, text input, voice input, image input, or other input methods. The data format of the instruction information 21 (linguistic information) may be selected appropriately depending on the embodiment. The instruction information 21 may be composed of, for example, text data, voice data, image data, or other types of data. The acquired natural language instruction data may be used as the instruction information 21 as is, or may be used as the instruction information 21 after converting its data format. As an example of the latter, the instruction data may be converted using a conversion model, such as from voice to text or from text to voice, and the converted instruction data may be used as the instruction information 21. The conversion may include any analysis process, such as voice analysis. The conversion model may be included in the inference module 3 or may be prepared separately from the inference module 3. The conversion model may be composed of at least one of a trained model (a trained machine learning model) and a rule-based model. The natural language instructions may be given manually by an operator or the like, or automatically by computer processing. The instruction information 21 may be given as appropriate depending on the task. In one example, the instruction information 21 may be given each time, such as each time a task is given. In another example, the instruction information 21 may be given when a target is specified in advance, such as the Tower of Hanoi. In one example of this embodiment, when the environment is observed by a sensor S and a task target is given by linguistic instructions, a highly descriptive output can be obtained regarding the control of the robot device R.

The configuration of the observation data 20 and the instruction information 21 may be modified as appropriate depending on the embodiment. The observation data 20 may include, in addition to sensing data, any data other than sensing data, such as manually provided data or data generated by computer processing. In another example, the observation data 20 may not include sensing data, and may instead be composed of any data other than sensing data. In another example, the instruction information 21 may be provided in a format other than natural language. For example, the instruction information 21 may be provided in a text format using symbols other than natural language. For example, the instruction information 21 may be composed of an image showing a target state of an object present in the environment. The image may be composed of at least one of a real image and a virtual image.

In one example, the information processing device 1 may be configured to further acquire environmental information 22 related to the environment. Generating the problem description 5 from the acquired observation data 20 and instruction information 21 may be configured by generating the problem description 5 from the acquired observation data 20, instruction information 21, and environmental information 22. According to one example of this embodiment, the environmental information 22 can be used to impose constraints on the environment in which the task is performed, thereby narrowing down the conditions for generating the problem description 5. In other words, by further using the environmental information 22 as an explanatory variable in addition to the observation data 20 and instruction information 21, it becomes possible to more accurately identify the environment in which the task is performed. Therefore, an improvement in the accuracy of generating the problem description 5 can be expected.

Note that the environment information 22 may include any information related to the relationships of the robot device R that may be involved in generating the problem description 5. In one example, the environment information 22 may include at least one of the domain description 23 and domain information 24.

The domain description 23 defines phenomena common to all problems, including the skills of the robot device R. As described above, the domain description 23, together with the problem description 5, may be used by the planner to generate an action plan. If the problem description 5 is provided in a predetermined format such as PDDL or PDDLStream, the domain description 23 may also be provided in a predetermined format. For example, if PDDL is used, the problem description 5 may be "problem.pddl" and the domain description 23 may be "domain.pddl." In one example, the domain description 23 may include descriptions that define the skills of the robot device R (e.g., "actions"), descriptions that define the states of target objects present in the environment (e.g., "predicates"), descriptions that define the types of target objects (e.g., "types"), and descriptions that confirm compatibility with the planner (e.g., "requirements"). The states of the target object may include states that the target object can take when the robot device R executes the defined skills (behavior).

The domain information 24 may include any information that limits the domain for generating the problem statement 5. The domain information 24 may also be referred to as domain knowledge. The domain information 24 may be composed of any information about the environment other than the domain description 23. The domain information 24 may supplement the conditions for generating the problem statement 5 together with the domain description 23. In one example, the domain information 24 may include attribute information 241 of objects present in the environment. The attribute information 241 may include, for example, the name and characteristics of the object. The characteristics may include, for example, characteristics related to appearance, such as color, shape, and size. Examples of characteristics related to appearance include a cutting board being round and a counter being black. This makes it possible to narrow down the characteristics of objects appearing in the observation data 20, thereby improving the accuracy of inference for the observation data 20. In another example, as described below, when the inference module 3 is configured to include a trained model for in-context learning, the domain information 24 may include one or more input/output samples 243 for the trained model. For example, the output sample may be a ground truth sample of a portion of the problem statement 5 that is generated by the trained model. The input sample may be a sample of a portion of the observation data 20, instruction information 21, and environment information 22 that are provided to obtain a ground truth sample of the problem statement 5 and that is input to the trained model.

The format in which the input data (observation data 20, instruction information 21, and environmental information 22) are provided to the inference module 3 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the observation data 20, instruction information 21, and environmental information 22 may be input to the inference module 3 as is. In another example, preprocessing may be applied to at least one of the observation data 20, instruction information 21, and environmental information 22, and the preprocessed data may be input to the inference module 3. The preprocessing may include any computational process, such as a process for analyzing information, a process for adding information, or a process for reducing information. The computational model that performs the preprocessing may be included in the inference module 3 or may be prepared separately from the inference module 3. The computational model may be composed of at least one of a trained model (a trained machine learning model) and a rule-based model. For example, in a case in which the observation data 20 is composed of image data, the inference module 3 may be configured to accept input of the image data. Alternatively, the image data may be analyzed using any method, such as image processing or an analytical model. This analysis process is an example of preprocessing. The analysis results may include, for example, bounding box detection results, identification results of objects appearing in the image data, etc. The inference module 3 may be configured to accept input of analysis results for the image data.

In one example, the generated problem description 5 may further include a description 50 of objects present in the environment. The object description 50 may correspond to a list of objects present in the environment. The range of objects included in the description 50 may be determined appropriately depending on the embodiment. For example, the object description 50 may consist of a list of all objects that can be observed in the target environment. Alternatively, for example, the object description 50 may consist of a list of a portion of objects that can be observed. These portions of objects may be objects of interest, such as objects that may be involved in the task. In this case, objects of no interest (e.g., objects that may not be involved in the task) may be omitted from the object description 50. According to one example of this embodiment, the object description 50 makes it easier to identify objects present in the environment, thereby making it easier for a planner (e.g., a symbolic planner 61) to generate an action plan from the problem description 5. Note that if the descriptions of the initial state and the goal state (51, 52) are obtained in a predetermined format, the object description 50 may also be obtained in a predetermined format. For example, when using PDDL, the description of objects in problem.pddl is an example of an object description 50, the description of init (initial state) is an example of an initial state description 51, and the description of goal is an example of a target state description 52.

Note that the input/output format of the inference module 3 need not be limited to the example shown in Figure 2 and may be modified as appropriate depending on the embodiment. At least a portion of the environmental information 22 may be omitted. In the problem statement 5, the object description 50 may be omitted. In one example, when the robot device R operates in an environment where no other objects exist other than the robot device R (for example, a drone flying in the air where there are no other objects), the object description 50 may be omitted from the problem statement 5.

[Modify the problem description]
In one example, the information processing device 1 may be configured to execute a process related to modifying the generated problem description 5. The process of modifying the problem description 5 may be executed automatically by a computer or manually by an operator or the like.

As an example, the information processing device 1 may determine whether the generated problem description 5 is compatible with a planner (e.g., a symbolic planner 61). This determination may be made by any method, such as using at least one of a trained model and a rule-based model. For example, if the problem description 5 is generated to conform to a predetermined format, the information processing device 1 may determine whether the generated problem description 5 conforms to the predetermined format by evaluating whether the generated problem description 5 conforms to the predetermined format. If the generated problem description 5 is determined to be compatible with the planner, the generated problem description 5 may be provided to the planner as appropriate. On the other hand, if the generated problem description 5 is determined to be incompatible with the planner, the information processing device 1 may modify the generated problem description 5 as appropriate. For example, the information processing device 1 may modify the generated problem description 5 using a modification model. The modification model may be composed of at least one of a trained model and a rule-based model. Furthermore, for example, the information processing device 1 may output the problem description 5 to an output device together with factors that cause the problem description 5 to be incompatible with the planner (e.g., indicating parts that do not conform to the predetermined format). The information processing device 1 may accept modifications to the problem description 5 by an operator via an input device. Then, the information processing device 1 may modify the problem description 5 in accordance with the received content. As another example, the information processing device 1 may execute the above-described process of modifying the problem description 5 after omitting the process of determining whether or not the problem description 5 is suitable for the planner.

The timing of executing the process to modify the problem description 5 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the information processing device 1 may execute the process to modify the problem description 5 before the problem description 5 is used to generate an action plan by a planner (e.g., symbolic planner 61). In another example, after the generated problem description 5 is provided to the planner, the information processing device 1 may execute the process to modify the problem description 5 in response to an error occurring in the planner's attempt to generate an action plan, such as outputting an error message or being unable to generate an appropriate action plan. Note that the process to modify the problem description 5 may be executed by an external computer other than the information processing device 1. Furthermore, the process to modify the problem description 5 may be omitted.

Figure 3 schematically illustrates an example of the problem statement 5 revision process according to this embodiment. In one example, the inference module 3 may be configured to include a trained model 39 for in-context learning. In-context learning refers to acquiring the ability to make specific inferences through the context of input (prompts), such as input/output samples. In one example, the trained model 39 can acquire the ability for in-context learning by including a self-attention mechanism and an autoregressive model. This trained model 39 may be, for example, a large-scale language model (LLM), a large-scale visual language model (LVLM), etc. The large-scale visual language model may include a visual question answering model, an open vocabulary object detection model, an open vocabulary object segmentation model, etc. Alternatively, the trained model 39 may be a large-scale language model combined with one or more other modalities (such as sound), such as audio question answering. Furthermore, the trained model 39 may be a large-scale model of one or more modalities other than language, such as a Large Audio Model. The data format of the input to the trained model 39 may be selected appropriately depending on the embodiment.

In the example of FIG. 3, the inference module 3 includes a trained model 39 capable of in-context learning, which allows the inference module 3 to adapt to errors when they occur and modify the problem statement 5 in response to the errors. Therefore, in the example of FIG. 3, the information processing device 1 may generate the problem statement 5 by providing input data 200 to the inference module 3 and executing the calculation process of the inference module 3. The input data 200 includes observation data 20 and instruction information 21 of a target scene for which an action plan is to be generated. In one example, the input data 200 may further include environmental information 22.

The generated problem description 5 may be provided to a planner as appropriate. The planner may attempt to generate an action plan using the provided problem description 5. When the planner has the configuration of FIG. 1 , the generated problem description 5 may be provided to a symbolic planner 61, which may attempt to generate an action sequence 63 from the problem description 5. The process of generating an action plan by this planner may be executed by at least one of the information processing device 1 and an external computer other than the information processing device 1. As described above, the domain description 23 may be used together with the problem description 5 to generate the action plan.

If the problem description 5 is appropriate, the planner can generate an appropriate action plan from the problem description 5. In the example of FIG. 1, an appropriate action sequence 63 is obtained, and the motion planner 65 can generate a sequence of control commands 67 accordingly. The generated action plan may be used as appropriate to control the movement of the robot device R. On the other hand, if the problem description 5 is inappropriate, an error occurs in the planner's generation of the action plan. An inappropriate problem description 5 may include, for example, the problem description 5 not being compatible with the planner, an inability to generate an action plan that avoids prohibited items, or an inability to generate an action plan with the given skills of the robot device R. An inappropriate problem description 5 may include, for example, the problem description 5 not conforming to a predetermined format. In this case, an error message 615 is output from the planner.

When the generated problem description 5 is provided to the planner to generate a behavior plan for the robot device R, if the planner outputs an error message 615, the information processing device 1 may acquire the output error message 615. The configuration of the error message 615 is not particularly limited and may be determined appropriately depending on the embodiment, such as the configuration of the planner. The information processing device 1 may then provide the acquired error message 615 and problem description 5 to the inference module 3 and re-execute the calculation processing of the inference module 3. In other words, the information processing device 1 may use the problem description 5 and error message 615 as a re-prompt to re-execute the process of generating the problem description 5. As a result, the information processing device 1 may use the inference module 3 to generate a new problem description 5 from the problem description 5 and error message 615 (i.e., modify the problem description 5). According to one example of this embodiment, if an appropriate problem description 5 cannot be obtained, the problem description 5 can be automatically modified.

Note that the data provided to the inference module 3 during a re-prompt does not have to be limited to the problem description 5 and the error message 615. The configuration of the re-prompt may be modified as appropriate depending on the embodiment. In another example, the information processing device 1 may provide at least a portion of the input data 200 to the inference module 3 along with the problem description 5 and the error message 615. This is expected to improve the accuracy of correcting the problem description 5.

The information processing device 1 may also recursively execute the process of correcting the problem description 5 through re-prompts. The number of repetitions is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the information processing device 1 may repeatedly execute the process of correcting the problem description 5 through re-prompts until the error message 615 is no longer output. In another example, the number of times the process of correcting the problem description 5 through re-prompts is executed may be specified in advance. If the error message 615 is still output after the information processing device 1 has executed the correction process through re-prompts a predetermined number of times, the information processing device 1 may stop the correction process through re-prompts and output the results of the execution up to that point (e.g., the problem description 5 generated, etc.).

Furthermore, the re-prompting method is not limited to the above example and may be modified as appropriate depending on the embodiment. In another example, the information processing device 1 may modify the problem statement 5 by using a prompt modified in response to the error message 615 as a re-prompt, together with or instead of the problem statement 5 and the error message 615. For example, when the error message 615 is output, the information processing device 1 may use a modified model to appropriately modify the prompt (input data 200) provided to the inference module 3 in response to the output error message 615. The modified model may be configured as appropriate using at least one of a trained model and a rule-based model. The information processing device 1 may then provide the modified prompt to the inference module 3 again and execute the calculation process of the inference module 3 to generate a new problem statement 5.

[Inference module]
The inference module 3 is configured to execute an inference process to generate a problem description 5 from input data including observation data 20 and instruction information 21. As long as the inference module 3 is capable of executing such an inference process, the configuration of the inference module 3 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, the inference module 3 may be configured by at least one of a rule-based model and a trained model (a trained machine learning model).

The rule-based model is configured to derive an inference result (in this embodiment, the result of generating problem statement 5) from a given input in accordance with rules. The rules may be set as appropriate. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used to calculate the desired inference (in this embodiment, the generation of problem statement 5). The machine learning model may be configured, for example, from a neural network, regression model, decision tree model, support vector machine, or other functional formula (calculation model). The machine learning method may be selected as appropriate depending on the machine learning model used (for example, backpropagation, etc.).

Machine learning involves adjusting (optimizing) values of computational parameters using training samples. Typically, a trained model may be generated by supervised learning using multiple training datasets, each of which is composed of a combination of input samples (training samples) and output samples (teacher signals, labels). For example, the input samples may be samples of input data (such as observation data 20 and instruction information 21), and the output samples may be samples of output data (such as problem descriptions 5). In supervised learning, the values of computational parameters of a machine learning model may be adjusted so that, when an input sample is provided, the output obtained from the machine learning model matches the corresponding output sample. However, the method for generating a trained model is not limited to this example and may be appropriately changed depending on the embodiment. The training dataset is not limited to the above example and may be selected appropriately depending on the embodiment. For example, when acquiring in-context learning capabilities, data other than those described above may be used for the training dataset. Furthermore, the learning method is not limited to supervised learning; for example, other methods such as unsupervised learning (including self-supervised learning) and reinforcement learning may be used. The machine learning model may be trained online or offline. The machine learning model may be appropriately tuned by transfer learning, re-learning, additional learning, etc. Additional learning may be performed using, for example, LoRA (Low-Rank Adaptation of Large Language Models) in Reference 7 (“LoRA”, [online], [searched October 24, 2023], Internet <URL: https://github.com/microsoft/LoRA>), Adapter in Reference 8 (Neil Houlsby et al., “Parameter-Efficient Transfer Learning for NLP”, [online], [searched October 24, 2023], Internet <URL: http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf>), or Brian Lester et al.
al., "The Power of Scale for Parameter-Efficient Prompt Tuning", [online], [Retrieved October 24, 2023], Internet <URL: https://arxiv.org/abs/2104.08691>), adds a module to an existing trained model, and adjusts the parameters of the additional module while leaving the parameters of the existing trained model unchanged. Furthermore, additional learning may include machine learning such as Prompt Tuning in Reference 10 (Kecheng Zheng et al., "Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models", [online], [Retrieved October 24, 2023], Internet <URL: https://
openaccess.thecvf.com/content/ICCV2023/papers/Zheng_Regularized_Mask_Tuning_Uncovering_Hidden_Knowledge_in_Pre-Trained_Vision-Language_Models_ICCV_2023_paper.pdf), etc. When a trained model capable of in-context learning, such as the large-scale language model described above, is used for at least a part of the inference module 3, the trained model may be specialized for a specific problem, domain, or other situation by performing additional learning.

In one example, the inference module 3 may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, and the connection relationships between the nodes in each layer. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or an LSTM (Long Short-Term Memory). The neural network may include any type of model, such as a diffusion model, a transformer model, or a generative model. The connection weights between each node included in the neural network and the threshold value for each node are examples of calculation parameters.

As described above, in one example, the inference module 3 may include a trained model 39 for in-context learning. In this case, the input data provided to the inference module 3 may include one or more input/output samples 243 for the trained model 39 as domain information 24 of the environment information 22. The input/output samples 243 may be prepared appropriately depending on the domain for which the problem description 5 is to be generated. In one example of this embodiment, by providing the input/output samples 243 to the trained model 39, in-context learning can be performed, and the trained model 39 can be adapted to the target domain. In other words, by preparing input/output samples 243 for each domain, the inference module 3 can generate a general-purpose problem description 5 without replacing the trained model 39. Therefore, according to this example of the present embodiment, it is expected that problem descriptions 5 can be generated for tasks in a variety of domains.

In one example, when the inference module 3 includes a trained model 39 for in-context learning, a domain description 23 may be prepared for each type of robot device R. Additionally, an output sample (a sample problem description 5) corresponding to the domain description 23 may be prepared for each type as domain information 24. In response, the information processing device 1 may accept a type designation. The designation method is not particularly limited and may be selected appropriately depending on the embodiment. Typically, the type may be designated manually, such as by an operator operating an input device, entering text, or entering voice information. The type designation may be included in the instruction information 21. The information processing device 1 may provide the domain description 23 and output sample (domain information 24) corresponding to the designated type as environment information 22 to the inference module 3 (trained model 39). This allows for the creation of a problem description 5 for generating a behavior plan for a robot device R having a skill of the designated type. In other words, this example of the present embodiment is expected to enhance the versatility of the inference module 3 and improve the accuracy of generating problem descriptions 5 that match the type.

The input/output form of the inference module 3 may be determined appropriately depending on the embodiment. As described above, in one example, the input data (observation data 20, instruction information 21, and environmental information 22) may be provided to the inference module 3 as is. In another example, any preprocessing may be applied to at least a portion of the input data before providing it to the inference module 3. The preprocessing may include computational processing such as feature extraction, other analysis, and inference. The inference module 3 may be provided with the preprocessed input data. In one example, the output of the inference module 3 may be configured to directly indicate the inference result (problem statement 5). In another example, the output of the inference module 3 may be configured to indirectly indicate the inference result. In this case, the inference result may be obtained by performing any information processing (interpretation processing) on the output of the inference module 3. The problem statement 5 may be generated in real time using data obtained at the present time (which may include past data) or retrospectively using data obtained in the past.

Furthermore, the inference module 3 may be configured as an integrated unit, or may be configured from a collection of multiple subelements. In one example, when the inference module 3 is configured from a collection of multiple subelements, each subelement may be appropriately configured to generate a portion (corresponding part) of the problem description 5. For example, one or more subelements may be prepared for each description (50, 51, 52) of the object, initial state, and goal state in the problem description 5. Each subelement may be configured from at least one of a machine learning model and a rule-based model.

(Example of inference module configuration)
FIG. 4 schematically illustrates an example of the configuration of the inference module 3 according to this embodiment. In the example of FIG. 4, a scenario is assumed in which the problem description 5 further includes descriptions 50 of objects present in the environment. In one example, the inference module 3 may include an object estimator 31, an initial state estimator 33, and a goal estimator 35. The object estimator 31 is an example of one or more subelements corresponding to the object descriptions 50. The initial state estimator 33 is an example of one or more subelements corresponding to the initial state descriptions 51. The goal estimator 35 is an example of one or more subelements corresponding to the goal state descriptions 52.

The object estimator 31 may be configured to generate descriptions 50 of one or more objects present in the environment from the acquired observation data 20. Generating the object descriptions 50 may include detecting the objects. Generating the problem statement 5 using the inference module 3 may include using the object estimator 31 to generate descriptions 50 of objects present in the environment from the acquired observation data 20. The observation data 20 contains information about objects present in the environment. Therefore, according to one example of this embodiment, the object descriptions 50 in the problem statement 5 can be appropriately generated.

As long as it is possible to generate an object description 50 from the observation data 20, the configuration of the object estimator 31 is not particularly limited and may be selected appropriately depending on the embodiment. The above description of the configuration of the inference module 3 may also be applied to the object estimator 31. The object estimator 31 may be configured using at least one of a rule-based model and a trained model.

The object estimator 31 may use a model that can infer object descriptions 50 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model that can infer object descriptions 50 generically as the object estimator 31, rather than a model that can infer object descriptions 50 only in a specific domain. In one example, the object estimator 31 may be equipped with a trained model for in-context learning. By providing a trained model capable of in-context learning, the versatility of the object estimator 31 can be increased. In other words, it can infer descriptions 50 of objects present in the environment generically, which can be expected to enable the generation of object descriptions 50 for tasks in various domains.

The input/output form of the object estimator 31 may be determined appropriately depending on the embodiment. The object estimator 31 may further receive input of any data other than the observation data 20. The input data may be provided to the object estimator 31 as is, or may be provided to the object estimator 31 after preprocessing has been applied. The object estimator 31 may further output any data other than the object description 50.

In one example, the information processing device 1 may further acquire environmental information 22 related to the environment. The object estimator 31 may further use the acquired environmental information 22 to generate the object description 50. In other words, generating the object description 50 from the acquired observation data 20 may be configured by generating the object description 50 from the acquired observation data 20 and environmental information 22. According to one example of this embodiment, by further using the environmental information 22, it is possible to expect an improvement in the accuracy of generating the object description 50 in the problem description 5.

The environmental information 22 provided to the object estimator 31 may be selected appropriately depending on the embodiment. The environmental information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environmental information 22 may include at least a portion of the domain description 23. Furthermore, for example, the environmental information 22 may include attribute information 241 of objects present in the environment as the domain information 24. This makes it possible to more accurately identify objects present in the environment using the attribute information 241, which is expected to improve the accuracy of generating the object description 50 in the problem description 5. Furthermore, for example, if the object estimator 31 is equipped with a trained model for in-context learning, the environmental information 22 may include one or more input/output samples 243 as the domain information 24. The input samples may be composed of samples of the observation data 20. If environmental information 22 other than the input/output samples 243 is provided to the object estimator 31, the input samples may include samples of that environmental information 22. The output samples may be composed of correct answer samples of the object description 50 corresponding to the input samples.

The domain description 23 and domain information 24 (attribute information 241, input/output samples 243) may be prepared as appropriate depending on the embodiment. In one example, the domain description 23 and domain information 24 may be prepared in advance. The domain description 23 and domain information 24 may be stored in memory resources within the information processing device 1, or may be provided to the information processing device 1 from an external source when used. In another example, at least a portion of the domain description 23 and domain information 24 may be generated as appropriate when used. For example, reference information including a list of objects and attribute information may be prepared in advance. The information processing device 1 may extract object candidates present in the environment from at least one of the instruction information 21 (e.g., linguistic information) and the domain description 23 (e.g., a description defining the type of target object). The information processing device 1 may compare the list of extracted object candidates with the list of reference information and extract attribute information of objects that match the object candidates from the reference information, thereby generating attribute information 241 to be provided to the object estimator 31. The process of generating this attribute information 241 may be performed as preprocessing, or as processing within the inference module 3.

The initial state estimator 33 may be configured to generate a description 51 of the initial state of one or more objects present in the environment from the acquired observation data 20. Generating the problem statement 5 using the inference module 3 may include generating a description 51 of the initial state of the objects present in the environment using the initial state estimator 33. The observation data 20 represents the initial states of the objects present in the environment. Therefore, according to one example of this embodiment, the description 51 of the initial state in the problem statement 5 can be appropriately generated.

As long as it is possible to generate an initial state description 51 from the observation data 20, the configuration of the initial state estimator 33 is not particularly limited and may be selected appropriately depending on the embodiment. As with the object estimator 31, the above description regarding the configuration of the inference module 3 may also be applied to the initial state estimator 33. The initial state estimator 33 may be configured using at least one of a rule-based model and a trained model.

The initial state estimator 33 may use a model capable of inferring the initial state description 51 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model capable of inferring the initial state description 51 generically as the initial state estimator 33, rather than a model capable of inferring the initial state description 51 only in such a specific domain. In one example, the initial state estimator 33 may include a trained model for in-context learning. By including a trained model capable of in-context learning, the versatility of the initial state estimator 33 can be increased. In other words, the initial state description 51 of an object existing in the environment can be inferred generically, which can be expected to support the generation of initial state descriptions 51 for tasks in various domains.

The input/output form of the initial state estimator 33 may be determined appropriately depending on the embodiment, similar to the object estimator 31. The initial state estimator 33 may further accept input of any data other than the observation data 20. The input data may be provided to the initial state estimator 33 as is, or may be provided to the initial state estimator 33 after preprocessing has been applied. The input data may be provided to the object estimator 31 first, and the calculation results (including intermediate calculation results) of the object estimator 31 on the input data may be provided to the initial state estimator 33. For example, providing the observation data 20 to the initial state estimator 33 may include providing the observation data 20 as is to the initial state estimator 33, providing the preprocessing results of the observation data 20 to the initial state estimator 33, and providing the calculation results of the object estimator 31 on the observation data 20 to the initial state estimator 33. The preprocessing results and the calculation results of the object estimator 31 may be, for example, the detection results of an object in the observation data 20. However, the order in which the input data is provided is not limited to this example. The input data may be provided to the initial state estimator 33 first, and the results of the calculations performed by the initial state estimator 33 on the input data may be provided to the object estimator 31. For example, providing the observation data 20 to the object estimator 31 may include providing the observation data 20 directly to the object estimator 31, providing the results of preprocessing the observation data 20 to the object estimator 31, and providing the results of the calculations performed by the initial state estimator 33 on the observation data 20 to the object estimator 31. The initial state estimator 33 may also output any data other than the description 51 of the initial state of the object.

In one example, similar to the object estimator 31, the initial state estimator 33 may further use the environmental information 22 to generate the initial state description 51. That is, generating the initial state description 51 from the observation data 20 may be configured by generating the initial state description 51 from the observation data 20 and the environmental information 22. The environmental information 22 provided to the initial state estimator 33 may be selected appropriately depending on the embodiment. The environmental information 22 may include at least one of the domain description 23 and the domain information 24. For example, the environmental information 22 may include at least a portion of the domain description 23. Furthermore, for example, the environmental information 22 may include attribute information 241 as the domain information 24. Furthermore, for example, when the initial state estimator 33 has a trained model for in-context learning, the environmental information 22 may include one or more input/output samples 243 as the domain information 24. The input sample may be configured from a sample of the observation data 20 (including the preprocessing result and the calculation result of the object estimator 31). When environmental information 22 other than the input/output samples 243 is provided to the initial state estimator 33, the input samples may include samples of that environmental information 22. The output samples may be composed of correct samples of the initial state description 51 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is expected that the accuracy of generating the initial state description 51 in the problem description 5 can be improved.

In one example, the calculation process of the object estimator 31 may be completed before the initial state estimator 33. In this case, the object description 50 generated by the object estimator 31 may be provided to the initial state estimator 33 as input data for generating the initial state description 51. That is, the initial state estimator 33 may use the object description 50 generated by the object estimator 31 to generate the initial state description 51.
0 may be further used to generate the initial state description 51. When the initial state estimator 33 comprises a trained model for in-context learning and the environmental information 22 provided to the initial state estimator 33 includes input/output samples 243, the input samples may include samples of the object description 50. However, the order of operations within the inference module 3 is not limited to this example. The operation process of the initial state estimator 33 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. When the operation process of the initial state estimator 33 is executed to completion before the object estimator 31, the initial state description 51 generated by the initial state estimator 33 may be provided to the object estimator 31. When the object estimator 31 comprises a trained model for in-context learning and the environmental information 22 provided to the object estimator 31 includes input/output samples 243, the input samples may include samples of the initial state description 51.

The goal estimator 35 may be configured to generate a goal state description 52 of one or more objects in the environment from the acquired instruction information 21. Generating the problem description 5 using the reasoning module 3 may include generating a goal state description 52 of objects in the environment using the goal estimator 35. The instruction information 21 contains information related to the goal of the task. Therefore, according to one example of this embodiment, the goal state description 52 in the problem description 5 can be appropriately generated.

As long as it is possible to generate a description 52 of the goal state from the instruction information 21, the configuration of the goal estimator 35 is not particularly limited and may be selected appropriately depending on the embodiment. As with the object estimator 31, etc., the above description of the configuration of the inference module 3 may also be applied to the goal estimator 35. The goal estimator 35 may be configured using at least one of a rule-based model and a trained model.

The goal estimator 35 may use a model that can infer goal state descriptions 52 only in a specific domain. However, in order to be commonly usable for tasks in various domains, it is desirable to use a model that can infer goal state descriptions 52 generically as the goal estimator 35, rather than a model that can infer goal state descriptions 52 only in such a specific domain. In one example, the goal estimator 35 may be equipped with a trained model for in-context learning. By providing a trained model capable of in-context learning, the versatility of the goal estimator 35 can be increased. In other words, it can infer goal state descriptions 52 of objects in the environment generically, which can be expected to enable the generation of goal state descriptions 52 for tasks in various domains.

The input/output format of the target estimator 35 may be determined appropriately depending on the embodiment, similar to the object estimator 31, etc. The target estimator 35 may further accept input of any data other than the instruction information 21. The input data may be provided to the target estimator 35 as is, or may be provided to the target estimator 35 after preprocessing has been applied. Similar to the relationship between the object estimator 31 and the initial state estimator 33, the calculation results (including intermediate calculation results) of at least one of the object estimator 31 and the initial state estimator 33 may be provided to the target estimator 35. Alternatively, the calculation results (including intermediate calculation results) of the target estimator 35 may be provided to at least one of the object estimator 31 and the initial state estimator 33. The target estimator 35 may also output any data other than the description 52 of the object's target state.

In one example, similar to the object estimator 31, the target estimator 35 may further use the environment information 22 to generate the target state description 52. That is, generating the target state description 52 from the instruction information 21 may be configured by generating the target state description 52 from the instruction information 21 and the environment information 22. The environment information 22 to be provided to the target estimator 35 may be appropriately selected depending on the embodiment. The environment information 22 may be generated by generating the target state description 52 from the instruction information 21 and the environment information 22.
The environmental information 22 may include at least one of the above. For example, the environmental information 22 may include at least a portion of the domain description 23. For example, the environmental information 22 may include attribute information 241 as the domain information 24. For example, if the goal estimator 35 has a trained model for in-context learning, the environmental information 22 may include one or more input/output samples 243 as the domain information 24. The input samples may be composed of samples of the instruction information 21. If environmental information 22 other than the input/output samples 243 is provided to the goal estimator 35, the input samples may include samples of the environmental information 22. The output samples may be composed of correct samples of the goal state description 52 corresponding to the input samples. The environmental information 22 (domain description 23, domain information 24) may be prepared as appropriate depending on the embodiment. By further using the environmental information 22, it is expected that the accuracy of generating the goal state description 52 in the problem description 5 can be improved. The environmental information 22 provided to the object estimator 31, the initial state estimator 33, and the target estimator 35 may or may not overlap at least partially.

In one example, the calculation process of the object estimator 31 may be completed before the target estimator 35. In this case, the object description 50 generated by the object estimator 31 may be provided to the target estimator 35 as input data for generating the target state description 52. Similarly, the calculation process of the initial state estimator 33 may be completed before the target estimator 35. In this case, the initial state description 51 generated by the initial state estimator 33 may be provided to the target estimator 35 as input data for generating the target state description 52. In other words, the target estimator 35 may further use at least one of the object description 50 and the initial state description 51 generated by the object estimator 31 and the initial state estimator 33 to generate the target state description 52. When the target estimator 35 has a trained model for in-context learning and the environmental information 22 provided to the target estimator 35 includes input/output samples 243, the input samples may include at least one of the object description 50 and the initial state description 51. However, the order of operations within the inference module 3 does not need to be limited to this example. The calculation process of the goal estimator 35 may be executed at least partially in parallel with the object estimator 31, or may be executed to completion before the object estimator 31. If the calculation process of the goal estimator 35 is executed to completion before the object estimator 31, the goal state description 52 generated by the goal estimator 35 may be provided to the object estimator 31. Similarly, the calculation process of the target estimator 35 may be executed at least partially in parallel with the initial state estimator 33, or may be executed to completion before the initial state estimator 33. If the calculation process of the target estimator 35 is executed to completion before the initial state estimator 33, the goal state description 52 generated by the target estimator 35 may be provided to the initial state estimator 33. If at least one of the object estimator 31 and the initial state estimator 33 includes a trained model for in-context learning, and the provided environment information 22 includes input/output samples 243, the input samples may include samples of the goal state description 52.

Note that the configuration of the inference module 3 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, at least any combination of the object estimator 31, the initial state estimator 33, and the target estimator 35 may be integrated. At least any one of the object estimator 31, the initial state estimator 33, and the target estimator 35 may be omitted as appropriate. Furthermore, in the example of FIG. 4 , the inference module 3 includes the trained model 39 for in-context learning, which may be configured by at least any one of the object estimator 31, the initial state estimator 33, and the target estimator 35 being equipped with a trained model for in-context learning. When the inference module 3 includes the trained model 39 for in-context learning and the environmental information 22 provided to the inference module 3 includes input/output samples 243, the information processing device 1 may use the input/output samples 243 as few-shot prompting. For example, the information processing device 1 may provide only the input/output sample 243 to the trained model 39 before generating the problem statement 5, or may provide the input/output sample 243 together with other input data to the trained model 39 when generating the problem statement 5. Then, the information processing device 1 may execute calculation processing on the trained model 39. In this way, the information processing device 1 may execute in-context learning on the trained model 39 and adapt the trained model 39 to the domain in which the problem statement 5 is generated.

(Example of an object estimator)
5A schematically illustrates an example of the object estimator 31 according to this embodiment. In one example, the object estimator 31 may include a trained model 311 and a rule-based model 313. When observation data 20 and environmental information 22 are input, the trained model 311 may be configured to detect an object appearing in the observation data 20 using the environmental information 22 as a clue, and output an object detection result 201. The trained model 311 may be a trained model capable of in-context learning.

For example, the sensor S may be a camera, and the observation data 20 may be image data. The environmental information 22 may be a list of attribute information 241 that indicates the attributes of an object in linguistic terms, such as "white bowl," "round cutting board," etc. The trained model 311 may be an open vocabulary object detection model or an open vocabulary object segmentation model proposed in Reference 11 (Shilong Liu et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection”, [online], [searched October 24, 2023], Internet <URL: https://arxiv.org/abs/2303.05499>), Reference 12 (Alexander Kirillov et al., “Segment Anything”, [online], [searched October 24, 2023], Internet <URL: https://arxiv.org/abs/2304.02643>), etc. The detection result 201 may include at least one of a result of detecting the area in which the object is captured as a bounding box and a result of identifying the object. The rule-based model 313 may be configured to generate an object description 50 from the object detection result 201 by the trained model 311 in accordance with rules. The rules may be set as appropriate.

Note that the configuration of FIG. 5A is merely one example of the object estimator 31. The configuration of the object estimator 31 is not limited to the example of FIG. 5A and may be modified as appropriate depending on the embodiment. In another example, the trained model 311 may be provided with environmental information 22 other than attribute information 241. The observation data 20 may further include data in a format other than image data, or may be composed solely of data in a format other than image data. The trained model 311 and the rule-based model 313 may be configured integrally. The rule-based model 313 may be replaced by the trained model. The trained model 311 may be configured to generate an object description 50. Accordingly, the rule-based model 313 may be omitted.

(Specific example of an initial state estimator)
FIG. 5B schematically illustrates an example of the initial state estimator 33 according to this embodiment. In one example, the initial state estimator 33 may include a detector 331 and a trained model 333. The information processing device 1 may extract partial data 205 corresponding to each object from the observation data 20 in accordance with the object detection results 201. The detector 331 may be configured to generate a caption 206 for each object from the extracted partial data 205 for each object. The caption 206 for each object may be configured to include information about the object appearing in the partial data 205. The trained model 333 may be configured to generate a description 51 of the initial state of the object from the object detection results 201 and the caption 206. The trained model 333's use of the object detection results 201 is an example of the initial state estimator 33 using the calculation results of the object estimator 31 as the observation data 20. The detector 331 and the trained model 333 may each be a trained model capable of in-context learning.

For example, the partial data 205 may be partial image data extracted by a bounding box of each object. The detector 331 is based on the method described in Reference 13 (Junnan Li et al., "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Image Encoders").
The pre-trained model 333 may be a visual question answering model proposed in, for example, "OpenAI, "GPT-4 Language Models", [online], [searched October 24, 2023], the Internet (URL: https://arxiv.org/abs/2301.12597>). A question may be provided to the detector 331 along with the partial data 205 as appropriate. In one example, the question may be composed of predetermined text data (such as "Q: what does this object describe? A: .") that asks about an object appearing in a partial image. The pre-trained model 333 may be a visual question answering model proposed in, for example, "OpenAI, "GPT-4 Technical Report", [online], [searched October 24, 2023], The caption 206 may be a large-scale language model (LLM) proposed on the Internet (URL: https://arxiv.org/pdf/2303.08774.pdf), etc. Accordingly, the caption 206 may be text data describing the partial data 205 (partial image data). The object detection result 201 may be appropriately converted into text data 202, and the obtained text data 202 may be provided to the trained model 333 as the detection result 201 (observation data 20). In one example, the text data 202 may include the name of the detected object (identification result) and the coordinates of the object's bounding box. The input sample 243 may be composed of text indicating the target range. The trained model 333 may further be provided with an object description 50 generated by the object estimator 31. In addition, input/output samples 243 may further be provided to the trained model 333 as the environment information 22 (domain information 24). In one example, the input sample may be composed of samples of the text data 202, the caption 206, and the object description 50, and the output sample may be composed of correct samples of the corresponding initial state description 51. The input/output samples 243 may be used for few-shot prompting of the trained model 333.

Note that the configuration of FIG. 5B is merely one example of the initial state estimator 33. The configuration of the initial state estimator 33 is not limited to the example of FIG. 5B and may be modified as appropriate depending on the embodiment. In another example, the trained model 333 may be provided with environmental information 22 other than the input/output samples 243. In another example, the trained model 333 may include an input section that accepts input of data of modalities other than text, such as a large-scale visual language model. In this way, the trained model 333 may be configured to be able to accept input of data of other modalities. Accordingly, the observation data 20 or the detection result 201 may be provided to the trained model 333 as is. The caption 206 may be composed of data in a format other than text data. Furthermore, the partial data 205 (observation data 20) may further include data in a format other than image data, or may be composed solely of data in a format other than image data. The data format of the detection result 201 is not particularly limited and may be selected as appropriate depending on the embodiment. The detector 331 may be omitted.

(Example of a target estimator)
5C is a schematic diagram illustrating an example of a target estimator 35 according to the present embodiment. In one example, the target estimator 35 may include a trained model 351. The trained model 351 may be configured to generate a target state description 52 of the object from the instruction information 21. The trained model 351 may be capable of in-context learning.

For example, the trained model 351 may be a large-scale language model (LLM) proposed in Reference 14 or the like. Accordingly, the instruction information 21 may be composed of text data instructing a target in natural language. The trained model 351 may further be provided with at least one of an object description 50 generated by the object estimator 31 and an initial state description 51 generated by the initial state estimator 33. Furthermore, input/output samples 243 may be further provided to the trained model 351 as the environment information 22 (domain information 24). In one example, the input samples may be composed of samples of the instruction information 21, the object description 50, and the initial state description 51, and the output samples may be composed of correct samples of the corresponding target state description 52. The input/output samples 243 may be used for few-shot prompting of the trained model 351. The trained model 351 may be prepared in common with the trained model 333 of the initial state estimator 33, or may be prepared separately.

Note that the configuration of Figure 5C is merely one example of the target estimator 35. The configuration of the target estimator 35 is not limited to the example of Figure 5C and may be modified as appropriate depending on the embodiment. In another example, the trained model 351 may be provided with environmental information 22 other than the input/output sample 243. In another example, the trained model 351 may be configured to be able to accept input of data of other modalities, similar to the trained model 333 described above. Accordingly, the trained model 351 may be provided with data in a format other than text data. The instruction information 21 may further include data in a format other than text data, or may be composed solely of data in a format other than text data.

[Robot device]
The robot device R is not particularly limited and may be selected appropriately depending on the embodiment. The robot device R may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile body configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, or an orthogonal robot. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a care robot, a security robot, or a transport robot. The content of the autonomous processing may be selected appropriately depending on the embodiment. The mobile body may include, for example, a cleaning robot, the above-mentioned autonomous robot configured to move (including a mobile robot), a vehicle configured to be self-driving, or an air vehicle capable of self-flying (such as a drone). The robot device R may exist in real space or virtual space. The action plan generated using the problem description 5 may be performed for controlling the robot device R in real space or for simulating the robot device R in virtual space.

The generation of the problem description 5 according to this embodiment may be applied to any task that can be accomplished by the robot device R. The task may be, for example, work, movement, etc. The work may be, for example, assembly, cooking, cleaning, chemical experiments, etc. If the robot device R is equipped with a communication device, the sensing data (observation data 20) may include data obtained through communication. For example, in a situation where a behavior plan (movement plan) for a moving object is generated, the sensing data may include data obtained through road-to-vehicle communication, vehicle-to-vehicle communication, etc.

The robot device R may also be used in situations where it coexists with humans. In such situations, the instruction information 21 may be provided by a human (operator). The generated problem description 5 may be output in a manner perceptible to humans (e.g., output to a display). The information processing device 1 may accept human intervention (manual correction) of the generated problem description 5. In one example, human intervention may correct the driving characteristics of the robot device R, such as the amount of movement, angle, posture, and sway. The problem description 5 may be generated through interactive interaction between the information processing device 1 and the operator by using a trained model capable of in-context learning in the inference module 3 (at least one of the object estimator 31, initial state estimator 33, and target estimator 35). Human input may be provided directly to the information processing device 1 or indirectly to the information processing device 1 via the robot device R. Human input may be provided by any method, such as operating an input device (including text input), recording, or capturing an image. The planner may be configured to generate an action plan from the problem statement 5, taking into account the presence of humans.

[Problem description output]
The output form of the problem description 5 is not particularly limited and may be selected appropriately depending on the embodiment. In a simple example, the generated problem description 5 may be output as is. Outputting the problem description 5 may include outputting the problem description 5 to at least one of a memory resource of the information processing device 1, an output device connected to the information processing device 1, and an external computer. The output method may be selected appropriately depending on the embodiment. For example, the problem description 5 may be output as text, an image, audio, or a combination thereof.

In another example, outputting the problem description 5 may include at least a part of providing the problem description 5 to a planner and generating an action plan using the planner. If generating an action plan using the planner is included, outputting the problem description 5 may further include controlling the operation of the robotic device R (i.e., driving the robotic device R) in accordance with the generated action plan.

The generation of an action plan by the planner may be executed by at least one of the information processing device 1 and one or more external computers. If the planner includes a symbolic planner 61 and a motion planner 65, the information processing device 1 may execute the calculations for one of the symbolic planner 61 and the motion planner 65, and the external computer may execute the calculations for the other. The calculations for both the symbolic planner 61 and the motion planner 65 may be executed by either the information processing device 1 or one or more external computers. If the calculations for both the symbolic planner 61 and the motion planner 65 are executed by one or more external computers, the external computer that executes the calculations for the symbolic planner 61 and the external computer that executes the calculations for the motion planner 65 may be the same or different.

Furthermore, the control of the robot device R according to the generated behavior plan may be executed by at least one of the information processing device 1 and an external computer. When at least a portion of the generation of the behavior plan by the planner and the control of the robot device R are executed by an external computer, the external computer that executes the generation of the behavior plan by the planner and the external computer that controls the robot device R may be the same or different. One or more external computers may execute at least a portion of the generation of the behavior plan, and the information processing device 1 may execute the control of the robot device R according to the generated behavior plan. The information processing device 1 and the external computer may exchange data as appropriate using any method, such as data communication.

§2 Configuration example [Hardware configuration]
6 is a schematic diagram illustrating an example of a hardware configuration of the information processing device 1 according to this embodiment. In the example of FIG. 6, the information processing device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

The control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, as a hard disk drive or solid-state drive. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the program 81 and module data 300.

The program 81 is a program for causing the information processing device 1 to execute information processing (see FIG. 8 described later) related to the generation of the problem statement 5. The program 81 includes a series of instructions for the information processing. The module data 300 indicates information related to the inference module 3.
As long as the inference module 3 can be reproduced when generating the module data 300, the configuration of the module data 300 is not particularly limited and may be determined appropriately depending on the embodiment. For example, if the module data 300 includes a machine learning model, the module data 300 may include information indicating values of calculation parameters adjusted by machine learning. In some cases, the module data 300 may further include information indicating the configuration of the machine learning model (e.g., the structure of a neural network). If the module data 300 includes a rule-based model, the module data 300 may include information indicating rules. If the inference module 3 has the configuration shown in FIG. 4 , information regarding the object estimator 31, the initial state estimator 33, and the target estimator 35 may be stored as separate data (files), or at least any combination of the information may be stored as the same data. The module data 300 may be incorporated into the program 81.

The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, etc. If the external interface 13 includes a communication port, the communication standard of the communication port may be selected arbitrarily. In this embodiment, the information processing device 1 may be connected to an external device (e.g., a sensor S, a robot device R, an external computer, etc.) via the external interface 13.

The input device 14 is a device for inputting information, such as a mouse or keyboard. The output device 15 is a device for outputting information, such as a display or speaker. An operator can operate the information processing device 1 using the input device 14 and output device 15. The input device 14 and output device 15 may be connected via the external interface 13. The input device 14 and output device 15 may also be integrated into one device, such as a touch panel display.

The drive 16 is a device for reading various information, such as programs, stored in the storage medium 91. At least one of the program 81 and the module data 300 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as stored programs) electrically, magnetically, optically, mechanically, or chemically so that a computer or other machine can read the information. The information processing device 1 may acquire at least one of the program 81 and the module data 300 from the storage medium 91. The storage medium 91 may be a disk-type storage medium, such as a CD or DVD, or a non-disk-type storage medium, such as a semiconductor memory (e.g., flash memory). The type of drive 16 may be selected appropriately depending on the type of storage medium 91. The drive 16 may be connected via the external interface 13.

Note that, with regard to the specific hardware configuration of the information processing device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The information processing device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be identical. Furthermore, the information processing device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a terminal device, or the like. When an external computer is used, the hardware configuration of the external computer may be the same as that of the information processing device 1. The external computer may be an information processing device designed specifically for the service to be provided, or may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, or the like.

[Software configuration]
7 schematically illustrates an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads a program 81 stored in the storage unit 12 into RAM and executes instructions included in the program 81 using the CPU. As a result, the information processing device 1 operates as a computer including an acquisition unit 111, a generation unit 112, and an output unit 113 as software modules. That is, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

The acquisition unit 111 is configured to acquire observation data 20 of the environment in which the robot device R operates, and instruction information 21 regarding the goal of the task to be given to the robot device R. In one example, the acquisition unit 111 may further acquire environmental information 22 regarding the environment. As the information processing device 1 holds module data 300, the generation unit 112 is provided with an inference module 3. The generation unit 112 is configured to generate a task problem description 5 from the acquired observation data 20 and instruction information 21 using the inference module 3. In one example, generating the task problem description 5 from the observation data 20 and instruction information 21 may be configured by generating the task problem description 5 from the observation data 20, instruction information 21, and environmental information 22. The output unit 113 is configured to output the generated problem description 5.

In this embodiment, an example is described in which each software module of the information processing device 1 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may also be implemented as a hardware module. With regard to the software configuration of the information processing device 1, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

§3 Operational Example Figure 8 is a flowchart showing an example of the processing procedure of the information processing device 1 according to this embodiment. The following processing procedure is an example of an information processing method executed by a computer. However, the following processing procedure of the information processing device 1 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the acquisition unit 111 and acquires observation data 20 and instruction information 21. In one example, the observation data 20 may be composed of sensing data from one or more sensors S, and the instruction information 21 may be composed of linguistic information that instructs a target in natural language. In another example, the control unit 11 may further acquire environment information 22. The environment information 22 may include at least one of a domain description 23 and domain information 24. The domain information 24 may include attribute information 241. If the inference module 3 includes a trained model 39 for in-context learning, the domain information 24 may include input/output samples 243. After acquiring the observation data 20 and the instruction information 21, the control unit 11 proceeds to the next step S102.

(Step S102)
In step S102, the control unit 11 operates as the generation unit 112, and uses the inference module 3 to generate a problem description 5 of the task from the acquired observation data 20 and instruction information 21. In one example, if environmental information 22 is further acquired, the control unit 11 may use the inference module 3 to generate the problem description 5 from the observation data 20, the instruction information 21, and the environmental information 22.

The generated problem description 5 includes descriptions (51, 52) of the initial and goal states of one or more objects present in the environment. In one example, the generated problem description 5 may follow a predetermined format. The generated problem description 5 may also include descriptions 50 of objects present in the environment.

In one example, generating the problem statement 5 using the inference module 3 may include at least one of generating an object statement 50 using the object estimator 31, generating an initial state statement 51 using the initial state estimator 33, and generating a goal state statement 52 using the goal estimator 35. If the inference module 3 includes a trained model 39 for in-context learning and the domain information 24 includes input/output samples 243, the control unit 11 may use the input/output samples 243 as few-shot prompting to adapt the trained model 39 to the domain for generating the problem statement 5. After generating the problem statement 5, the control unit 11 proceeds to the next step S103.

(Step S103)
In step S103, the control unit 11 operates as the output unit 113 and outputs the generated problem description 5. In one example, the control unit 11 may output the problem description 5 to at least one of the memory resources of the information processing device 1, the output device 15, and an external computer. In another example, the control unit 11 may provide the problem description 5 to a planner and execute at least a part of the process of generating an action plan by the planner as the process of step S103. When the information processing device 1 executes the calculation process of the planner, the control unit 11 may appropriately acquire information about the planner. For example, the information about the planner may be stored in at least one of the memory resources (e.g., RAM, storage unit 12) of the information processing device 1 and the external computer, and the control unit 11 may acquire the information about the planner from either of them. When the process of generating an action plan by the planner is executed by the external computer, the control unit 11 may provide the generated problem description 5 to the external computer as the process of step S103. In another example, when a behavior plan is generated by the information processing device 1 or an external computer, the control unit 11 may control the operation of the robot device R in accordance with the generated behavior plan as the processing of step S103. Controlling the operation of the robot device R may include directly controlling the robot device R and indirectly controlling the robot device R by giving instructions to a controller of the robot device R. Note that at least one of the generation of the behavior plan by the planner and the control of the robot device R may be executed separately from the processing of step S103. The control of the robot device R may be omitted. When the control of the robot device R is omitted, the generation of the behavior plan by the planner may also be omitted. When the output of the problem description 5 is completed, the control unit 11 ends the processing procedure of the information processing device 1 according to this operation example.

In one example, when the planner outputs an error message 615 during the process of generating an action plan using the planner, the control unit 11 may execute a process for modifying the problem statement 5. For example, as shown in FIG. 3, when the inference module 3 includes a trained model 39 for in-context learning, the control unit 11 may acquire the output error message 615. The control unit 11 may then provide the acquired error message 615 and problem statement 5 to the inference module 3 and execute the calculation process of the inference module 3 again. During this re-prompt, the control unit 11 may further provide at least a portion of the input data 200 to the inference module 3. As a result, the control unit 11 may generate a new, modified problem statement 5. The control unit 11 may recursively execute the process of modifying the problem statement 5 using this re-prompt.

[Features]
In this embodiment, the problem description 5 generated in step S102 includes descriptions (51, 52) of the initial and goal states of one or more objects in the environment, so that the planner can generate an action plan that reaches a goal state from the initial state of the task. The initial state description 51 describes the initial state of each of the one or more objects in the environment. Meanwhile, the goal state description 52 describes the goal state of each of the one or more objects that is reached by completing the task. In other words, each description (51, 52) indicates the state before and after the task is performed and is human-interpretable. Therefore, the problem description 5 has high interpretability. Therefore, according to this embodiment, a highly interpretable output can be obtained to obtain an action sequence (control command) to be given to the robot device R.

§4 Modifications Although the embodiments of the present invention have been described above in detail, the above description is merely an example of the present invention in every respect. Various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, the sensor S may be attached to or include a driving device, thereby enabling the observation direction to be changed. For example, if the sensor S includes a camera, the camera may be attached to an electric pan head, thereby enabling the shooting direction to be changed. The electric pan head is an example of a driving device. The information processing device 1 may be configured to be able to directly or indirectly drive the driving device. In this way, the information processing device 1 may appropriately change the observation direction of the sensor S.

In addition, the environmental information 22 may include a designation list of objects to be observed. In step S101, the control unit 11 of the information processing device 1 may acquire the environmental information 22, including the designation list. The attribute information 241 may also serve as the object designation list. In step S102, the control unit 11 may determine whether all objects included in the designation list have been detected from the observation data 20. In the example of FIG. 4, the control unit 11 may determine whether all objects included in the designation list have been detected from the observation data 20 during the calculation process of the object estimator 31 or the initial state estimator 33. In one example, when an object detector, such as the trained model 311 of FIG. 5A, is provided, the control unit 11 may determine whether all objects included in the designation list have been detected based on the object detection results of the detector for the observation data 20. In another example, the control unit 11 may determine whether all objects included in the designation list have been detected based on whether all objects included in the designation list are included in at least one of the object description 50 and the initial state description 51. If at least some of the objects included in the designation list are not detected in the observation data 20, the control unit 11 may drive the drive device of the sensor S and appropriately change the observation direction of the sensor S. The direction and amount of change may be determined by any method, for example, randomly, according to a predetermined rule, or the like. The control unit 11 may then reacquire the observation data 20 and perform the process of step S102 again. The control unit 11 may repeat these processes until all of the objects included in the designation list are detected. On the other hand, if all of the objects included in the designation list are detected in the observation data 20, the control unit 11 may proceed with the process of generating the problem description 5. According to this modification, the observation direction of the sensor S is corrected in an appropriate direction, and the accuracy of generating the problem description 5 can be expected to improve.

The above-described process for confirming detected objects may be modified as appropriate depending on the embodiment. For example, the designation list may further include objects that may or may not be observed (recommended objects).
If only the recommended object is not detected, the control unit 11 may output an alert and then proceed with the process of generating the problem description 5. Alternatively, the control unit 11 may stop the process of generating the problem description 5 and ask the operator whether or not to continue the process of generating the problem description 5.

<4.2>
The domain description 23 may include a portion dependent on the robot device R (e.g., a description defining the skills of the robot device R) and a portion related to the environment (e.g., a description defining the type of target object). Therefore, similar to the method for generating the problem description 5 in the above embodiment, the information processing device 1 may use a calculation module to generate at least a portion of the portion related to the environment of the domain description 23 from at least a portion of the observation data 20, the instruction information 21, and the environmental information 22. The calculation module may be configured with at least one of a trained model and a rule-based model. This modification is expected to reduce the effort required to prepare the domain description 23.

<4.3>
4, the inference module 3 is configured to individually generate descriptions (50, 51, 52) of the objects, initial states, and goal states in the problem statement 5. However, the configuration of the inference module 3 is not limited to this example. In another example, the inference module 3 may be configured to integrally generate at least any combination of descriptions (50, 51, 52) of the objects, initial states, and goal states in the problem statement 5.

9 is a schematic diagram showing an example of the configuration of an inference module 3A according to another embodiment. The inference module 3A includes an object detector 371, a caption model 373, and a large-scale language model 375. In the example shown in FIG. 9, the observation data 20 may be composed of image data, and the instruction information 21 may be composed of text data instructing a target in natural language. The object detector 371 includes an open vocabulary object model (OVO) proposed in the above-mentioned references 11 and 12.
For example, a detection model or an open vocabulary object segmentation model may be used. For the caption model 373, a visual question answering model proposed in the above-mentioned Reference 13 may be used. For the large-scale language model 375, a model proposed in the above-mentioned Reference 14 may be used.

First, the control unit 11 of the information processing device 1 may use the object detector 371 to detect objects from the observation data 20. The control unit 11 may extract partial data (partial image data) of each object from the observation data 20 according to the object detection results by the object detector 371, and provide the extracted partial data to the caption model 373. The control unit 11 may use the caption model 373 to generate a caption for each object from the extracted partial data of each object. Then, the control unit 11 may use the large-scale language model 375 to generate each description (50, 51, 52) of the problem description 5 from the text data of the object detection results by the object detector 371, the text data of the caption by the caption model 373, and the instruction information 21. This variant example simplifies the calculation process.

<4.4>
In the above embodiment, the re-prompting method in Fig. 3 may be appropriately improved depending on the embodiment. As an example, the re-prompting method in Fig. 3 may be based on the method described in Reference 15 (Jason Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", [online], [Retrieved October 24, 2023], Internet <URL: https://arxiv.org/pdf/2201.11903.pdf>), Reference 16 (Takeshi Kojima et al., "Large Language Models are Zero-Shot Reasoners", [online], [Retrieved October 24, 2023], Internet <URL: https://arxiv.org/abs/2205.11916>), Reference 17 (Denny Zh
ou et al., “Least-to-Most Prompting Enables Complex Reasoning in Large Language
Chain-of-Thought prompts such as those proposed in "Arxiv: PDDL Models," [online], [searched October 24, 2023], Internet <URL: https://arxiv.org/abs/2205.10625>" may be used. Chain-of-Thought prompts are a method of performing intermediate inference steps and then layering step-by-step inferences to obtain a final inference result (in this embodiment, the result of generating a problem description 5). For example, the control unit 11 of the information processing device 1 may provide the inference module 3 with template data that asks a question about the error, such as "What part of the PDDL problem do you think is causing this error?", along with the problem description 5 and the error message 615, to cause the inference module 3 to generate an explanation for the error. The control unit 11 may then provide the problem description 5 and the error message 615 to the inference module 3, and cause the inference module 3 to generate a new problem description 5.

§5 Experimental Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

(inference module)
The inference module configuration shown in FIG. 4 was used for the first example. The object detector configuration was the configuration shown in FIG. 5A. The trained model for the object detector was the model proposed in Reference 11. The initial state estimator configuration was the configuration shown in FIG. 5B. The detector for the initial state estimator was the model proposed in Reference 13. The trained model for the initial state estimator was the model proposed in Reference 14. The target estimator configuration was the configuration shown in FIG. 5C. The trained model for the target estimator was the model proposed in Reference 14.

Image data was used as the observation data. Text data specifying the target in natural language was used as the instruction information. The trained model of the object estimator was given attribute information indicating the object's attributes in linguistic expressions along with the observation data (image data). The trained model of the initial state estimator was given text data indicating the object detection results, text data of captions by the detector, a description of the object generated by the object estimator, and input/output samples used for few-shot prompting. The trained model of the target estimator was given instruction information (text data), a description of the object generated by the object estimator, a description of the initial state generated by the initial state estimator, and input/output samples used for few-shot prompting. The number of input/output samples was three.

The inference module configuration shown in Figure 9 was used for the second example. The object detector of the inference module was the model proposed in Reference 11. The caption model was the model proposed in Reference 13. The large-scale language model was the model proposed in Reference 14. The large-scale language model was provided with instruction information (text data), text data of the detection results by the object detector, text data of captions by the caption model, and input/output samples used for few-shot prompting.

The problem description format was PDDL. A symbolic planner was prepared to verify whether an action plan could be generated using the generated problem description. The environment for running the symbolic planner was Fast Downward and VAL (Reference 18: "KCL-Planning/VAL", [online], [searched October 24, 2023], Internet URL: https://github.com/KCL-Planning/VAL). In the inference modules of the first and second examples, a reprompting correction mechanism (Figure 3) was implemented. The reprompting consisted of the input data, the generated problem description, and an error message. The reprompting method used was the Chain-of-Thought prompt described above. The maximum number of reprompting attempts to correct the problem description was set to two.

(Dataset)
Figure 10 shows the types (object types), predicates, and actions of the domain description for each domain prepared. As shown in Figure 10, datasets were prepared for three domains: Cooking, Blocksworld, and Tower of Hanoi. The cooking task was assumed to be slicing vegetables and putting them into a bowl. Two robot arms, positioned on the left and right, were assumed as the robot device R. The state and location of the vegetables were specified as the target state.

Block-world is a classic domain used in references such as Reference 19 (Naresh Gupta et al., "On the Complexity of Blocks-World Planning", [online], [searched October 24, 2023], Internet <URL: https://www.semanticscholar.org/paper/On-the-Complexity-of-Blocks-World-Planning-Gupta-Nau/db01349fd0d29c9443e37b0b0aa4ddb948ace5ce>). Seven unique colored blocks were used for each problem. The robot arm was initially configured so that it did not always grasp something. The goal state specified the relationship between the blocks.

Hanoi is based on the reference 20 (Ron Alford et al., “Translating HTNs to PDDL: A Small
This is a classic domain used in places such as "The Amount of Domain Knowledge Can Go a Long Way," [online], [searched October 24, 2023], Internet (URL: https://www.cs.umd.edu/~nau/papers/alford2009translating.pdf). Three pegs and ten disks of six colors were set up. Disks of the same color were numbered in descending order of width. The three pegs were numbered from left to right. The initial state and goal state were specified by the positions of the disks.

Figures 11A, 11B, and 11C show examples of observation data and instruction information given in the Cooking, Block World, and Hanoi domains. One domain description was prepared for each domain. Sample correct answer problem descriptions were also prepared for each problem. In Hanoi, the same instruction information was used for all problems.

(Evaluation indicators)
To evaluate whether a problem description was generated appropriately, four indices (R _syntax , R _plan , R _part , and R _all ) were prepared. R _syntax was defined as the proportion of the generated problem description that conformed to the PDDL syntax. If VAL returned no warnings or exit codes, the generated problem description was considered syntactically correct. R _plan was defined as the proportion of cases in which the symbolic planner did not output an error message when generating an action plan from the generated problem description. An action plan was attempted to be generated using the symbolic planner, and if VAL returned no error messages, the action plan was considered to have been generated validly. R _part and R _all were defined as the proportion of the generated problem description that included all descriptions of the correct sample to evaluate reproducibility. R _part was calculated individually for the descriptions of the objects, initial states, and goal states. _{R all} was calculated for the entire generated problem description.

(Experimental Example)
In the first experimental example, problem descriptions were generated for each of three domains using the inference module of the first embodiment. The system was set up to allow up to two re-prompts to modify the problem descriptions. 100 problems were prepared for each domain. Ten problem descriptions were generated for each problem by varying the combination of input and output samples. The four evaluation indices (R _syntax , R _plan , R _part , and R _all ) were calculated for the generated problem descriptions.

In the second experimental example, problem descriptions were generated in each of the three domains using the inference module of the second embodiment. Three evaluation indices (R _syntax , R _plan , and R _all ) were calculated for the generated problem descriptions. All other conditions were set to be the same as in the first experimental example.

In the third experimental example, using the reasoning module of the first embodiment, problem statements were generated for each of three domains under four conditions: (A) with revision reprompts (CR) and Chain-of-Thought (CoT) prompts (up to two revisions), (B) with revision reprompts (CR) and Chain-of-Thought (CoT) prompts (up to one revision), (C) with revision reprompts (CR) and no Chain-of-Thought (CoT) prompts (up to one revision), and (D) without revision reprompts (CR) and Chain-of-Thought (CoT) prompts. Three evaluation metrics (R _syntax , R _plan , and R _all ) were calculated for the generated problem statements. The other conditions were the same as those in the first experimental example.

(Experimental results)
12, 13, and 14 show the calculation results of each evaluation index in Experimental Example 1, Experimental Example 2, and Experimental Example 3. The values in parentheses in Fig. 13 indicate the difference from the result of Experimental Example 1.

As shown in Figure 12, in the Hanoi domain of the first experimental example, although R _plan was slightly low, the scores for R _syntax and R _plan were high in each domain. In other words, syntactically correct problem descriptions were generated. Furthermore, while the R _all score was low in the Block World and Hanoi domains, the scores for both R _part and R _all were high in the Cooking domain. Therefore, it was found that high reproducibility can be achieved in generating problem descriptions. These results demonstrate that effective problem descriptions can be generated according to this embodiment.

Furthermore, as shown in Figure 13, the inference module of the second embodiment was able to generate syntactically correct problem descriptions equivalent to those of the inference module of the first embodiment. Furthermore, although the R _all score decreased in the cooking and block world domains, the R _all score increased in the Hanoi domain. These results demonstrate that effective problem descriptions can be generated even when the integrated configuration shown in Figure 9 is adopted as the inference module configuration.

Furthermore, as shown in Figure 14, the scores for R _syntax , R _plan , and R _all increased as a result of repeated use of correction reprompts (CR) and Chain-of-Thought (CoT) prompts. These results indicate that correction reprompts (CR) and Chain-of-Thought (CoT) prompts are effective in obtaining appropriate problem descriptions.

This specification includes the following disclosure.
[Appendix 1]
Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task from the acquired observation data and the instruction information using an inference module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description.
a control unit configured to perform
Information processing device.
[Appendix 2]
the problem description generated follows a predetermined format;
2. The information processing device according to claim 1.
[Appendix 3]
the generated problem description further includes a description of the objects present in the environment;
3. The information processing device according to claim 1 or 2.
[Appendix 4]
the observation data is composed of sensing data from a sensor;
the instruction information is composed of linguistic information that indicates the target in natural language;
4. An information processing device according to any one of claims 1 to 3.
[Appendix 5]
the inference module is configured to include a trained model for in-context learning;
The control unit provides the generated problem description to a planner, and when the planner outputs an error message during a process of generating a behavior plan for the robot device,
obtaining the output error message; and generating a new problem description from the problem description and the error message using the reasoning module.
and further configured to perform
5. An information processing device according to any one of claims 1 to 4.
[Appendix 6]
In the acquiring step, the control unit is configured to further acquire environmental information related to the environment;
generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information;
6. An information processing device according to any one of claims 1 to 5.
[Appendix 7]
the generated problem description further comprises a description of the objects present in the environment;
the inference module includes an object estimator;
generating the problem statement using the reasoning module includes generating descriptions of the objects present in the environment from the acquired observation data using the object estimator;
7. An information processing device according to any one of claims 1 to 6.
[Appendix 8]
the object estimator comprises a trained model for in-context learning;
8. The information processing device according to claim 7.
[Appendix 9]
In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment;
generating a description of the object from the acquired observation data comprises generating a description of the object from the acquired observation data and the attribute information;
9. The information processing device according to claim 7 or 8.
[Supplementary Note 10]
the inference module includes an initial state estimator;
generating the problem statement using the reasoning module includes generating a description of the initial states of the objects in the environment using the initial state estimator.
10. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 9.
[Appendix 11]
the initial state estimator comprises a trained model for in-context learning;
11. The information processing device according to claim 10.
[Appendix 12]
the inference module includes a target estimator;
generating the problem statement using the reasoning module includes generating a description of the goal state of the objects in the environment using the goal estimator.
12. The information processing device according to claim 1.
[Appendix 13]
the goal estimator comprises a trained model for in-context learning;
13. The information processing device according to claim 12.
[Appendix 14]
The computer
Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task from the acquired observation data and the instruction information using an inference module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description.
To execute
Information processing methods.
[Appendix 15]
On the computer,
Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task from the acquired observation data and the instruction information using an inference module, the problem description including descriptions of initial and goal states of objects in the environment; and outputting the generated problem description.
In order to execute
program.

1...information processing device,
11...control unit, 12...storage unit, 13...external interface,
14...input device, 15...output device, 16...drive,
81... program, 91... storage medium,
111... Acquisition unit, 112... Generation unit, 113... Output unit,
3...inference module, 300...module data,
31... object estimator, 33... initial state estimator,
35...Target estimator,
20... Observation data, 21... Instruction information,
22...Environmental information,
23...Domain description, 24...Domain information,
241...Attribute information,
5...Problem description,
50...Description of object, 51...Description of initial state,
52...Description of target state

Claims (20)

Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task using a reasoning module , the reasoning module including a goal estimator, the problem description including a description of a goal state of an object in the environment generated from the acquired observation data and the instruction information using the goal estimator; and outputting the generated problem description.
a control unit configured to perform
Information processing device. the step of outputting the problem description includes outputting the problem description, including a description of the goal state, to a planner that generates an operation plan for the task of the robotic device.
The information processing device according to claim 1 . the problem description generated follows a predetermined format;
The information processing device according to claim 1 . the generated problem description further includes a description of the objects present in the environment;
The information processing device according to claim 1 . the observation data is composed of sensing data from a sensor;
the instruction information is composed of linguistic information that indicates the target in natural language;
The information processing device according to claim 1 . the inference module is configured to include a trained model for in-context learning;
the control unit obtaining an error message for the generated problem description; and using the inference module to generate a new problem description from the problem description and the error message.
and further configured to perform
The information processing device according to claim 1 . In the acquiring step, the control unit is configured to further acquire environmental information related to the environment;
generating the problem description from the acquired observation data and the instruction information comprises generating the problem description from the acquired observation data, the instruction information, and the environment information;
The information processing device according to claim 1 . the generated problem description further comprises a description of the objects present in the environment;
the inference module includes an object estimator;
generating the problem statement using the reasoning module includes generating descriptions of the objects present in the environment from the acquired observation data using the object estimator;
The information processing device according to claim 1 . the object estimator comprises a trained model for in-context learning;
The information processing device according to claim 8 . In the acquiring step, the control unit is configured to further acquire attribute information of the object present in the environment;
generating a description of the object from the acquired observation data comprises generating a description of the object from the acquired observation data and the attribute information;
The information processing device according to claim 8 or 9 . the inference module includes an initial state estimator;
generating the problem statement using the reasoning module includes generating a description of an initial state of the objects in the environment using the initial state estimator.
The information processing device according to claim 1 . the initial state estimator comprises a trained model for in-context learning;
The information processing device according to claim 11 . the goal estimator comprises a trained model for in-context learning;
The information processing device according to claim 1 . The computer
Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task using a reasoning module , the reasoning module including a goal estimator, the problem description including a description of a goal state of an object in the environment generated from the acquired observation data and the instruction information using the goal estimator; and outputting the generated problem description.
To execute
Information processing methods. On the computer,
Obtaining observation data of an environment in which the robotic device operates and instruction information regarding a goal of a task to be given to the robotic device;
generating a problem description for the task using a reasoning module , the reasoning module including a goal estimator, the problem description including a description of a goal state of an object in the environment generated from the acquired observation data and the instruction information using the goal estimator; and outputting the generated problem description.
In order to execute
program. the instruction information in the natural language includes at least one of text input, voice input, and image input ;
The information processing device according to claim 5 . the sensing data includes at least one of image data, depth data, infrared data, measurement data of an optical sensor, radar data, LiDAR data, sound data, position data, measurement data of a robot device, and data obtained by communication;
The measurement data of the robot device includes at least one of joint angles, hand positions, tactile data at the hand, force data at the hand, and posture measurement data.
The information processing device according to claim 5 . the description of the target state includes position information of the target of the object;
The information processing device according to claim 1 . the problem statement further includes an initial state of objects in the environment;
The information processing device according to claim 1 . the error message is an error message output by a planner in a process of generating a behavior plan for the robot device by providing the generated problem description to the planner.
The information processing device according to claim 6 .