CN120941397B - Mechanical arm control method, device, equipment and medium based on meta reinforcement learning - Google Patents
Mechanical arm control method, device, equipment and medium based on meta reinforcement learningInfo
- Publication number
- CN120941397B CN120941397B CN202511226596.1A CN202511226596A CN120941397B CN 120941397 B CN120941397 B CN 120941397B CN 202511226596 A CN202511226596 A CN 202511226596A CN 120941397 B CN120941397 B CN 120941397B
- Authority
- CN
- China
- Prior art keywords
- target
- action
- value
- mechanical arm
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Program-controlled manipulators
- B25J9/16—Program controls
- B25J9/1628—Program controls characterised by the control loop
- B25J9/163—Program controls characterised by the control loop learning, adaptive, model based, rule based expert control
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Manipulator (AREA)
- Feedback Control In General (AREA)
Abstract
本申请实施例提供了一种基于元强化学习的机械臂控制方法和装置、设备及介质，属于机械臂控制技术领域。该方法包括：获取机械臂控制任务的历史交互数据；基于历史交互数据，对机械臂控制任务进行特征提取，得到任务特征；基于所述任务特征，通过预设策略网络确定目标机械臂动作；通过预设优势函数对所述目标机械臂动作计算优势值，得到目标动作优势；对目标动作优势进行归一化处理，得到目标动作权重；基于目标动作权重，对预设策略网络进行参数更新，得到目标策略网络；利用目标策略网络进行机械臂控制。本申请实施例能够提高机械臂控制方法的泛化能力。
This application provides a robotic arm control method, apparatus, device, and medium based on meta-reinforcement learning, belonging to the field of robotic arm control technology. The method includes: acquiring historical interaction data of a robotic arm control task; extracting features from the historical interaction data to obtain task features; determining a target robotic arm action based on the task features using a preset policy network; calculating an advantage value for the target robotic arm action using a preset advantage function to obtain a target action advantage; normalizing the target action advantage to obtain a target action weight; updating the parameters of the preset policy network based on the target action weight to obtain a target policy network; and using the target policy network for robotic arm control. This application embodiment can improve the generalization ability of robotic arm control methods.
Description
Technical Field
The application relates to the technical field of mechanical arm control, in particular to a mechanical arm control method, device, equipment and medium based on meta reinforcement learning.
Background
The mechanical arm is used as a core executing mechanism of an industrial automation, medical operation and service robot, and the motion control of the mechanical arm needs to meet the real-time response requirements under high precision, high adaptability and dynamic environment. However, the current conventional control methods (such as PID control, adaptive control, etc.) depend on accurate mathematical models, but such mathematical models have low generalization ability when facing complex dynamic tasks, and are difficult to adapt to the diversity of tasks. Therefore, how to improve the generalization capability of the mechanical arm control method becomes a technical problem to be solved.
Disclosure of Invention
The embodiment of the application mainly aims to provide a mechanical arm control method, device, equipment and medium based on meta reinforcement learning, aiming at improving the generalization capability of the mechanical arm control method.
To achieve the above object, a first aspect of an embodiment of the present application provides a method for controlling a mechanical arm based on meta reinforcement learning, the method including:
Acquiring historical interaction data of a control task of the mechanical arm;
based on historical interaction data, performing feature extraction on the mechanical arm control task to obtain task features;
determining the action of a target mechanical arm through a preset strategy network based on the task characteristics;
Calculating a dominance value for the target mechanical arm action through a preset dominance function to obtain a target action dominance;
normalizing the target action advantages to obtain target action weights;
based on the target action weight, carrying out parameter updating on a preset strategy network to obtain a target strategy network;
and controlling the mechanical arm by using the target strategy network.
To achieve the above object, a second aspect of the embodiments of the present application provides a mechanical arm control device based on meta reinforcement learning, the device including:
The interaction data acquisition module is used for acquiring historical interaction data of the control task of the mechanical arm;
the task feature extraction module is used for carrying out feature extraction on the control task of the mechanical arm based on the historical interaction data to obtain task features;
the action and state determining module is used for determining the action of the target mechanical arm through a preset strategy network based on the task characteristics;
the target action advantage calculation module is used for calculating an advantage value of the target mechanical arm action through a preset advantage function to obtain a target action advantage;
the normalization processing module is used for carrying out normalization processing on the target action advantages to obtain target action weights;
the parameter updating module is used for updating parameters of the preset strategy network based on the target action weight to obtain a target strategy network;
And the mechanical arm control module is used for controlling the mechanical arm by utilizing the target strategy network.
To achieve the above object, a third aspect of the embodiments of the present application proposes an electronic device, including a memory storing a computer program and a processor implementing the method according to the first aspect when the processor executes the computer program.
To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method of the first aspect.
The mechanical arm control method, the mechanical arm control device, the mechanical arm control equipment and the mechanical arm control medium based on the meta reinforcement learning can generate task characteristics representing task characteristics by extracting characteristics of historical interaction data, guide a strategy network to select actions through the task characteristics, improve adaptability of the strategy network to different tasks, calculate target action advantages through a preset dominance function, quantify the good and bad degree of the target mechanical arm action, normalize the target action advantages, convert the target action advantages into weights, update parameters of the preset strategy network based on the target action weights, ensure that the high dominant actions obtain high weights to dominate training directions, enable the strategy network to be fast adapted to unknown tasks only by a small number of samples, and improve generalization capability of the mechanical arm control method.
Drawings
FIG. 1 is a flowchart of a method for controlling a mechanical arm based on meta reinforcement learning according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a process of meta reinforcement learning;
FIG. 3 is another flowchart of a method for controlling a mechanical arm based on meta reinforcement learning according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a mechanical arm control device based on meta reinforcement learning according to an embodiment of the present application;
Fig. 5 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
It should be noted that although functional block division is performed in a device diagram and a logic sequence is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.
First, several nouns involved in the present application are parsed:
meta reinforcement learning (Meta reinforcement learning, meta-RL) Meta reinforcement learning is a machine learning method aimed at training an agent (e.g., AI agent) to quickly adapt to new tasks, by learning generic learning strategies over multiple tasks, to make efficient decisions in unseen environments. Meta-reinforcement learning, which is a sub-field of artificial intelligence, is a interdisciplinary of reinforcement learning and Meta-learning, and is also commonly referred to as Meta-reinforcement learning or Meta-RL. Meta reinforcement learning includes meta policy optimization, model independent meta learning (MAML), and gradient-based adaptation algorithms, etc. The method is commonly used in the technical fields of robot control, game agent development, automatic driving system, resource management, personalized recommendation and the like.
Meta training META TRAINING meta training is a training phase in a meta learning framework that involves optimizing a meta model (e.g., a neural network) over multiple related tasks to learn shared knowledge among the tasks so that the model can be generalized to new tasks. Meta training is part of the machine learning process, and is a core link of meta learning algorithm design and implementation, and is often referred to as a meta optimization stage. The meta training comprises the steps of task sampling, meta loss function design, internal circulation training, external circulation updating and the like.
Meta-test META TESTING the meta-test is an evaluation stage in the meta-learning framework for testing the generalization ability and adaptation efficiency of the meta-training model on new tasks, verifying the performance of the model by simulating an unknown environment. Meta-testing is part of the machine learning process, and is a key link of meta-learning verification and deployment, and is often referred to as a meta-evaluation stage. Meta-tests include task fitness tests, performance metrics (such as accuracy or rewards score), and robustness analysis, among others.
In the related art, referring to fig. 2, in offline meta-reinforcement learning, since tasks used in a meta-training stage and a meta-testing stage are different, tasks used in the meta-training stage are distributed by tasks constructed based on historical data or a predefined task library, and a meta-testing stage is to process new tasks which are not seen at all, tasks in the meta-testing cannot appear in the meta-training tasks, and there is a problem of distribution deviation. The distribution offset problem can cause the generalization performance of the intelligent agent to be insufficient, so that the intelligent agent cannot finish the effective fine adjustment through a small amount of samples, and the generalization capability is poor.
Based on the above, the embodiment of the application provides a mechanical arm control method, a device, equipment and a medium based on meta reinforcement learning, aiming at improving the generalization capability of the mechanical arm control method.
The embodiment of the application provides a mechanical arm control method and device based on meta reinforcement learning, electronic equipment and storage medium, and specifically describes the mechanical arm control method based on meta reinforcement learning in the embodiment of the application through the following embodiment.
The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Wherein artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) is the theory, method, technique, and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain optimal results.
Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.
The embodiment of the application provides a mechanical arm control method based on meta reinforcement learning, and relates to the technical field of mechanical arm control. The mechanical arm control method based on meta reinforcement learning provided by the embodiment of the application can be applied to a terminal, a server side and software running in the terminal or the server side. In some embodiments, the terminal may be a notebook computer, a desktop computer, etc., the server may be configured as an independent physical server, may be configured as a server cluster or a distributed system formed by a plurality of physical servers, and may be configured as a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligent platforms, etc., and the software may be an application for implementing a mechanical arm control method based on meta reinforcement learning, etc., but is not limited to the above form.
The application is operational with numerous general purpose or special purpose computer system environments or configurations. Such as a personal computer, a server computer, a hand-held or portable device, a multiprocessor system, a microprocessor-based system, a programmable consumer electronics, a network PC, a minicomputer, a mainframe computer, a distributed computing environment that includes any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
It should be noted that, in each specific embodiment of the present application, when related processing is required according to user information, user behavior data, user history data, user location information, and other data related to user identity or characteristics, permission or consent of the user is obtained first, and the collection, use, processing, and the like of the data comply with related laws and regulations and standards. In addition, when the embodiment of the application needs to acquire the sensitive personal information of the user, the independent permission or independent consent of the user is acquired through popup or jump to a confirmation page and the like, and after the independent permission or independent consent of the user is definitely acquired, the necessary relevant data of the user for enabling the embodiment of the application to normally operate is acquired.
Fig. 1 is an optional flowchart of a method for controlling a mechanical arm based on meta reinforcement learning according to an embodiment of the present application, where the method in fig. 1 may include, but is not limited to, steps S101 to S107.
Step S101, historical interaction data of a mechanical arm control task is obtained;
Step S102, carrying out feature extraction on a mechanical arm control task based on historical interaction data to obtain task features;
Step S103, determining the action of the target mechanical arm through a preset strategy network based on the task characteristics;
Step S104, calculating a dominance value for the target mechanical arm action through a preset dominance function to obtain a target action dominance;
Step S105, carrying out normalization processing on the target action advantages to obtain target action weights;
step S106, based on the target action weight, carrying out parameter updating on the preset strategy network to obtain a target strategy network;
Step S107, performing mechanical arm control by utilizing a target strategy network.
As will be readily appreciated, the historical interaction data includes trajectory data generated by an agent interacting with the environment in a multitasking scenario, such as an action-state-rewards sequence. The target mechanical arm action can comprise action probability distribution (namely probability values of different actions are selected) output by the strategy network, and the quality degree of each action is quantified through a preset dominance function. And carrying out normalization processing on the target action advantages to obtain target action weights. And updating parameters of the preset strategy network based on the target action weight to obtain the target strategy network. And in the meta-test stage, inputting the environmental state and the task characteristics of the new task to a target strategy network, and enabling the next mechanical arm of the target strategy network to act so as to finish tasks such as grabbing, stacking and the like.
The method comprises the steps S101 to S107 shown in the embodiment of the application, task characteristics representing task characteristics can be generated by extracting characteristics of historical interaction data, the strategy network is guided to select actions through the task characteristics, the adaptability of the strategy network to different tasks is improved, the target action advantage is calculated through a preset advantage function, the good and bad degree of the target mechanical arm action is quantized, the target action advantage is normalized and converted into weight, the parameters of the preset strategy network are updated based on the target action weight, the high-advantage action is ensured to obtain high weight to lead the training direction, the strategy network can be rapidly adapted to unknown tasks only by a small number of samples, and the generalization capability of the mechanical arm control method is improved.
The embodiment of the application can be applied to motion control of various mechanical arms and robots, such as industrial mechanical arms deployed on a production line, robots which need to execute complex tasks, robots deployed in space stations to complete related operations and other practical application scenes. Taking a robot for performing complex tasks as an example, the robot needs to perform a wide variety of tasks with complex procedures and high automaticity, such as picking up certain objects, unlocking a door lock, etc. By applying the method provided by the embodiment of the application, the interaction data between the robots with certain task types and quantity and the environment are collected for training, and the robot can well complete similar tasks through the target strategy network, so that the defects of the traditional method are effectively overcome, and the use cost of the robot is reduced.
In step S102 of some embodiments, the interaction data may be encoded by a preset inference network to make it follow gaussian posterior distribution, so as to obtain posterior distribution parameters z (such as mean and variance) of the mechanical arm control task as task features.
In some embodiments, the mechanical arm control method based on meta reinforcement learning further comprises setting a loss function of the inference network asWhere beta represents a regularization coefficient,Representing an inference network for outputting posterior distribution parameters z, p (z) =n (0,I) based on the input data (i.e., interaction data c). The embodiment of the application limits the information quantity of the potential posterior distribution parameter z through KL divergence, and essentially limits an inference networkThe difference between the output data and the a priori distribution p (z), which is usually assumed to be a normal distribution N (0,I), ifThe output data carries excessive information irrelevant to tasks, the KL value can be increased, and the distribution of z is forced to approach to simple prior distribution by minimizing KL divergence, so that the information quantity of the output data is limited, and the overfitting of the extracted potential posterior distribution parameter z caused by insufficient data is prevented.
In some embodiments, step S103 may include, but is not limited to including, step S301 to step S302:
Step S301, obtaining an initial environment state;
step S302, selecting actions through a preset strategy network according to the task characteristics and the initial environment state, and obtaining the actions of the target mechanical arm.
As will be readily appreciated, the initial environmental conditions may include robotic arm pose, robotic arm position, object position, and the like. The target mechanical arm action can be a motion parameter of each joint of the mechanical arm. After the target mechanical arm is executed, the environmental state can be updated based on the environmental feedback, and the target environmental state is obtained.
In some embodiments, step S104 may include, but is not limited to including, step S401 to step S403:
Step S401, calculating the action value of the target mechanical arm through a preset action value network to obtain a target action value;
Step S402, calculating the value of an initial environment state through a preset state value network to obtain the value of the initial state;
Step S403, calculating a dominance value for the motion of the target mechanical arm according to the target motion value and the initial state value by a preset dominance function, and obtaining the dominance of the target motion.
In step S403 of some embodiments, the difference between the target action value and the initial state value is taken as the target action advantage. Specifically, the dominance function is set to a (s, a) =q (s, a) -V(s), where Q (s, a) is the target action value output by the action value network (i.e., Q-value network), and V(s) is the initial state value output by the state value network (i.e., V-value network).
In some embodiments, the method for controlling a mechanical arm based on meta reinforcement learning further includes steps S501 to S506:
step S501, according to the action of the target mechanical arm and the initial environmental state, updating the environmental state to obtain the target environmental state;
step S502, calculating an average value of task features to obtain a task feature average value;
Step S503, calculating the value of the target environment state based on the task feature mean value to obtain a target state value;
Step S504, obtaining rewards of the action of the target mechanical arm;
step S505, calculating action value loss based on the target action value, rewards and target state value;
Step S506, based on the action value loss, the parameters of the action value network are updated.
In some embodiments, the method for controlling a mechanical arm based on meta reinforcement learning further includes steps S601 to S602:
step S601, calculating to obtain state value loss based on target action value and initial state value;
Step S602, based on the state value loss, updating parameters of the state value network.
In particular, referring to FIG. 3, trace data generated by an Agent (Agent) interacting with an Environment (Environment), such as a Markov decision tuple (action-state-rewards), is collected as the Agent performs different tasks. And modeling the Markov decision process by using a probability modeling method to obtain task distribution p (T), wherein T= { S, A, T and R }, S is a state space, A is an action space, T is a state transition probability and R is a reward function. Track data T i of the ith task is randomly sampled from the task profile p (T) and then stored in an empirical playback bufferAs historical interaction data. And extracting the characteristics of the historical interaction data through a transducer model to generate task characteristics z. Based on the task feature z and the current policy, the policy network (Actor) outputs an action probability distribution. The action value network (Critic) includes two Q value networks (Q1 and Q2), and outputs an action value (Q value) based on the task feature z and the action probability distribution. The state Value Network (Value Network) outputs a state Value (V Value) according to the state.
Setting a V-value loss function to calculate state value loss Where V (s, z) is the initial state value, Q θ (s, a, z) is the target action value, and E (s,a,r,s′) represents the desire to have a joint distribution of initial state, action, rewards, and target state.
Calculating action value loss by setting Q value loss function Wherein, the Representing a desire for a joint distribution of initial states, actions, rewards, and target states sampled from an empirical playback buffer, Q θ (s, a, z) being a target action value, r being a reward; For the value of the state of the goal, The method comprises the steps of sampling a characteristic average value of a plurality of tasks, s' respectively representing a current time state (namely an initial environment state) and a next time state (namely a target environment state), and a is a current time action (namely a target mechanical arm action). In the training process, Q θ is made to approach the optimal value function gradually with the aim of minimizing the action value loss.
According to the embodiment of the application, the environment state s and the potential posterior distribution parameter z are used for updating the V-value network parameter, and the Q-value network parameter is output and updated through the V-value network, so that the conservatism of Q-value network estimation is ensured, and the situations of performance degradation, error and the like caused by overestimation of action value can be avoided.
In step S105 of some embodiments, the target motion advantage may be normalized by softmax to obtain a target motion weight ω (S, a).
In step S106 of some embodiments, the target action weight may be multiplied by the action probability distribution output by the policy network to obtain a weighted action probability distribution, and the policy loss weighting value is calculated based on the weighted action probability distribution, and the target policy network is obtained by minimizing the policy loss weighting value through gradient descent.
In some embodiments, step S106 may include, but is not limited to including, step S701 through step S705:
Step S701, calculating a weighted value of probability distribution of the motion of the target mechanical arm according to the target motion weight to obtain a first distribution;
step S702, calculating the action value of the action of the target mechanical arm to obtain the target action value;
Step S703, calculating probability distribution of the target action value to obtain second distribution;
step S704, calculating the difference between the first distribution and the second distribution to obtain policy loss;
Step S705, based on the strategy loss, carrying out parameter updating on the strategy network to obtain the target strategy network.
As will be readily appreciated, the first distribution represents a probability distribution that the policy network takes different actions for a particular state and task feature under the current policy parameters. The second distribution represents probability distribution of different action values under given state and task characteristics, and represents optimal distribution of the action values. The policy loss measures the performance of the policy network by calculating the difference between the first distribution and the second distribution, and updates the parameters of the policy network by minimizing the policy loss, so that the policy network can be quickly adapted and fine-tuned when facing a new task, thereby improving the operation performance and efficiency of the mechanical arm in a complex environment.
The method comprises the steps of S701 to S705, wherein the steps are shown in the embodiment of the application, the probability distribution of the target mechanical arm action is weighted based on the target action weight to generate a first distribution, the high-advantage action is given higher weight to highlight the contribution of the high-advantage action, the action value of the target mechanical arm action is calculated to obtain the target action value, the expected accumulated return of the action is quantized, the probability distribution of the target action value is used as a second distribution, the policy loss is generated according to the difference between the first distribution and the second distribution, the policy loss reflects the deviation between the output of the current policy network and the value optimal distribution, the action advantage information can be converted into a directional optimization signal of the policy network, and the policy network parameters are forced to update towards the direction of increasing the high-value action probability, so that more accurate action selection policy, acceleration policy network convergence and sample utilization efficiency are realized in the mechanical arm control task.
In step S704 of some embodiments, policy losses are set:
Wherein, the Is a policy penalty; for ensuring that the product of the target action weight ω (s, a) and the policy network pi (a|s) forms an effective probability distribution over the action space.
In step S705 of some embodiments, policy-based lossAnd performing gradient descent updating on parameters of the strategy network. After the meta-training phase is finished, fine tuning can be performed on new tasks which are not seen in the meta-testing phase by using the trained strategy network.
Referring to fig. 4, an embodiment of the present application further provides a mechanical arm control device based on meta reinforcement learning, which can implement the mechanical arm control method based on meta reinforcement learning, where the device includes:
The interaction data acquisition module is used for acquiring historical interaction data of the control task of the mechanical arm;
the task feature extraction module is used for carrying out feature extraction on the control task of the mechanical arm based on the historical interaction data to obtain task features;
the action and state determining module is used for determining the action of the target mechanical arm through a preset strategy network based on the task characteristics;
the target action advantage calculation module is used for calculating an advantage value of the target mechanical arm action through a preset advantage function to obtain a target action advantage;
the normalization processing module is used for carrying out normalization processing on the target action advantages to obtain target action weights;
the parameter updating module is used for updating parameters of the preset strategy network based on the target action weight to obtain a target strategy network;
And the mechanical arm control module is used for controlling the mechanical arm by utilizing the target strategy network.
The specific implementation of the mechanical arm control device based on meta reinforcement learning is basically the same as the specific embodiment of the mechanical arm control method based on meta reinforcement learning, and is not repeated here.
The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the mechanical arm control method based on the meta reinforcement learning when executing the computer program. The electronic equipment can be any intelligent terminal such as a vehicle-mounted computer and the like.
Referring to fig. 5, fig. 5 illustrates a hardware structure of an electronic device according to another embodiment, where the electronic device includes:
The processor 901 may be implemented by using a general-purpose CPU (central processing unit), GPU (GraphicsProcessingUnit, image processor), microprocessor, application specific integrated circuit (ApplicationSpecificIntegratedCircuit, ASIC), or one or more integrated circuits, etc. to execute related programs to implement the technical solutions provided by the embodiments of the present application;
The memory 902 may be implemented in the form of read-only memory (ReadOnlyMemory, ROM), static storage, dynamic storage, or random access memory (RandomAccessMemory, RAM), among others. The memory 902 may store an operating system and other application programs, and when the technical solution provided in the embodiments of the present disclosure is implemented by software or firmware, relevant program codes are stored in the memory 902, and the processor 901 invokes the mechanical arm control method based on meta reinforcement learning to execute the embodiments of the present disclosure;
An input/output interface 903 for inputting and outputting information;
The communication interface 904 is configured to implement communication interaction between the device and other devices, and may implement communication in a wired manner (e.g. USB, network cable, etc.), or may implement communication in a wireless manner (e.g. mobile network, WIFI, bluetooth, etc.);
A bus 905 that transfers information between the various components of the device (e.g., the processor 901, the memory 902, the input/output interface 903, and the communication interface 904);
Wherein the processor 901, the memory 902, the input/output interface 903 and the communication interface 904 are communicatively coupled to each other within the device via a bus 905.
The embodiment of the application also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the mechanical arm control method based on the meta reinforcement learning when being executed by a processor.
The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The mechanical arm control method based on the meta reinforcement learning, the mechanical arm control device based on the meta reinforcement learning, the electronic equipment and the storage medium can generate task characteristics representing task characteristics by extracting characteristics of historical interaction data, instruct a strategy network to select actions through the task characteristics, improve adaptability of the strategy network to different tasks, calculate target action advantages through a preset advantage function, quantify the good and bad degree of the target mechanical arm action, normalize the target action advantages, convert the target action advantages into weights, update parameters of the preset strategy network based on the target action weights, ensure that the high-advantage action obtains the high weights to lead the training direction, and enable the strategy network to be quickly adapted to unknown tasks only by a small number of samples, thereby improving generalization capability of the mechanical arm control method.
The embodiments described in the embodiments of the present application are for more clearly describing the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application, and those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.
It will be appreciated by persons skilled in the art that the embodiments of the application are not limited by the illustrations, and that more or fewer steps than those shown may be included, or certain steps may be combined, or different steps may be included.
The above described apparatus embodiments are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof.
The terms "first," "second," "third," "fourth," and the like in the description of the application and in the above figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "and/or" is used to describe an association relationship of an associated object, and indicates that three relationships may exist, for example, "a and/or B" may indicate that only a exists, only B exists, and three cases of a and B exist simultaneously, where a and B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one of a, b or c may represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the above-described division of units is merely a logical function division, and there may be another division manner in actual implementation, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including multiple instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the various embodiments of the present application. The storage medium includes various media capable of storing programs, such as a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory RAM), a magnetic disk, or an optical disk.
The preferred embodiments of the present application have been described above with reference to the accompanying drawings, and are not thereby limiting the scope of the claims of the embodiments of the present application. Any modifications, equivalent substitutions and improvements made by those skilled in the art without departing from the scope and spirit of the embodiments of the present application shall fall within the scope of the claims of the embodiments of the present application.
Claims (7)
1. The mechanical arm control method based on meta reinforcement learning is characterized by comprising the following steps:
Acquiring historical interaction data of a control task of the mechanical arm;
based on the historical interaction data, extracting characteristics of the mechanical arm control task to obtain task characteristics;
determining the action of a target mechanical arm through a preset strategy network based on the task characteristics;
calculating the action value of the target mechanical arm through a preset action value network to obtain a target action value;
calculating the value of the initial environment state through a preset state value network to obtain the value of the initial state;
Calculating a dominance value for the motion of the target mechanical arm through a preset dominance function according to the target motion value and the initial state value to obtain a target motion dominance;
according to the action of the target mechanical arm and the initial environmental state, updating the environmental state to obtain a target environmental state;
calculating the average value of the task features to obtain a task feature average value;
Calculating the value of the target environment state based on the task feature mean value to obtain a target state value;
Acquiring rewards of the action of the target mechanical arm;
calculating action value loss based on the target action value, the rewards and the target state value;
Based on the action value loss, updating parameters of the action value network;
Normalizing the target action advantages to obtain target action weights;
According to the target action weight, calculating a weighted value of probability distribution of the target mechanical arm action to obtain a first distribution;
calculating the action value of the action of the target mechanical arm to obtain a target action value;
Calculating probability distribution of the target action value to obtain second distribution;
calculating the difference between the first distribution and the second distribution to obtain policy loss;
based on the strategy loss, carrying out parameter updating on the preset strategy network to obtain a target strategy network;
and controlling the mechanical arm by using the target strategy network.
2. The method of claim 1, wherein determining, based on the task characteristics, a target robotic arm action through a preset policy network, comprises:
Acquiring an initial environment state;
And selecting actions through a preset strategy network according to the task characteristics and the initial environment state to obtain the actions of the target mechanical arm.
3. The method according to claim 1, wherein the method further comprises:
calculating to obtain state value loss based on the target action value and the initial state value;
And updating parameters of the state value network based on the state value loss.
4. The method according to claim 1, wherein the performing feature extraction on the mechanical arm control task based on the historical interaction data to obtain task features includes:
And encoding the historical interaction data through a preset reasoning network to obtain posterior distribution parameters of the mechanical arm control task as the task characteristics.
5. A mechanical arm control device based on meta reinforcement learning, the device comprising:
The interaction data acquisition module is used for acquiring historical interaction data of the control task of the mechanical arm;
The task feature extraction module is used for carrying out feature extraction on the mechanical arm control task based on the historical interaction data to obtain task features;
the action and state determining module is used for determining the action of the target mechanical arm through a preset strategy network based on the task characteristics;
The system comprises a target action advantage calculation module, a target action advantage calculation module and a target action calculation module, wherein the target action advantage calculation module is used for calculating the action value of the target mechanical arm through a preset action value network to obtain a target action value;
The device is also used for updating the environment state according to the target mechanical arm action and the initial environment state to obtain a target environment state, calculating the average value of the task characteristics to obtain a task characteristic average value, calculating the value of the target environment state based on the task characteristic average value to obtain a target state value, acquiring rewards of the target mechanical arm action, calculating to obtain action value loss based on the target action value, the rewards and the target state value, and updating parameters of the action value network based on the action value loss;
The normalization processing module is used for carrying out normalization processing on the target action advantages to obtain target action weights;
The parameter updating module is used for calculating a weighted value of the probability distribution of the target mechanical arm action according to the target action weight to obtain a first distribution, calculating an action value of the target mechanical arm action to obtain a target action value, calculating a probability distribution of the target action value to obtain a second distribution, calculating the difference between the first distribution and the second distribution to obtain a strategy loss, and carrying out parameter updating on the preset strategy network based on the strategy loss to obtain a target strategy network;
and the mechanical arm control module is used for controlling the mechanical arm by utilizing the target strategy network.
6. An electronic device comprising a memory storing a computer program and a processor implementing the meta reinforcement learning based robotic arm control method of any one of claims 1 to 4 when executing the computer program.
7. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the meta reinforcement learning-based robot arm control method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511226596.1A CN120941397B (en) | 2025-08-29 | 2025-08-29 | Mechanical arm control method, device, equipment and medium based on meta reinforcement learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511226596.1A CN120941397B (en) | 2025-08-29 | 2025-08-29 | Mechanical arm control method, device, equipment and medium based on meta reinforcement learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN120941397A CN120941397A (en) | 2025-11-14 |
| CN120941397B true CN120941397B (en) | 2026-03-24 |
Family
ID=97620265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202511226596.1A Active CN120941397B (en) | 2025-08-29 | 2025-08-29 | Mechanical arm control method, device, equipment and medium based on meta reinforcement learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120941397B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115481688A (en) * | 2022-09-19 | 2022-12-16 | 山东大学 | Robot assembly method and system based on feature adaptive transfer reinforcement learning |
| CN116992977A (en) * | 2023-07-18 | 2023-11-03 | 西北工业大学 | A confidence inverse reinforcement learning method for robot skill training |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20230082326A1 (en) * | 2020-02-07 | 2023-03-16 | Deepmind Technologies Limited | Training multi-objective neural network reinforcement learning systems |
| CN111515961B (en) * | 2020-06-02 | 2022-06-21 | 南京大学 | Reinforcement learning reward method suitable for mobile mechanical arm |
| CN113510704A (en) * | 2021-06-25 | 2021-10-19 | 青岛博晟优控智能科技有限公司 | A Motion Planning Method for Industrial Robot Arm Based on Reinforcement Learning Algorithm |
| CN114939870B (en) * | 2022-05-30 | 2023-05-09 | 兰州大学 | Model training method and device, strategy optimization method, equipment and medium |
| CN115256372B (en) * | 2022-06-30 | 2024-07-12 | 兰州大学 | Mechanical arm control method, device, control equipment and storage medium |
| CN117464687B (en) * | 2023-12-05 | 2024-11-15 | 哈尔滨工业大学 | Non-cooperative target capturing method based on deep reinforcement learning |
| CN118061178A (en) * | 2024-02-27 | 2024-05-24 | 河北大学 | A method for adjusting the behavior of neural networks for robotic tasks based on human feedback |
| CN118744436A (en) * | 2024-08-16 | 2024-10-08 | 江西求是高等研究院 | A robotic arm control method and system based on hierarchical reinforcement learning and combinatorial optimization |
| CN118876062A (en) * | 2024-08-20 | 2024-11-01 | 山东大学 | A robot assembly method and system based on metric learning and meta-reinforcement learning |
| CN119567245B (en) * | 2024-10-31 | 2025-09-19 | 西北工业大学 | Homotopy-based meta-reinforcement training for pick-and-place tasks in space manipulators |
-
2025
- 2025-08-29 CN CN202511226596.1A patent/CN120941397B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115481688A (en) * | 2022-09-19 | 2022-12-16 | 山东大学 | Robot assembly method and system based on feature adaptive transfer reinforcement learning |
| CN116992977A (en) * | 2023-07-18 | 2023-11-03 | 西北工业大学 | A confidence inverse reinforcement learning method for robot skill training |
Also Published As
| Publication number | Publication date |
|---|---|
| CN120941397A (en) | 2025-11-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112052948B (en) | Network model compression method and device, storage medium and electronic equipment | |
| CN114139637A (en) | Multi-agent information fusion method and device, electronic equipment and readable storage medium | |
| CN119987521B (en) | Game paddle energy-saving control method and system based on intelligent analysis | |
| CN111078847A (en) | Power user intent identification method, apparatus, computer equipment and storage medium | |
| CN116827685B (en) | Dynamic defense strategy method of micro-service system based on deep reinforcement learning | |
| US20220343216A1 (en) | Information processing apparatus and information processing method | |
| CN109784015A (en) | A kind of authentication identifying method and device | |
| CN121315961A (en) | Machine Learning-Based Method and System for Motion Trajectory of Heavy-Duty Robots | |
| CN117957549A (en) | Method and device for incremental learning of meta-few-shot classes | |
| CN110610140A (en) | Training method, device and equipment of face recognition model and readable storage medium | |
| CN115319741B (en) | Training method of robot control model and robot control method | |
| CN120941397B (en) | Mechanical arm control method, device, equipment and medium based on meta reinforcement learning | |
| CN121189514A (en) | A method, apparatus, device, and medium for inference optimization of a large code generation model. | |
| CN113554105B (en) | Missing data completion method for Internet of things based on space-time fusion | |
| CN119884848B (en) | Target damage assessment method and system, electronic equipment and storage medium | |
| CN119397418B (en) | Object classification method and device based on distributed multi-agent reinforcement learning | |
| JP2021140749A (en) | Electronic device for precise action profiling for implanting human's knowledge to artificial intelligence, and operation method thereof | |
| JP7579632B2 (en) | Estimation device, system and method | |
| WO2025170866A1 (en) | Generating text-to-motion animations from partially annotated datasets | |
| CN119204117A (en) | Pulse neural network supervised learning method, device, electronic device and storage medium | |
| CN118018429A (en) | Network service satisfaction analysis method, system, device and storage medium | |
| JP2021012600A (en) | Method for diagnosis, method for learning, learning device, and program | |
| KR102781636B1 (en) | Electronic device for high-precision profiling to develop artificial inntelligence with human-like intelligence, and operating method thereof | |
| CN120197844A (en) | Resource allocation strategy adjustment method and device, computer equipment and storage medium | |
| CN116778264A (en) | Object classification method, image classification method and related equipment based on incremental learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |