JP7637345B2 - AUTONOMOUS MOBILE DEVICE, SERVER DEVICE, ANOMALYSIS DETECTION METHOD, AND PROGRAM - Google Patents

Description

This disclosure relates to an autonomous mobile device, a server device, a learning device, an anomaly detection method, and a program.

In recent years, autonomously moving devices such as drones and self-driving cars have been actively researched, and some of them are actually being used. Furthermore, autonomous moving devices are expected to become more widespread in the future due to the decline in the working population, cost reduction, and business efficiency. For example, as tasks such as transporting and delivering luggage are replaced by people and cars driven by people, it is expected that autonomous moving devices will be used in situations closely related to people's lives. In that case, autonomous moving devices will be operated in people's living spaces, so they will be required to operate safely as well as reliably and stably.

Patent Document 1 describes an emergency response instruction device for a drone that includes a receiving means, a determining means, a forming means, and a transmitting means. The receiving means receives a status notification signal, which is a signal from a drone, which is an unmanned aerial vehicle, and includes status information indicating the status of the drone. When the determining means receives a status notification signal through the receiving means, the determining means determines whether the status of the drone is an emergency based on the status information included in the status notification signal. When the determining means determines that the status of the drone is an emergency, the forming means forms response instruction information for the drone. The transmitting means transmits the response instruction information formed by the forming means.

Regarding abnormality detection, Patent Document 2 describes a plant operation support device that includes an abnormal state database, an operation manual database, a plant monitoring unit, an abnormal state discrimination unit, a countermeasure search unit, and a notification unit. The abnormal state database records the range of parameters when the plant body is in an abnormal state, and the operation manual database records the operation procedure for eliminating the abnormal state. The plant monitoring unit collects parameters from the plant body. The abnormal state discrimination unit refers to the abnormal state database and determines whether the plant body is in an abnormal state based on the collected parameters. The countermeasure search unit extracts the operation procedure from the operation manual database when the plant body is in an abnormal state. The notification unit notifies the terminal device of the operation procedure.

JP 2018-165115 A JP 2003-005828 A

The present inventors have considered safe and stable operation of an autonomous mobile device.
In order to operate an autonomous mobile device safely, that is, to make it difficult for a collision to occur in the first place even if an abnormality occurs in the autonomous mobile device or the surrounding environment, the following safety measures can be considered: That is, possible safety measures include operating the autonomous mobile devices with a sufficiently large distance between each other and between the autonomous mobile device and surrounding structures or people, operating at a sufficiently slow speed, and operating only in areas where a crash or collision will not cause any problems.

However, assuming that autonomous mobile devices will be fully deployed for purposes such as home delivery, they will need to be able to operate in large numbers, at high density, and at high speeds in crowded city areas, but the safety measures described above will not meet these requirements. In other words, ensuring safety by operating in limited areas, with sufficiently wide intervals between devices, or at low speeds will not allow for the operation of large numbers of autonomous mobile devices at high density and high speeds. Therefore, a system is needed that allows for operation in large numbers, at high density, and at high speeds, and that also provides safety measures in the event of an emergency.

On the other hand, some autonomous mobile devices have a mechanism for stopping the device for safety reasons with a very simple judgment. For example, there is a mechanism for monitoring the operating status of the autonomous mobile device's operating parts and stopping the device if a malfunction is suspected, or a mechanism for monitoring the thresholds of sensor values from attitude sensors, acceleration sensors, etc., and stopping the device if a certain sensor exceeds a certain threshold.

However, because autonomous mobile devices are used in a variety of situations, immediately stopping movement due to overdetection, malfunction, or abnormal value detection may be preventative and excessive, failing to meet operational requirements, or may not necessarily be considered safe.

For example, if a self-driving car, a type of autonomous mobile device, is traveling in the passing lane on a highway, stopping the car on the spot even if an abnormality is suspected could lead to a collision with the car behind, which is dangerous. Therefore, unless there is an emergency that outweighs the risk, the car should continue driving to a safe place where it is safe to stop.

As another example, drones, a type of autonomous mobile device, operated in urban areas for transportation purposes, etc., pose a risk of contact with people or vehicles on the ground, power lines, important facilities, etc., and therefore it is not advisable to immediately land the drone on the spot even if an abnormality is suspected. In other words, unless there is an emergency that outweighs the risk, the drone should be operated to a safe location for landing. As mentioned above, it is expected that high-density, high-speed operation will be required in the future for drones as well, and so sudden landing operations pose a risk of collision with other drones nearby, just as in the case of self-driving cars on highways.

In addition, if the operational requirements are to transport important goods for transportation purposes, etc., even in an emergency, it may not be desirable to stop or land the aircraft in places where recovery would be difficult or impossible, such as deep in the mountains, in remote areas, on the sea, on lakes, or in rivers.

In other words, making excessive emergency stops of an autonomous mobile device due to suspected abnormalities is inconvenient from an operational standpoint, and depending on operational requirements, an emergency stop may not be the best way to deal with the situation. In such cases, it is desirable for the operator, rather than the developer, to be able to define the criteria and countermeasures according to the operational requirements.

Another possible method would be for humans to remotely operate the autonomous mobile devices only in emergencies, but this method would require a large number of personnel if a large number of autonomous mobile devices were to be operated, which could lead to the following problems. That is, problems would arise in terms of costs and securing manpower, and in the event of a multiple accident, there would be a limit to the number of autonomous mobile devices that can be used.

We have explained the issues from a safety perspective, but problems can also arise from the perspective of stable operation if extreme measures such as an emergency stop or emergency landing are implemented to halt operations, triggered by simple criteria such as suspected abnormalities, the presence or absence of a single failure in a redundant part, or a threshold in sensor information. This problem frequently interferes with the operation of the autonomous mobile device, which is the original purpose of the autonomous mobile device, such as transportation and movement, and frequently makes it impossible to meet operational requirements, such as delivering transported goods by a set time.

Therefore, it is desirable for autonomous mobile devices to have a means of determining the degree and type of abnormality, and, if the abnormality is minor, to allow the operator, rather than the developer, to flexibly and easily set the conditions for determining the abnormality and the method of dealing with it in accordance with operational requirements.

In contrast, the technology described in Patent Document 1 can take measures after detecting an abnormality in order to deal with an abnormality in a drone, but it cannot improve the accuracy of abnormality determination before an abnormality occurs, avoid emergency landings as much as possible, and increase the availability of the drone. Furthermore, the technology described in Patent Document 2 can identify an abnormal state, search for the cause, and extract measures to address the cause, but the extracted measures are based on data in each database, making it difficult for operators to change them for each plant, and lacking versatility.

The objective of the present disclosure is to provide an autonomous mobile device, a server device, a learning device, an anomaly detection method, a program, etc., that are capable of detecting abnormal conditions that require countermeasures using judgment conditions desired by the operator of the autonomous mobile device.

The autonomous mobile device according to the first aspect of the present disclosure includes a sensor group that detects the current state of the autonomous mobile device, a state estimation unit that estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the sensor group, a first abnormality determination unit that determines whether the current state is an abnormal state that requires a measure to change the state based on the estimation result by the state estimation unit, a second abnormality determination unit that determines which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times by the state estimation unit, an output unit that outputs the determination results by the first abnormality determination unit and the determination results by the second abnormality determination unit, and a criterion setting unit that sets a judgment criterion for the second abnormality determination unit. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

The server device according to the second aspect of the present disclosure includes a receiving unit that receives, from an autonomous mobile device, a detection result obtained by a sensor group that detects the current state of the autonomous mobile device; a state estimation unit that estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the sensor group; a first abnormality determination unit that determines whether the current state is an abnormal state that requires a measure to change the state based on the estimation result by the state estimation unit; a second abnormality determination unit that determines which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times by the state estimation unit; an output unit that outputs the determination results by the first abnormality determination unit and the determination results by the second abnormality determination unit; and a criterion setting unit that sets a judgment criterion for the second abnormality determination unit. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

The learning device according to the third aspect of the present disclosure includes an acquisition unit that acquires detection results acquired by a group of sensors that detect the current state of an autonomous mobile device, and a control unit, and the control unit generates a trained model for outputting countermeasure labels by inputting multiple detection results acquired by the acquisition unit and outputting one of multiple countermeasure labels and a normal label. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the multiple estimated results is a historical abnormal state that requires countermeasures to change the state, and the normal label is a label that does not require countermeasures.

The anomaly detection method according to the fourth aspect of the present disclosure includes an acquisition step of acquiring the results of the current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device, a state estimation step of estimating which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection results by the group of sensors, a first anomaly determination step of judging whether the current state is an abnormal state that requires a measure to change the state based on the estimation result in the state estimation step, a second anomaly determination step of judging which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times at different times in the state estimation step, an output step of outputting the judgment results in the first anomaly determination step and the judgment results in the second anomaly determination step, and a criterion setting step of setting a judgment criterion for the second anomaly determination step. The multiple countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a measure to change the state, and the normal label is a label that does not require a measure.

The program according to the fifth aspect of the present disclosure causes a computer provided in an autonomous mobile device or connected to the autonomous mobile device to execute the following steps: an acquisition step of acquiring the results of the current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection results by the group of sensors; a first abnormality determination step of determining whether the current state is an abnormal state that requires a measure to change the state based on the estimation result in the state estimation step; a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times at different times in the state estimation step; an output step of outputting the determination results in the first abnormality determination step and the determination results in the second abnormality determination step; and a criterion setting step of setting a judgment criterion for the second abnormality determination step. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

This disclosure makes it possible to provide an autonomous mobile device, a server device, a learning device, an anomaly detection method, a program, and the like that can detect abnormal conditions that require countermeasures using the judgment criteria desired by the operator of the autonomous mobile device.

1 is a block diagram showing a configuration example of an autonomous mobile device according to a first embodiment. FIG. 2 is a block diagram showing an example configuration of a learning device that generates a trained model used in an autonomous mobile device according to embodiment 1. FIG. 11 is a block diagram showing a configuration example of an autonomous mobile device according to a second embodiment. 4 is a block diagram showing an example of the configuration of a state estimation unit in the autonomous mobile device of FIG. 3 . FIG. 11 is a diagram illustrating an example of learning data used when generating a state model. FIG. 13 is a diagram illustrating an example of state classification. 11 is a conceptual diagram for explaining an example of a procedure for generating a state model 122 when a correct label is unknown. FIG. FIG. 13 is a conceptual diagram for explaining an example of a procedure for generating a state model 122 when a correct label is known. FIG. 11 is a schematic diagram for explaining an example of state estimation. FIG. 11 is a diagram illustrating an example of data. 4 is a block diagram showing an example of the configuration of a state abnormality determination unit in the autonomous mobile device of FIG. 3 . FIG. 11 is a diagram illustrating an example of a status abnormality rule. FIG. 11 is a diagram illustrating an example of an abnormality determination. 4 is a block diagram showing an example of the configuration of a state history abnormality determination unit in the autonomous mobile device of FIG. 3 . FIG. 13 is a diagram illustrating an example of a state history anomaly rule. 4 is a block diagram showing an example of the configuration of a test execution unit in the autonomous mobile device of FIG. 3 . FIG. 13 is a diagram illustrating an example of a test rule. 4 is a block diagram showing an example of the configuration of a countermeasure decision unit in the autonomous mobile device of FIG. 3 . FIG. 13 is a diagram illustrating an example of a countermeasure rule. FIG. 4 is a flow diagram for explaining an example of the operation of the autonomous mobile device of FIG. 3 . 5 is a flow diagram for explaining an example of the operation of the state estimation unit 12 in FIG. 4. 12 is a flow chart for explaining an example of the operation of the abnormal state determination unit in FIG. 11 . 15 is a flow chart for explaining an example of the operation of the state history abnormality determination unit of FIG. 14; 17 is a flow diagram for explaining an example of the operation of the test execution unit in FIG. 16. 4 is a flow diagram for explaining an example of the operation of a countermeasure decision unit in the autonomous mobile device of FIG. 3 . FIG. 11 is a block diagram showing a configuration example of an autonomous mobile system according to a third embodiment. FIG. 11 is a block diagram showing another configuration example of the autonomous mobile system according to the third embodiment. FIG. 2 illustrates an example of a hardware configuration of the apparatus.

Hereinafter, the embodiments will be described with reference to the drawings. Note that in the embodiments, the same or equivalent elements may be given the same reference numerals, and duplicate descriptions will be omitted as appropriate.

<Embodiment 1>
An autonomous mobile device (autonomous mobile equipment) according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the autonomous mobile device according to the first embodiment.

As shown in FIG. 1, the autonomous mobile device 1 according to this embodiment includes a plurality of sensors (sensor group) 1a, a state estimation unit 1b, a first abnormality determination unit 1c, a second abnormality determination unit 1d, an output unit 1e, and a standard setting unit 1f. The autonomous mobile device 1 may also include a communication unit (not shown) that communicates with an external device. This communication unit may be at least one of a wireless communication unit and a wired communication unit, depending on the type of the autonomous mobile device 1, the timing of data communication with the outside, etc.

The autonomous mobile device 1 is a device that can move autonomously, and can detect whether there are any abnormal conditions during movement (operation) based on the detection results (sensor data) output from the sensor group 1a.

The autonomous mobile device 1 according to this embodiment can be applied to any mobile device that requires monitoring (detection) of whether the autonomous mobile device 1 is in an abnormal state while moving, and can be suitably used for managing such devices. The autonomous mobile device 1 has information processing capabilities, controls actuators and the like, moves autonomously, and can be any device that is responsible for transportation or movement.

Examples of the autonomous mobile device 1 include self-driving automobiles, self-driving trains, self-navigating ships, self-flying aircraft, drones, AGVs (automated guided vehicles) used in factories and warehouses, etc., robots with running and leg-based movement functions, robotic electric wheelchairs, motorcycles, etc. Here, the drone is not limited to an air vehicle such as an unmanned aerial vehicle that flies in the sky, but can be operated in any environment, such as on the ground, in the sky, on water, underwater, or in space. In addition, the robot can be moved by any method, such as running or walking. Furthermore, examples of the autonomous mobile device 1 include construction vehicles such as forklifts, construction machines, and heavy machinery, vehicles used in logistics such as railway vehicles, taxis, and trucks, police vehicles, and fire engines. Here, the railway vehicle can be moved by any method, such as light rail, iron wheel type, new transportation system, monorail, magnetic levitation type, etc. The autonomous mobile device 1 also includes devices that can switch between modes, operating autonomously at times and in response to human instructions or operations at other times, and devices that move in coordination with human operations or instructions for part of their autonomous operation. In other words, the autonomous mobile device 1 only needs to be capable of moving autonomously, and can also have the function of moving under control from an external controller or under the operation of the passenger.

The sensor group 1a detects the current state of the autonomous mobile device 1. Here, the current state can refer to the sensor value of the sensor included in the sensor group 1a as a result of measuring the detection target at the installation location, or the state indicated by the sensor value, and mainly includes the moving state (operating state). The moving state can include the inclination and position of the autonomous mobile device 1. The installation location and the detection target generally differ depending on the type and shape of the autonomous mobile device 1. The sensor group 1a can be composed of multiple sensors necessary for controlling the autonomous mobile device 1, and can include not only a sensor group that collects the state of the autonomous mobile device 1 itself, but also a sensor group that collects the state of the surroundings (surrounding environment). The detection interval in the sensor group 1a does not matter, but it is desirable for the detection to be short for quick detection. The detection intervals of the sensors belonging to the sensor group 1a are preferably the same, but may be different, and detection data with a slight difference in detection time in each sensor can be treated as the same time.

Some or all of the units 1b to 1f can be configured as a control unit (not shown) that controls the entire autonomous mobile device 1. This control unit can be realized, for example, by a CPU (Central Processing Unit), a working memory, and a non-volatile storage device that stores a program. This program can be a program that executes processing to realize the functions of some or all of the units 1b to 1f. This control unit can also be realized, for example, by an integrated circuit.

The state estimation unit 1b estimates which of a number of predefined state labels the current state of the autonomous mobile device 1 corresponds to, based on the detection results of the sensor group 1a. The state estimation by the state estimation unit 1b makes it possible to convert a combination of information from a number of sensors (e.g., numerical values, ON/OFF, mode, etc.) into a state label that represents the state of the autonomous mobile device 1, which is easy for humans to understand. This makes it simple and easy to define a method of dealing with the state of the autonomous mobile device 1. Furthermore, this allows the user of the autonomous mobile device 1 to flexibly define emergency operations at the operation stage, rather than the developer incorporating them in advance as control rules, mechanisms, circuits, and software at the design and development stage of the autonomous mobile device 1. Here, the definition of a method of dealing with an emergency refers to the definition of a countermeasure, which will be described later, and the definition of the criteria for determining when to implement the countermeasure.

The first abnormality determination unit 1c determines whether or not the current state is an abnormal state (state requiring action) that requires action to change the state, based on the estimation result by the state estimation unit 1b (i.e., the state label estimated by the state estimation unit 1b).

The second abnormality determination unit 1d determines whether the state corresponds to one of multiple countermeasure labels or normal labels based on the results of multiple estimations made by the state estimation unit 1b at different times (multiple times or multiple types of state labels). To make this determination, the estimation results made by the state estimation unit 1b can be stored as history.

The multiple countermeasure labels are labels that are predefined for each countermeasure to indicate that the state of the autonomous mobile device 1 indicated by the results of multiple estimations is a historical abnormal state (state requiring countermeasures) that requires countermeasures to change the state. In other words, a countermeasure label is set for each countermeasure to be executed according to the results of multiple estimations, or in other words, a countermeasure is set for each countermeasure label indicated by the judgment result. In addition, the normal label is a label that does not require countermeasures, and if it is determined that it corresponds to a normal label, it means that no countermeasures are required.

The output unit 1e outputs the judgment result of the first abnormality judgment unit 1c and the judgment result of the second abnormality judgment unit 1d. The destination of these outputs can include, for example, a movement control unit (drive control unit) (not shown) that controls the movement of the autonomous mobile device 1, thereby making it possible to control the movement according to the judgment results.

Then, the standard setting unit 1f sets the judgment criteria for the second abnormality judgment unit 1d. The standard setting unit 1f can be an operation unit (operation receiving unit) provided on the main body of the autonomous mobile device 1, a receiving unit that receives a setting signal so that the setting can be made by communication from an external device, or both.

The second abnormality determination unit 1d can also perform a determination using a first trained model that inputs the results of estimations made multiple times at different times by the state estimation unit 1b, inputs the estimation results from the state estimation unit 1b, and outputs one of multiple countermeasure labels or a normal label. This trained model and the algorithms in the trained model described below are not important as long as a determination can be made. Of course, the second abnormality determination unit 1d can also perform a determination using a predetermined definition formula or the like without using a trained model, and this is also true for the first abnormality determination unit 1c and the state estimation unit 1b.

The first abnormality judgment unit 1c also preferably judges whether the current state corresponds to one of a number of countermeasure labels predefined for each type of countermeasure to change the current state, or a normal label for which no countermeasure is required. The countermeasure labels and normal labels here can be defined in the same way as those in the second abnormality judgment unit 1d, but the difference is that the judgment is based on the result of a single state detection, so the judgment criteria are basically different. In other words, the label group of countermeasure labels and normal labels in the first abnormality judgment unit 1c can have labels in common with the label group of countermeasure labels and normal labels in the second abnormality judgment unit 1d, but at least the judgment criteria will be different.

The first anomaly determination unit 1c can also perform the determination using a second trained model that inputs the estimation result from the state estimation unit 1b and outputs one of multiple countermeasure labels or a normal label.

The state estimation unit 1b can also perform estimation using a third learned model that inputs the detection results from the sensor group 1a and outputs one of multiple state labels. In this way, state estimation by the state estimation unit 1b using machine learning can convert a combination of information from multiple sensors (such as numerical values, ON/OFF, and mode) into a state label that represents the state of the autonomous mobile device that is easy for humans to understand. By using machine learning, it is possible to estimate the state label more accurately.

Each trained model that can be used for judgment and estimation will be described later with reference to FIG. 2. Each trained model can also be constructed as a single trained model. In addition, each trained model can be stored inside the autonomous mobile device 1, such as by storing it inside the control unit, but it can also be stored in an external server device. When each trained model is stored in the server device, the autonomous mobile device 1 can transmit the detection results of the sensor group 1a and obtain the judgment or estimation results in response.

As described above, the use of state labels allows the user (operator) of the autonomous mobile device 1 to flexibly define emergency actions (actions when measures need to be taken) at the operational stage, and this is also true when a trained model is not used. Furthermore, the use of state labels that represent the state of the autonomous mobile device in a way that is easy for humans to understand allows the user to define more complex emergency actions for the autonomous mobile device 1. For example, the following becomes possible.

In other words, it is possible to provide a means for humans to easily define countermeasures for state labels that represent the state of the autonomous mobile device 1 that are easy for humans to understand, and a means for changing the operation of the autonomous mobile device 1 based on that definition. In addition, with regard to this countermeasure, since the state of the autonomous mobile device 1 can now be expressed by state labels, it becomes possible for the criteria setting unit 1f to define more complex and abnormal ways of using the autonomous mobile device that take into account the passage of time, and to define countermeasures for them, as in the judgment by the second abnormality judgment unit 1d. In addition, while the criteria setting unit 1f is configured to be able to set the judgment criteria for the second abnormality judgment unit 1d, it is preferable to also configure it so that it is possible to set the judgment criteria for the first abnormality judgment unit 1c. In this case, if the judgment result is output as a label by the first abnormality judgment unit 1c as well, the definition can be easily changed.

These mechanisms not only enable the autonomous mobile device 1 to automatically take appropriate countermeasures when it falls into a specific state, such as when an abnormality occurs, but also allow the operator to easily change countermeasures that had to be pre-fixed and built-in during the design and development of the autonomous mobile device. These changes can be made by the operator depending on, for example, the value and characteristics of the cargo at each time of operation, the contract conditions for operation, etc.

As described above, state estimation by the state estimation unit 1b allows the state of the autonomous mobile device 1 to be expressed in a format that is easy for people to understand, making it easy to define how to deal with that state, i.e., various rules can be described concisely. Here, we say that the rules can be described concisely because they can be described in terms of highly readable combination labels, rather than being defined for a large number of sensor values, status values, and other values, combinations and ranges.

For example, by describing the rules concisely, the behavior of the autonomous mobile device in the event of an abnormality can be varied according to the transported goods and the surrounding environment, the price of the vehicle and cargo, and the purpose of operation. Furthermore, because the state of the autonomous mobile device 1 is expressed in a format that is easy for people to understand, it is easy to define countermeasures for that state and how to judge them (for example, the definition of the tests and how to judge them as described in embodiment 2).

Also, by being able to concisely describe rules, users can concisely describe anomaly detection rules as definition files, rather than implementing them as control laws in the control unit (which requires specialized knowledge in the design and development of the autonomous mobile device 1). Also, by being able to concisely describe rules, users can concisely define not only simple anomaly detection rules, but also more complex conditions that take history into account.

For example, as an example of a judgment rule based on the number of occurrences, it can be defined so that if state A occurs a certain number of times within a certain time period, it is judged to be an abnormality. As an example of a rule for combinations that occur in no particular order, it can be defined so that if states A, B, and C occur within a certain time period, it is judged to be an abnormality. As an example of a rule for combinations where the order is important, it can be defined so that if state A, state B, and state C occur within a certain time period, it is judged to be an abnormality. As shown in these examples, complex conditions can be easily defined. Of course, the combinations of states are not limited to A, B, and C, and combinations of two or more states can be easily defined.

In addition, by allowing the operator to easily set rules in this way, there will be no need to take immediate emergency action in situations where excessive abnormality detections would occur, thereby increasing the availability of the autonomous mobile device 1.

As described above, in the autonomous mobile device 1 according to this embodiment, the classification of anomalies and the corresponding actions can be defined more precisely and flexibly and dynamically by the operator, rather than by the developer, in accordance with business requirements. In other words, according to this embodiment, it becomes possible to detect abnormal conditions that require countermeasures using the judgment conditions desired by the operator of the autonomous mobile device 1. This makes the autonomous mobile device 1 more available than autonomous mobile devices that can only handle standard responses such as moving to a specified location or making an emergency stop (landing) when an abnormality occurs.

Next, the trained model used by the autonomous mobile device 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example configuration of a learning device that generates a trained model used by the autonomous mobile device 1.

As shown in FIG. 2, the learning device 2 can include an acquisition unit 2a and a control unit 2b. Here, the control unit 2b can include a memory unit 2c. The learning device 2 can also be provided inside the autonomous mobile device 1.

The acquisition unit 2a acquires the detection results acquired by a group of sensors that detect the current state of the autonomous mobile device 1. When the learning device 2 is outside the autonomous mobile device 1, the acquisition unit 2a can be a reading unit that reads the detection results via a portable recording medium, or a receiving unit that receives the detection results via wired or wireless communication. When the learning device 2 is provided inside the autonomous mobile device 1, the acquisition unit 2a can acquire the detection results from the group of sensors 1a via internal wiring.

The control unit 2b inputs the detection results acquired by the acquisition unit 2a multiple times and generates a trained model for outputting countermeasure labels, which outputs either multiple countermeasure labels or a normal label. Note that the multiple countermeasure labels and normal labels are as described above.

The trained model 2e for outputting countermeasure labels can be a trained model constructed by inputting multiple detection results into the untrained model 2d stored in the memory unit 2c and performing machine learning, and this can also be stored in the memory unit 2c.

The control unit 2b can also be configured to generate a trained model for outputting a state label, which inputs the detection results acquired by the acquisition unit 2a and outputs one of a number of predetermined state labels to indicate the current state of the autonomous mobile device 1. Similarly, the trained model for outputting a state label can also be a trained model constructed by inputting the detection results into an untrained model 2d stored in the memory unit 2c and performing machine learning, and this can also be stored in the memory unit 2c.

The control unit 2b, like the control unit that controls the entire autonomous mobile device 1, can be realized by, for example, a CPU, a working memory, and a non-volatile storage device that stores a program. This program can be a trained model generation program.

In this way, the learning device 2 according to this embodiment can generate a trained model that can be used in an autonomous mobile device that can detect abnormal conditions that require countermeasures using the judgment conditions desired by the operator of the autonomous mobile device.

<Embodiment 2>
3 to 25, the second embodiment will be described with a focus on the differences from the first embodiment, but the various examples described in the first embodiment can also be applied. The following description will be given with an example in mind in which a drone is used as the autonomous mobile device according to this embodiment, but the present invention will be essentially the same even if the drone is replaced with another type of autonomous mobile device.

Figure 3 is a block diagram showing an example of the configuration of an autonomous mobile device according to embodiment 2. As shown in Figure 3, the autonomous mobile device 10 according to this embodiment can include a sensor group 11, a state estimation unit 12, a state abnormality determination unit 13, a state history abnormality determination unit 14, a test implementation unit 15, a countermeasure decision unit 16, a movement control unit 17, and a standard setting unit 18. The various examples exemplified as the autonomous mobile device 1 according to embodiment 1 can be applied as the autonomous mobile device 10. An autonomous mobile device generally only includes a sensor group 11, a movement control unit 17, and a simple abnormality detection means and operates autonomously, but the autonomous mobile device 10 according to this embodiment further includes each of the units 12 to 16, 18.

The sensor group 11 corresponds to the sensor group 1a, the state estimation unit 12 is an example of the state estimation unit 1b, the state abnormality determination unit 13 is an example of the first abnormality determination unit 1c, the state history abnormality determination unit 14 and the test implementation unit 15 are an example of the second abnormality determination unit 1d, and the countermeasure decision unit 16 is an example of the output unit 1e. The standard setting unit 18 is an example of the standard setting unit 1f. Note that only one of the state history abnormality determination unit 14 and the test implementation unit 15 may be installed.

Each component shown in FIG. 3 may be a program on a computer, may be composed of an electric/electronic circuit, or may be a combination of a program and an electric/electronic circuit. Information can be exchanged between components connected by lines in FIG. 3. If the components are programs, information between the components may be exchanged via shared memory or storage, or if the units are separate for implementation, information may be exchanged via communication. In addition, all or part of the components other than the sensor group 11 in FIG. 3 may be mounted on an external device connected via communication to the autonomous mobile device 10 and processed therein, and this example will be described later as embodiment 3. For example, a server device (which may be a cloud server device) such as a management device connected via communication to the autonomous mobile device 10 may execute the same process instead of the autonomous mobile device 10. Details of each component are described below.

(Sensor group 11)
First, the sensor group 11 is a group of sensors mounted on the autonomous mobile device 10. Not limited to the autonomous mobile device 10, the sensor group 11 can be a plurality of various sensors that are generally mounted on an autonomous mobile device to collect information on its own state and the state of the surroundings that are necessary for its control.

An example of the sensor group 11 will be described using a drone, which is a type of autonomous mobile device, as an example. The sensor group 11 can include a speed sensor that observes the speed of the aircraft, an acceleration sensor that observes the acceleration of the aircraft, a gyro sensor that observes the roll, pitch, and yaw of the aircraft, and a rotation sensor that observes the rotation speed of each rotor used in flight. The sensor group 11 can also include a pressure sensor and an altitude sensor that observe the altitude of the aircraft, a geomagnetic sensor that observes the direction from the aircraft, and a temperature sensor that detects heat generation in the circuits and rotors. Of course, the sensor group 11 does not need to include all of these examples, and can include only a portion of them, and can also include other types of sensors without being limited to these examples.

For example, other sensors included in the sensor group 11 include a sensor that acquires position information in the Global Navigation Satellite System (GNSS). GNSS is a general term for satellite positioning systems such as the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), Galileo, and the Quasi-Zenith Satellite System. Here, the Quasi-Zenith Satellite System is the Quasi-Zenith Satellite System (QZSS). With this sensor, the autonomous mobile device 10 can receive GNSS signals, measure its own position in space, and obtain position information. The sensors included in the sensor group 11 can also be optical cameras or stereo cameras, and the position and attitude of the device can be obtained by processing the images acquired by the cameras. The sensors included in the sensor group 11 can also be sonic sensors, ultrasonic sensors, 2D-LiDAR, 3D-LiDAR, and the like. In this case, the autonomous mobile device 10 can process waveform data and point cloud data obtained from a sensing device using sound waves or lasers to determine its own position, attitude, surrounding conditions, and the like, and the information obtained in this way can also be considered sensor information. Therefore, sensors that require information processing such as these can also be included in the sensor group 11.

In addition, the operating mode of the autonomous mobile device 10 (mode On/Off, current mode, etc.) and the state of other mounted devices (such as the rotor, etc.) and log data of the control device used for autonomous control can also be considered sensor information, so devices that provide such information can also be included in the sensor group 11. Here, examples of the rotor state include valid/invalid, normal/abnormal, rotation speed, output, temperature, etc.

In the case of a general autonomous mobile device, rather than the present embodiment, the sensing results collected by the sensor group 11 are input to the mobile control unit 17, which will be described next, and simple autonomous control is performed, but the autonomous mobile device 10 according to this embodiment is equipped with each of the units 12 to 16, 18.

(Movement control unit 17)
The movement control unit 17 is a control unit that controls the movement of the autonomous mobile device 10, and can also be called a drive control unit. The movement control unit 17 is a part that controls actuators such as rotors mounted on the autonomous mobile device 10 based on sensing information provided by the sensor group 11, and is the brain that performs autonomous control. Since the autonomous mobile device 10 performs autonomous control so as to operate along the route, coordinates, attitude, and speed provided at the start of operation, even if the position is slightly shifted due to control errors or disturbances such as wind, the actuators can be controlled based on the sensing information to maintain the original position, attitude, and speed by itself. This is a function that is generally installed in autonomous mobile devices. In addition, in general autonomous mobile devices, for simple abnormalities that are easy to detect, such as the stop of some rotors, loss of communication with the ground, and large deviation from the route, a minimum response function may be built in, such as moving to a predetermined coordinate, lowering the altitude and making an emergency landing on the spot. However, the autonomous mobile device 10 according to this embodiment enables control as desired by the operator even in situations where this response function is executed excessively.

(State Estimation Unit 12)
The state estimation unit 12 will be described with reference to Fig. 4 to Fig. 10. Fig. 4 is a block diagram showing an example of the configuration of the state estimation unit 12 in the autonomous mobile device 10 of Fig. 3.

The state estimation unit 12 is a functional unit that converts a combination of various sensor information sent from the sensor group 11 into a name that represents a state that is easy for people to understand and its accuracy. As in the first embodiment, the name that represents this state is called a state label.

As shown in FIG. 4, the state estimation unit 12 can be composed of an estimation unit 121 and a state model 122. The state model 122 can be constructed as shown in the following example. The creation procedure for the state model 122 differs depending on whether the correct label is unknown or known.

With reference to Figures 5 and 6, examples of learning data and states used when generating state model 122 will be described. Figure 5 shows an example of learning data used when generating state model 122, and Figure 6 shows an example of state classification.

Drone sensor data includes acceleration, gyro, magnetism, etc. For convenience of explanation, the raw data acquired from the drone is called raw sensor data. It is desired that the state model 122 is one that can properly classify the state based on the raw sensor data. This part "want to properly classify the state" is referred to as the task for explanation. In order to improve the accuracy of the task, the analyst can process the sensor data 40 in addition to using it as it is, as shown in FIG. 5. For convenience of explanation, this is called processed sensor data 41. The raw sensor data 40 and the processed sensor data 41 can each be called a feature (feature vector). The state model 122 is constructed as a model for classifying states. For this reason, it is necessary to know in which state the sensor data acquired in advance is data, and this is called a correct answer label. Data that combines the correct answer label and the feature is called training data 42.

Figure 6 shows an example in which two feature quantities are used to classify each state into a normal state, a state where the vehicle is not moving forward, and a state where the propeller is stopped. The feature vectors shown with circles in Figure 6 indicate the correct labels for each state. Note that two feature quantities are used here for the sake of simplicity, but the raw sensor data and processed sensor data for each output value of each sensor can also be used as feature quantities.

An example of generating the state model 122 will be described in more detail with reference to Figures 7 and 8. Figure 7 is a conceptual diagram for explaining an example of the procedure for generating the state model 122 when the correct answer label is unknown, and Figure 8 is a conceptual diagram for explaining an example of the procedure for generating the state model 122 when the correct answer label is known.

When the correct label is known, the correct label is given as the answer and the model is trained. This is called supervised learning, and a model that predicts the state is generated by combining the teacher (correct label) and the feature quantity and performing machine learning. The learning algorithm (learning device) can be determined by the analyst. The learning result is output as the state model 122. If the learning device or parameters are changed and learning is performed, the same number of state models 122 can be generated. The analyst can select and use the best model from the multiple state models generated. Alternatively, the state estimation unit 12 can perform state estimation by taking a majority vote among the multiple state models 122 generated. When generating the state model 122, as described above, the method of selecting the algorithm and its parameters, the method of selecting from among the multiple state models, etc. are not important, and the same applies to the case of training other models described below.

When the correct labels are unknown, they must first be created. Clustering is one approach to this. Clustering is the process of grouping data that is close to each other into a single cluster. A correct label is then assigned to indicate the state of the data in each cluster created in this way. This correct label can be assigned by an expert, or by using a small amount of data with correct labels collected in advance. The data with correct labels created in this way becomes raw sensor data, and training data is created based on this data. The steps from this point on are the same as when the correct labels are known.

The estimation unit 121 will be described with reference to Figures 9 and 10. Figure 9 is a schematic diagram for explaining an example of state estimation, and Figure 10 is a diagram showing an example of data. The estimation unit 121 can be constructed as follows.

For convenience of explanation, the sensor data of the drone currently flying is called current data. As in the case of processing the raw sensor data to create the state model 122, the current data is also processed (converted into features) and input. As shown in FIG. 9, the estimation unit 121 inputs the features of the current data and the state model 122 and obtains an estimation result of the state as an output. A trained model for estimation can be used for this estimation, and various inference algorithms (classification algorithms) and parameters can be used as the method of this model. Also, as described above, the state model 122 can be created by changing the learning device and its parameters and learning it. The estimation unit 121 can obtain the result of state estimation by adopting the best model out of the multiple state models 122 as the state model 122 to be input to the estimation model. Alternatively, the estimation unit 121 can obtain the result of state estimation by taking a majority vote based on the result of estimation using the estimation model.

In the estimation unit 121, the state is represented as multiple values rather than two values, so the class into which the current data is classified is output as a probability. For example, as shown in FIG. 10, if there are seven samples of current data, the estimation unit 121 outputs the probability that each sample is in the normal class (0), the probability that it is in the no-movement class (1), and the probability that it is in the propeller-stopped class (2). The class (state) into which each sample is classified is determined by the class with the highest probability value.

(Abnormal condition determination unit 13)
The abnormal condition determination unit 13 will be described with reference to Figures 11 to 13. Figure 11 is a block diagram showing an example of the configuration of the abnormal condition determination unit 13.

The abnormal state determination unit 13 is a functional unit that detects when it can be determined that countermeasures or testing are necessary immediately by looking at only one state label issued by the state estimation unit 12, and requests countermeasures or testing from the countermeasure decision unit 16 and the test implementation unit 15. In this embodiment, a case in which it is determined that countermeasures or testing are necessary is called an abnormality, which means that a state requires control different from normal autonomous operation, and this is also the case in embodiment 1.

As shown in FIG. 11, the abnormal status determination unit 13 can be composed of a determination unit 131 and an abnormal status rule 132. The abnormal status rule 132 can be determined, for example, as shown in FIG. 12. FIG. 12 is a diagram showing an example of the abnormal status rule 132.

The abnormal state rule 132 is a rule for determining that each state (each state label) in the state estimation unit 12 is abnormal. This rule can be determined by a human and can be input and registered in advance before the autonomous mobile device 10 is operated. The abnormal state rule 132 defines the conditions under which a state label notified from the state estimation unit 12 should be determined to be abnormal (a state that requires control different from normal autonomous operation), and an abnormality label that indicates the type of abnormality when the condition is met. This abnormality label can also include a label (called a warning label) that causes the test implementation unit 15, described later, to perform a test.

In the example of Figure 12, for the no forward movement state label, "speeding faster than the speed limit" is used as an example of a condition rule, and "headwind abnormality" is used as an example of an abnormality label that is output as a result. Also, for the propeller stopped state label, "propeller motor not working" is used as an example of a condition rule, and "propeller failure abnormality" is used as an example of an abnormality label that is output as a result.

In the example of Figure 12, an example is given in the JSON (JavaScript Object Notation) format, but as long as the necessary items are present and their relationships can be described, the data format is not important. For example, it may be CSV (Comma Separated Value) format, XML (Extensible Markup Language), or a format in which data is stored in a table format in an RDB (Relational Database). The storage location may also be in the working memory or in the memory section of a database. Anomaly labels can be defined as names that communicate to the countermeasure decision unit 16 what type of anomaly there is, and can also be defined to request processing from the test implementation unit 15.

The judgment unit 131 judges whether or not there is an abnormality according to the rules defined in the abnormality rule 132, and issues an instruction to the test implementation unit 15 or to the countermeasure decision unit 16. If none of the abnormality rules apply, nothing is done.

The abnormality determination in the determination unit 131 will be described with reference to FIG. 13. FIG. 13 is a diagram showing an example of an abnormality determination. The example in FIG. 13 shows the case of a headwind and the case of a propeller failure.

The determination unit 131 determines that there is a headwind warning when, for example, the vehicle is moving forward, there is swaying, and the vehicle is moving at a high speed. For example, it determines that the vehicle is moving forward when the forward propulsive force applied is 40% or more of the maximum value, and that there is swaying when the left-right or up-down swaying is 60% or more of a threshold value. It also determines that the vehicle is moving at a high speed when the speed is greater than a threshold value. If these conditions overlap, the determination unit 131 determines that there is a headwind warning, and outputs a warning label indicating this. The determination unit 131 then notifies the test execution unit 15, which will be described later, of this warning label, and if the test execution unit 15 stops the forward propulsive force and performs a test, and as a result the swaying reaches 100%, it determines that there is swaying, and in that case outputs an abnormality label indicating an abnormal headwind.

The determination unit 131 determines that there is a propeller failure warning when, for example, the vehicle is moving forward, there is shaking, or there is rotation. For example, it determines that the vehicle is moving forward when the forward thrust caused by the drive is 40% or more of the maximum value, determines that there is shaking when the left-right or up-down shaking is 30% or more of the threshold value, and determines that there is rotation when any of the roll, pitch, and yaw is 30% or more of the threshold value. When these conditions overlap, the determination unit 131 determines that there is a propeller failure warning and outputs a warning label indicating this. The determination unit 131 then notifies the test execution unit 15, which will be described later, of this warning label, and when the test execution unit 15 references the history and the result exceeds the threshold value, it outputs an abnormality label indicating a propeller failure, assuming that the propeller has deteriorated. The history referenced here includes the number of warnings in the last few seconds, abnormal rotation in the last few seconds, and past collision history.

(State History Abnormality Determination Unit 14)
The status abnormality determination unit 13 will be described with reference to Fig. 14 to Fig. 17. Fig. 14 is a block diagram showing an example of the configuration of the status history abnormality determination unit 14, and Fig. 15 is a diagram showing an example of a status history abnormality rule.

The state history anomaly determination unit 14 refers to the state history, which is the history (history information indicating the background and history) of the state label estimated by the state estimation unit 12, and performs a determination as to whether or not the state is in an abnormal state. The state history anomaly determination unit 14 can also be called a countermeasure state history anomaly determination unit or a history anomaly determination unit. It is desirable that the state history referred to here includes state history for other autonomous mobile devices of the same type.

Specifically, the state history anomaly determination unit 14 is a functional unit that, when it determines that countermeasures or testing are necessary based on multiple state labels issued by the state estimation unit 12, detects this and requests countermeasures or testing from the countermeasure decision unit 16 and the test implementation unit 15.

Cases where multiple state labels are examined and detection is performed and it is determined that countermeasures or testing is necessary (this is referred to as an abnormality here, but it means that the state requires control different from normal autonomous operation) can refer to the following cases. For example, there are cases where the detection condition is the order in which state A occurs followed by state B within a certain period of time, and cases where the detection condition is a combination of state A and state B occurring in any order within a certain period of time. There are also cases where the detection condition is the number of times or frequency that state A occurs a certain number of times within a certain period of time, and cases where these three types of cases are combined. Note that the explanation was given using two states, state A and state B, but there is no intention to limit the relationship to two states, and more than two states are acceptable.

As shown in FIG. 14, the state history anomaly determination unit 14 can be composed of a determination unit 141, a state history 142, and a state history anomaly rule 143. The state history 142 receives the state label issued by the state estimation unit 12 via the determination unit 141, and holds and stores it together with a timestamp. The state history anomaly rule 143 can be determined, for example, as illustrated in FIG. 15. FIG. 15 is a diagram showing an example of the state history anomaly rule 143. Note that, in the example of FIG. 15, as in the example of FIG. 12, the data format does not matter, and the same applies to the examples of FIG. 17 and FIG. 19 described below.

The state history abnormality rule 143 is a rule for determining an abnormality from the state history for each state (each state label) in the state estimation unit 12. This rule can be determined by a person and can be input and registered in advance before the autonomous mobile device 10 starts operating. The state history abnormality rule 143 defines the conditions under which multiple state labels notified multiple times by the state estimation unit 12 should be determined to be abnormal (a state that requires control different from normal autonomous operation). The state history abnormality rule 143 also defines an abnormality label that indicates the type of abnormality when the condition is met. This abnormality label can also include a label (called a warning label or test label) that causes the test implementation unit 15, described later, to perform a test.

In the example of FIG. 15, for the state label of not moving forward, the condition rule is "occurrence 10 times or more" or "occurrence 5 times or more in 10 seconds", and in either case, the abnormal label output as a result is "headwind abnormality". For the state label of not moving forward, if the condition other than these conditions is not met and a judgment cannot be made, the test execution unit 15 is requested to perform a "stop test". For the propeller stop state label, the condition rule is "two propellers stopped" or "occurrence 2 times or more in 10 seconds", and in either case, the test execution unit 15 is requested to perform a "rotation test". For the propeller stop state label, if the condition other than these conditions is not met and a judgment cannot be made, the test execution unit 15 is requested to perform a "rotation test". For the state label of being swept to the right, the condition rule is "swept to the left last time", and the abnormal label output as a result is "propeller failure abnormality (wandering abnormality)".

In this way, the status history anomaly rules 143 define the conditions for determining whether countermeasures or testing are necessary by looking at multiple status labels. The output anomaly label can be defined as a name that conveys to the countermeasure decision unit 16 what type of anomaly it is, and can also be defined to request processing from the test implementation unit 15 (send a test instruction).

The judgment unit 141 judges whether or not the condition matches the state history anomaly rule 143, and if it matches, that is, if the above condition (detection condition) is met, it outputs an anomaly label. In other words, the judgment unit 141 judges whether or not there is an anomaly according to the rules defined in the state history anomaly rule 143, and issues an instruction to the test implementation unit 15 or issues an instruction to the countermeasure decision unit 16. If none of the state history anomaly rules 143 is met, nothing is done.

(Test Execution Unit 15)
The test execution section 15 will be described with reference to Fig. 16 and Fig. 17. Fig. 16 is a block diagram showing an example of the configuration of the test execution section 15, and Fig. 17 is a diagram showing an example of test rules.

The test execution unit 15 is a functional unit that receives a test request and performs a test operation and a test to determine whether or not there is an abnormality when the status abnormality determination unit 13 or the status history abnormality determination unit 14 determines that it is necessary to perform a test operation and determine whether or not there is an abnormality. Therefore, the output destination of the determination result by the status abnormality determination unit 13 includes the second abnormality determination unit 1d (here, the test execution unit 15).

When the determination result by the abnormal state determination unit 13 indicates an abnormal state, the test execution unit 15 executes a test to change the control of the autonomous mobile device 10 and operate it on a trial basis. The test execution unit 15 executes a determination as to whether or not a historical abnormal state exists based on the determination result by the abnormal state determination unit 13 before the test and the determination result by the abnormal state determination unit 13 after the test.

Alternatively, if the judgment result of the state abnormality judgment unit 13 indicates an abnormal state or if the judgment result of the state history abnormality judgment unit 14 indicates a history abnormal state, the test execution unit 15 executes a test to change the control of the autonomous mobile device 10 and operate it on a trial basis. In this case, the test execution unit 15 executes a final judgment as to whether or not it is a history abnormal state based on the judgment results of the state abnormality judgment unit 13 or the state history abnormality judgment unit 14 before and after the test.

In this case, it is preferable to use labels as well. That is, it is preferable for the abnormal status determination unit 13 to determine whether the current status corresponds to one of a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current status, or a normal label for which no countermeasure is required. These multiple countermeasure labels will include one or more types of test labels indicating that a test needs to be performed. When the abnormal status determination unit 13 indicates a result that corresponds to one or more test labels, the test implementation unit 15 implements a test corresponding to the test label indicated by the determination result.

As shown in FIG. 16, the test implementation unit 15 can be composed of an implementation unit 151, test rules (test implementation abnormality rules) 152, test history 153, and a result determination unit 154. The implementation unit 151 accepts test requests (or corresponding labels) from the state abnormality determination unit 13 and the state history abnormality determination unit 14, and makes a test operation request to the movement control unit 17 based on the test rules 152. The implementation unit 151 also registers a record of the test operation request in the test history 153 together with the request. The test history 153 holds a history (history of test labels) of the test operation requests made by the implementation unit 151 to the movement control unit 17 based on the test rules 152, with a timestamp.

The test rules 152 are rules for determining whether an abnormality has occurred based on tests performed for each state (each state label) of the state estimation unit 12 or for each warning label in the state history abnormality determination unit 14. These rules are determined by a person, and can be entered and registered by a person in advance before the autonomous mobile device 10 is operated. The test rules 152 define test labels, which are the names of tests to be performed, for multiple state labels notified by the state estimation unit 12 and warning labels notified by the state history abnormality determination unit 14. In addition, the test rules 152 define, for each test label, a test operation, an expected operation for the test operation, and whether the operation is normal or abnormal when it is detected, along with the abnormality label if it is abnormal. The abnormality label here is a label that indicates the type of abnormality if it corresponds to an abnormality.

In the example of Figure 17, for the test label showing test number 1 (headwind), an example of a condition rule is "If stopped, it will be blown backwards," and an example of an abnormal label that is output as a result of this condition being met is "headwind abnormality." Also, for the test label showing test number 2 (propeller failure), an example of a condition rule is "If rotated, it will not rotate," and an example of an abnormal label that is output as a result of this condition being met is "propeller failure abnormality."

The result determination unit 154 determines the state resulting from the test operation performed in response to the test operation request, and outputs the determination result to the countermeasure decision unit 16 while also registering the determination result in the test history 153. Specifically, the result determination unit 154 reads out the test rules 152 and compares the labels using the state label received from the state estimation unit 12, the warning label received from the state history anomaly determination unit 14, and the test history in the test history 153 as keys. In this way, the result determination unit 154 checks the test results and determines whether they are abnormal or normal. If it determines that they are abnormal, it makes a countermeasure request to the countermeasure decision unit 16 and sends an abnormality label. If it determines that they are normal, it does nothing.

In this embodiment, as explained in the first embodiment, the rules can be written succinctly, so that countermeasures for a certain state and the conditions for judging them can be defined succinctly, and these countermeasures can include carrying out tests. By carrying out tests, it is possible to isolate abnormalities and their causes, and thus it is possible to distinguish between true abnormalities that require the operation to be stopped and states that are similar to abnormalities but in which operation can be continued, thereby reducing overdetection of abnormalities.

In this way, in this embodiment, a human can easily define a test operation method and a method for determining the operation test result for a state label that represents the state of the autonomous mobile device 10 that is easy for humans to understand, and the operation of the autonomous mobile device 10 can be changed based on the operation test result. This makes it possible to perform a test and, based on the result, make a more precise determination as to whether an abnormality should be determined and action taken, or whether nothing needs to be done. In other words, according to this embodiment, excessive preventive measures can be reduced, contributing to the stability of operation. Furthermore, by taking testing into consideration, even in situations where excessive abnormality detections would have been made and immediate emergency action would have been taken, this need not be done, and the operational availability of the autonomous mobile device 10 can be increased.

(Countermeasures Decision Unit 16)
The countermeasure decision section 16 will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a block diagram showing an example of the configuration of the countermeasure decision section 16, and Fig. 19 is a diagram showing an example of countermeasure rules.

The countermeasure decision unit 16 receives an abnormality label as an abnormality notification from the status abnormality determination unit 13, the status history abnormality determination unit 14, and the test implementation unit 15, determines a countermeasure based on the received abnormality label against the countermeasure rule, and issues a countermeasure request to the movement control unit 17. As shown in FIG. 18, the countermeasure decision unit 16 can be composed of a decision unit 161 and the countermeasure rule 162, which is the countermeasure rule.

Countermeasure rules 162 are rules that determine which countermeasures should be taken for each abnormality (each abnormality label). These rules are determined by a person, and can be input and registered in advance before the autonomous mobile device 10 begins operation.

In the example of FIG. 19, for the headwind abnormality label, an example of a condition rule is "not reported to air traffic control in the past," and the result of this condition being met is "report that the wind is strong to air traffic control." For the headwind abnormality label, an example of a condition rule is "receive a route change order from air traffic control," and the result of this condition being met is "fly in a place where the wind is not blowing." For the headwind abnormality label, if the condition is not met and a judgment cannot be made, "land." For the propeller failure abnormality label, an example of a condition rule is "is about to crash," and the result of this condition being met is "emergency landing." For the propeller failure abnormality label, an example of a condition rule is "can fly for a little while," and the result of this condition being met is "land at a nearby landing point." For the propeller failure abnormality label, an example of a condition rule is "can fly for a little while," and the result of this condition being met is "land at a nearby landing point." For the propeller failure abnormality label, if the condition is not met and a judgment cannot be made, "open the parachute."

The decision unit 161 refers to the countermeasure rule 162 to decide on a corresponding countermeasure for the received abnormal label, and issues an instruction to the movement control unit 17. The countermeasure decision unit 16 is an example of the output unit 1e, and as in this example, can include the movement control unit 17 as its output destination. In particular, the decision unit 161 of the countermeasure decision unit 16 can be configured to output a control command corresponding to the necessary countermeasure to the movement control unit 17 as a judgment result. Of course, the movement control unit 17 may interpret the judgment result from the decision unit 161 and execute the corresponding control.

(Standard setting unit 18)
The criterion setting unit 18 sets the judgment criteria for the state history abnormality judgment unit 14 and the test implementation unit 15. The criterion setting unit 18 can be an operation unit (operation receiving unit) provided in the main body of the autonomous mobile device 1, a receiving unit that receives a setting signal so that the setting can be made by communication from an external device, or both. This allows the operator to set the above-mentioned state history abnormality rules 143 and test rules 152, and the output of processing results according to them.

The autonomous mobile device 10 can also be provided with a setting unit that sets the judgment criteria of the abnormal status judgment unit 13. Here, an example is given in which this setting unit is provided in the standard setting unit 18. This allows the operator to set the above-mentioned abnormal status rules 132 and the output of the processing results that follow them. The autonomous mobile device 10 also preferably has a countermeasure setting unit that sets control commands according to the necessary countermeasures output by the countermeasure decision unit 16. This countermeasure setting unit can be provided as part of the standard setting unit 18. This allows the operator to set the above-mentioned countermeasure rules 162 and the output (command output) of the processing results that follow them.

The state model 122 of the state estimation unit 12, the state abnormality rule 132 of the state abnormality determination unit 13, the state history abnormality rule 143 of the state history abnormality determination unit 14, the test rule 152 of the test implementation unit 15, and the countermeasure rule 162 of the countermeasure decision unit 16 can be written as follows. That is, a setting file or the like can be created in advance and written in advance from the standard setting unit 18 as rules for each part of the autonomous mobile device 10, which makes it even easier to update the settings.

(Operation Example of the Autonomous Mobile Device 10)
An example of the internal operation of the autonomous mobile device 10 will be described with reference to Figs. 20 to 25. Fig. 20 is a flow diagram for explaining an example of the operation of the autonomous mobile device 10, in which the entire sequence is expressed in the notation of UML (Unified Modeling Language) 2.0. Fig. 20 shows a flow of a series of operations that are repeatedly performed each time sensor information (detection results from the sensor group 11) is generated during operation of the autonomous mobile device 10, or each time the sensor information is notified to other functional units from the sensor group 11 at regular intervals or other timings.

In the autonomous mobile device 10, each time the sensor group 11 notifies the sensor information (step S11), the state estimation unit 12 performs state estimation, and the result is notified to one or more of the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test implementation unit 15 (step S12). Then, the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test implementation unit 15 that received the notification can perform an abnormality determination.

If the autonomous mobile device 10 finds an abnormality as a result of the determination, it sends an abnormality notification to the countermeasure decision unit 16 (steps S13, S16, S18). The countermeasure decision unit 16 receives the abnormality notification, decides on a countermeasure, and has the autonomous mobile device 10 take countermeasure action via the mobile control unit 17 (step S14), thereby taking action to deal with the abnormal operation.

If the status abnormality determination unit 13 determines that there is an abnormality (warning) that requires testing as a countermeasure, it sends a test request to the test execution unit 15 (step S15). If the status history abnormality determination unit 14 determines that there is an abnormality (warning) that requires testing as a countermeasure, it sends a test request to the test execution unit 15 (step S110). The test execution unit 15 receives the test request and sends a test operation request to the movement control unit 17 (step S113), and then the abnormality is determined.

Next, an example of the internal operation of the state estimation unit 12 will be described with reference to FIG. 21. FIG. 21 is a flow diagram for explaining an example of the operation of the state estimation unit 12 in FIG. 4. Prior to this series of processes, the state model has already been stored with a state model trained with each state as a correct label.

The estimation unit 121 receives a sensor information notification (step S11) from the sensor group 11 (step S21), estimates the state based on the received sensor information notification against the state model 122, and obtains state label information (step S22). When the estimation unit 121 receives a state notification (step S12), it transmits the state label information to the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test implementation unit 15 (step S23), and ends the process.

An example of the internal operation of the abnormal state determination unit 13 will be described with reference to FIG. 22. FIG. 22 is a flow diagram for explaining an example of the operation of the abnormal state determination unit 13 of FIG. 11.

The judgment unit 131 receives a state notification (step S12) from the state estimation unit 12 and receives state label information (step S31). The judgment unit 131 checks the state label information against the state abnormality rules 132, and if there is a match, obtains an abnormality label or a test label (step S32). Here, if there is no match, nothing is obtained.

Then, the judgment unit 131 judges whether a label has been obtained (step S33), and if NO, ends the process, and if YES, judges the type of the obtained label (step S34). In step S34, it is judged whether the label is an abnormal label or a test label. If an abnormal label has been obtained, the judgment unit 131 notifies the countermeasure decision unit 16 of the abnormal label by an abnormality notification (step S13) (step S35), and ends the process. If a test label has been obtained, the judgment unit 131 notifies the test implementation unit 15 of the test label by a test request (step S15) (step S36), and ends the process.

An example of the internal operation of the state history anomaly determination unit 14 will be described with reference to FIG. 23. FIG. 23 is a flow diagram for explaining an example of the operation of the state history anomaly determination unit 14 in FIG. 14.

The determination unit 141 receives a sensor information notification (step S11) from the sensor group 11 (step S41), and adds the received sensor information notification to the state history 142 (step S42). Next, the determination unit 141 reads out the history of the sensor information notification from the state history 142, and, if there is a match, obtains an abnormal label or a test label in light of the rules read out from the state history anomaly rules 143 (step S43). Here, if there is no match, nothing is obtained.

Then, the judgment unit 141 judges whether a label has been obtained (step S44), and if NO, ends the process, and if YES, judges the type of the obtained label (step S45). In step S45, it is determined whether the label is an abnormal label or a test label. If an abnormal label has been obtained, the judgment unit 141 notifies the countermeasure decision unit 16 of the abnormal label in an abnormality notification (step S13) (step S46) and ends the process. The judgment unit 141 notifies the test implementation unit 15 of the test label in a test request (step S15) (step S47) and ends the process.

An example of the internal operation of the test execution unit 15 will be described with reference to FIG. 24. FIG. 24 is a flow diagram for explaining an example of the operation of the test execution unit 15 in FIG. 16.

The implementation unit 151 receives a test request from the state abnormality determination unit 13 (step S15) or a test request from the state history abnormality determination unit 14 (step S110) (step S51). Next, the implementation unit 151 refers to the test rules 152 and reads out a test rule corresponding to the test request from the test rules 152 (step S52). The implementation unit 151 requests the movement control unit 17 to perform a test operation of the autonomous mobile device 10 according to the read test rule by a test operation request (step S113), and registers the test operation in the test history 153 (step S53).

Next, the result determination unit 154 receives a state notification (step S12) from the state estimation unit 12 (step S54). In step S54, the operation of the autonomous mobile device 10 in response to the test request is received. Next, the result determination unit 154 compares the received state notification (step S12) with the pair of the test history 153 and the test rule 152 (step S55) and determines whether the test result is in accordance with the test rule (normal) or not (step S56). If the result is YES in step S56, the result determination unit 154 ends the process. If the result is NO in step S56, that is, if the test result differs from the test rule (is abnormal), the result determination unit 154 notifies the countermeasure decision unit 16 of the abnormality label in the abnormality notification (step S16) (step S57) and ends the process.

An example of the internal operation of the countermeasure decision unit 16 will be described with reference to FIG. 25. FIG. 25 is a flow diagram for explaining an example of the operation of the countermeasure decision unit 16.

The decision unit 161 receives either the abnormality notification from the test implementation unit 15 (step S16), the abnormality notification from the state abnormality determination unit 13 (step S13), or the abnormality notification from the state history abnormality determination unit 14 (step S18) (step S61). Next, the decision unit 161 determines a countermeasure based on the received abnormality notification (steps S16, S13, S18) against the countermeasure rule 162 (step S62). Then, the decision unit 161 transmits a countermeasure request (steps S14, S17) to the movement control unit 17, and the movement control unit 17 issues a control instruction for the countermeasure operation to the autonomous mobile device 10 (step S63), terminating the process.

As described above, according to this embodiment, like the first embodiment, it is possible to detect abnormal conditions that require countermeasures using the judgment conditions desired by the operator of the autonomous mobile device 10, and also to achieve the effects of the components individually provided. For example, by providing at least one of the state history abnormality judgment unit 14 and the test implementation unit 15, it is possible to improve the accuracy of detection of abnormal conditions and increase the availability of the autonomous mobile device 10.

In addition, in this embodiment, even if the autonomous mobile device 10 is configured without the test implementation unit 15, the state abnormality determination unit 13, or the state history abnormality determination unit 14, it is possible to maintain a certain level of accuracy in detecting abnormal states. Even in this case, the advantage that the operator can easily define abnormality determination criteria and countermeasures using the criteria setting unit 18 is maintained.

In addition, one or more of the status abnormality rules 132, the status history abnormality rules 143, and the countermeasure rules 162 may be replaced with a status abnormality model, a status history abnormality model, and a countermeasure rule model, respectively. Any of the models can be generated, for example, using the same method as the generation method of the status model 122. In this case, since the operator needs to train the model instead of creating each definition file, it is preferable to make at least one rule easily settable from the standard setting unit 18.

<Embodiment 3>
The third embodiment will be described with reference to Fig. 26 and Fig. 27, focusing on the differences from the second embodiment. The third embodiment can appropriately use various examples described in the first and second embodiments. Fig. 26 is a block diagram showing one configuration example of an autonomous mobile system according to the third embodiment, and Fig. 27 is a block diagram showing another configuration example of the autonomous mobile system according to the third embodiment.

The autonomous mobile system according to this embodiment shown in FIG. 26 includes an autonomous mobile device 10 and a server device 50 that is connected to the autonomous mobile device 10 and manages the operation of the autonomous mobile device 10. The server device 50 can have a management unit (operation management unit) 51 that manages the operation of one or more autonomous mobile devices 10. The method of management by the operation management unit 51 does not matter as long as it can manage the operation of the autonomous mobile device 10. The operation management unit 51 manages the operation of multiple autonomous mobile devices 10, for example, in a manner similar to air traffic control in an aircraft.

The server device 50 can also include a standard setting unit 18. This allows the state model 122 of the state estimation unit 12, the state abnormality rule 132 of the state abnormality determination unit 13, the state history abnormality rule 143 of the state history abnormality determination unit 14, the test rule 152 of the test implementation unit 15, and the countermeasure rule 162 of the countermeasure decision unit 16 to be registered from the server device 50. These can also be rewritten from the server device 50. In this way, the server device 50 can be provided with a standard setting unit 18, and configured to allow definitions to be added, deleted, or edited, for example, from a user-oriented setting screen. In this case, the standard setting unit 18 on the autonomous mobile device 10 side can also be omitted.

The operation management unit 51 can be configured as a control unit (not shown) that controls the entire server device 50 and a communication unit (not shown) that communicates with the autonomous mobile device 10. This control unit can be realized, for example, by a CPU, working memory, and a non-volatile storage device that stores a program. This program can be a operation management program that executes processing to realize the functions of the operation management unit 51 in cooperation with the communication unit. This control unit can also be realized, for example, by an integrated circuit.

The autonomous mobile system according to this embodiment shown in FIG. 27 includes an autonomous mobile device 10a and a server device 50a that is connected to the autonomous mobile device 10a and manages the operation of the autonomous mobile device 10a. The server device 50a can have an operation management unit 51 that manages the operation of one or more autonomous mobile devices 10a. The method of management by the operation management unit 51 does not matter as long as it can manage the operation of the autonomous mobile device 10a.

The server device 50a also includes a state estimation unit 12, a state abnormality determination unit 13, a state history abnormality determination unit 14, a test implementation unit 15, and a countermeasure decision unit 16, and these functions are removed from the autonomous mobile device 10 in the autonomous mobile device 10a. In other words, the autonomous mobile device 10a includes a sensor group 11 and a mobile control unit 17, and executes processes such as transmitting information from the sensor group 11 and receiving command signals to the mobile control unit 17 while communicating with the server device 50a through a communication unit (not shown). Therefore, the server device 50a does not include a sensor group, but includes a receiving unit (communication unit) that receives detection results obtained by the sensor group 11 that detects the current state of the autonomous mobile device 10a from the autonomous mobile device 10a. The server device 50a can also transmit command signals to the mobile control unit 17 through this communication unit.

In addition, each unit such as the operation management unit 51, the state estimation unit 12, etc. can be configured as a control unit (not shown) that controls the entire server device 50a and a communication unit (not shown) that communicates with the autonomous mobile device 10a, similar to the case described for the server device 50.

In addition, while the server device 50a has been shown as having the state estimation unit 12, the state abnormality determination unit 13, the state history abnormality determination unit 14, the test implementation unit 15, and the countermeasure decision unit 16, it may have only one or more of these components, with the remaining components being provided on the autonomous mobile device 10a side. In addition, the autonomous mobile system may be constructed as a system that does not have the operation management unit 51 in the configuration example of FIG. 26 or FIG. 27.

<Other embodiments>
[a]
In each embodiment, the functions of an autonomous mobile device, a learning device, a server device, etc. have been described, but each device is not limited to the configuration examples shown, and it is sufficient if each device can realize these functions.

[b]
Each device according to each embodiment may have the following hardware configuration. Fig. 28 is a diagram showing an example of the hardware configuration of the device. The same applies to the other embodiment [a] above.

The device 100 shown in FIG. 28 may have a processor 101, a memory 102, and an interface 103. The processor 101 may be, for example, a microprocessor, a microprocessor unit (MPU), or a CPU. The processor 101 may include multiple processors. The memory 102 is, for example, configured by a combination of a volatile memory and a non-volatile memory. The functions of each device described in each embodiment are realized by the processor 101 reading and executing a program stored in the memory 102. At this time, input and output of information can be performed via an interface 103 such as a communication interface that communicates with other internal parts or other external devices. For example, if the device 100 is an autonomous mobile device, the interface 103 can include at least an interface with each sensor.

In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[c]
Further, as the procedures of the processing in the autonomous mobile device or the server device are exemplified in each of the above-mentioned embodiments, the present disclosure also includes a form as an anomaly detection method in the autonomous mobile device or the server device. This anomaly detection method can include the following acquisition step, state estimation step, first anomaly determination step, second anomaly determination step, output step, and reference setting step. The acquisition step acquires a result of the current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device. The state estimation step estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the group of sensors. The first anomaly determination step determines whether or not the current state is an abnormal state that requires measures to change the state based on the estimation result in the state estimation step. The second anomaly determination step determines whether or not the state indicated by the results of the multiple estimations at different times in the state estimation step is a historical abnormal state that requires measures to change the state. The output step outputs the determination result in the first anomaly determination step and the determination result in the second anomaly determination step. The above-mentioned criterion setting step sets a criterion for the second abnormality determination step. Note that other examples are as described in the above-mentioned embodiments. In addition, the present disclosure may include a safe automatic control method for an autonomous mobile device including such an abnormality detection method.

The program for the autonomous mobile device can be said to be a program for causing a computer (control computer) provided within the autonomous mobile device to execute the above-mentioned acquisition step, state estimation step, first abnormality determination step, second abnormality determination step, output step, and reference setting step. The program for the server device can be said to be a program for causing a computer connected to the autonomous mobile device to execute the above-mentioned acquisition step, state estimation step, first abnormality determination step, second abnormality determination step, output step, and reference setting step.

As the processing procedures in the learning device have been described in each of the above-mentioned embodiments, the present disclosure also includes a form as a learning method in a learning device. This learning method includes the following acquisition step and generation step. This acquisition step acquires detection results acquired by a group of sensors that detect the current state of the autonomous mobile device. This generation step inputs the detection results acquired in the acquisition step multiple times, and generates a trained model for outputting countermeasure labels, which outputs one of multiple countermeasure labels and a normal label. Note that other examples are as described in each of the above-mentioned embodiments. In addition, it can be said that the program related to the learning device is a program for causing a computer provided in the autonomous mobile device or connected to the autonomous mobile device to execute the above acquisition step and generation step.

Note that this disclosure is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, this disclosure can be implemented by combining the respective embodiments as appropriate.

A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
(Appendix 1)
A group of sensors that detects a current state of the autonomous mobile device;
a state estimation unit that estimates which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors;
a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit;
a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit;
an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit;
a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
Autonomous mobility device.
(Appendix 2)
the second anomaly determination unit performs determination using a first trained model that receives a result estimated a plurality of times at different times by the state estimation unit, receives an estimation result by the state estimation unit, and outputs one of the plurality of countermeasure labels and the normal label;
2. The autonomous mobile device of claim 1.
(Appendix 3)
The second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of the state label estimated by the state estimation unit.
3. The autonomous mobile device according to claim 1 or 2.
(Appendix 4)
The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device.
4. The autonomous mobile device of claim 3.
(Appendix 5)
An output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit,
The second abnormality judgment unit has a test implementation unit that, when the judgment result of the first abnormality judgment unit indicates the abnormal state or the judgment result of the history abnormality judgment unit indicates the history abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a final judgment of whether or not the history abnormality state exists based on the judgment result of the first abnormality judgment unit or the history abnormality judgment unit before the test and the judgment result of the first abnormality judgment unit or the history abnormality judgment unit after the test.
5. The autonomous mobile device according to claim 3 or 4.
(Appendix 6)
An output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit,
The second abnormality determination unit has a test execution unit that, when the determination result by the first abnormality determination unit indicates an abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a determination as to whether or not the historical abnormality state exists based on the determination result by the first abnormality determination unit before the test and the determination result by the first abnormality determination unit after the test.
5. The autonomous mobile device according to claim 1 .
(Appendix 7)
the first abnormality determination unit determines whether the current state corresponds to one of a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label indicating that no countermeasure is required;
The plurality of countermeasure labels include one or more types of test labels indicating that the test needs to be performed,
the test implementation unit, when the first abnormality determination unit indicates a result that corresponds to the one or more test labels, implements a test corresponding to the test label indicated by the determination result.
7. The autonomous mobile device according to claim 5 or 6.
(Appendix 8)
the first abnormality determination unit determines whether the current state corresponds to a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label indicating that no countermeasure is required;
8. The autonomous mobile device according to claim 1.
(Appendix 9)
The first abnormality determination unit performs a determination using a second trained model that receives an estimation result from the state estimation unit and outputs one of the plurality of countermeasure labels and the normal label.
9. The autonomous mobile device of claim 8.
(Appendix 10)
A setting unit that sets a judgment criterion for the first abnormality judgment unit,
10. The autonomous mobile device according to any one of claims 1 to 9.
(Appendix 11)
A movement control unit that controls movement of the autonomous moving device,
The output unit outputs a control command corresponding to a necessary measure to the movement control unit as a determination result.
11. The autonomous mobile device according to any one of claims 1 to 10.
(Appendix 12)
A countermeasure setting unit that sets a control command corresponding to the necessary countermeasure output by the output unit,
12. The autonomous mobile device of claim 11.
(Appendix 13)
The state estimation unit performs estimation using a third trained model that receives a detection result by the group of sensors and outputs one of the plurality of state labels.
13. The autonomous mobile device according to any one of claims 1 to 12.
(Appendix 14)
a receiving unit that receives, from the autonomous mobile device, detection results acquired by a group of sensors that detect a current state of the autonomous mobile device;
a state estimation unit that estimates to which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds based on a detection result by the group of sensors;
a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit;
a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit;
an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit;
a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
Server device.
(Appendix 15)
The autonomous mobile device further includes a management unit that manages the operation of the autonomous mobile device.
15. The server device according to claim 14.
(Appendix 16)
16. An autonomous mobile system comprising: the server device according to claim 14 or 15; and an autonomous mobile device equipped with the group of sensors.
(Appendix 17)
An autonomous mobile system comprising: an autonomous mobile device according to any one of claims 1 to 13; and a server device connected to the autonomous mobile device and managing the operation of the autonomous mobile device.
(Appendix 18)
an acquisition unit that acquires detection results acquired by a group of sensors that detect a current state of the autonomous mobile device;
A control unit;
Equipped with
The control unit inputs the detection results acquired by the acquisition unit multiple times, and generates a trained model for outputting countermeasure labels, which outputs one of a plurality of countermeasure labels and a normal label;
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
Learning device.
(Appendix 19)
the control unit receives the detection result acquired by the acquisition unit, and outputs one of a plurality of predetermined state labels to indicate a current state of the autonomous mobile device, and generates a trained model for outputting a state label;
19. The learning device of claim 18.
(Appendix 20)
an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device;
a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors;
a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step;
a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step;
an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step;
a criterion setting step of setting a judgment criterion for the second abnormality judgment step;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
Anomaly detection methods.
(Appendix 21)
an acquisition step of acquiring detection results acquired by a group of sensors that detect a current state of the autonomous mobile device;
A generation step of generating a trained model for outputting a countermeasure label, the trained model being configured to input a plurality of detection results acquired in the acquisition step and output one of a plurality of countermeasure labels and a normal label;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
How to learn.
(Appendix 22)
A computer provided in the autonomous mobile device or connected to the autonomous mobile device,
an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device;
a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors;
a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step;
a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step;
an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step;
a criterion setting step of setting a judgment criterion for the second abnormality judgment step;
Run the command,
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
program.
(Appendix 23)
A computer provided in the autonomous mobile device or connected to the autonomous mobile device,
an acquisition step of acquiring detection results acquired by a group of sensors that detect a current state of the autonomous mobile device;
A generation step of generating a trained model for outputting a countermeasure label, the trained model being configured to input a plurality of detection results acquired in the acquisition step and output one of a plurality of countermeasure labels and a normal label;
Run the command,
the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.
program.

1, 10, 10a Autonomous mobile device 1a, 11 Sensor group 1b, 12 State estimation unit 1c First abnormality determination unit 1d Second abnormality determination unit 1e Output unit 1f, 18 Standard setting unit 2 Learning device 2a Control unit 2b Control unit 2c Memory unit 2d Unlearned model 2e Learned model 13 State abnormality determination unit 14 State history abnormality determination unit 15 Test implementation unit 16 Countermeasure decision unit 17 Mobile control unit 40 Raw sensor data 41 Processed sensor data 42 Learning data 50, 50a Server device 51 Traffic management unit 100 Device 101 Processor 102 Memory 103 Interface

Claims (7)

A group of sensors that detects a current state of the autonomous mobile device;
a state estimation unit that estimates to which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds based on a detection result by the group of sensors;
a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit;
a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit;
an output unit that outputs a determination result of the first abnormality determination unit;
a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure;
the second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of state labels estimated by the state estimation unit,
The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device.
Autonomous mobility device. An output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit,
The second abnormality judgment unit has a test implementation unit that, when the judgment result of the first abnormality judgment unit indicates the abnormal state or the judgment result of the history abnormality judgment unit indicates the history abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a final judgment of whether or not the history abnormality state exists based on the judgment result of the first abnormality judgment unit or the history abnormality judgment unit before the test and the judgment result of the first abnormality judgment unit or the history abnormality judgment unit after the test.
The autonomous mobile device according to claim 1 . the first abnormality determination unit determines whether the current state corresponds to one of a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label indicating that no countermeasure is required;
The plurality of countermeasure labels include one or more types of test labels indicating that the test needs to be performed,
the test implementation unit, when the first abnormality determination unit indicates a result that corresponds to the one or more test labels, implements a test corresponding to the test label indicated by the determination result.
The autonomous mobile device according to claim 2 . A movement control unit that controls movement of the autonomous moving device,
The output unit outputs a control command corresponding to a necessary measure to the movement control unit as a determination result.
The autonomous mobile device according to any one of claims 1 to 3 . a receiving unit that receives, from the autonomous mobile device, detection results acquired by a group of sensors that detect a current state of the autonomous mobile device;
a state estimation unit that estimates to which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds based on a detection result by the group of sensors;
a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit;
a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit;
an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit;
a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures in order to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure;
the second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of state labels estimated by the state estimation unit,
The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device.
Server device. an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device;
a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors;
a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step;
a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step;
an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step;
a criterion setting step of setting a judgment criterion for the second abnormality judgment step;
Equipped with
the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure;
The second abnormality determination step includes a history abnormality determination step of determining whether or not the state is the history abnormal state by referring to a state history that is a history of the state label estimated in the state estimation step,
The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device.
Anomaly detection methods. A computer provided in the autonomous mobile device or connected to the autonomous mobile device,
an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device;
a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors;
a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step;
a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step;
an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step;
a criterion setting step of setting a judgment criterion for the second abnormality judgment step;
Run the command,
the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure;
The second abnormality determination step includes a history abnormality determination step of determining whether or not the state is the history abnormal state by referring to a state history that is a history of the state label estimated in the state estimation step,
The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device.
program.

Images