JP7640189B2 - AUTONOMOUSLY MOVING WORK DEVICE, METHOD AND PROGRAM FOR MODIFYING WORK PLAN - Google Patents

Description

The present invention relates to an autonomous mobile work device, a method for modifying a work plan, and a program.

An autonomous mobile work device that visualizes the working condition of a work surface is known (see, for example, Patent Document 1). The autonomous mobile work device described in Patent Document 1 is communicatively connected to a mobile terminal, and a dirt detection sensor is provided on the autonomous mobile work device, and a camera is provided on the mobile terminal. A map of the work surface is created using SLAM (Simultaneous Localization and Mapping) or the like, and the detection results of the dirt detection sensor are added to the map. The map reflecting the working condition is then transmitted from the autonomous mobile work device to the mobile terminal, and this map is superimposed on an image captured by the camera and displayed on the display of the mobile terminal.

JP 2019-082807 A

In the autonomous mobile work device described in Patent Document 1, a map reflecting the work state is overlaid on the captured image of the work surface, making it possible to visualize the work state of the work surface. While it is possible to make the operator aware of the work state of the work surface, it is not possible to move the autonomous mobile work device so as to obtain work results that reflect the operator's intentions.

The present invention has been made in consideration of these points, and aims to provide an autonomous mobile work device, a method for modifying a work plan, and a program that can reflect the operator's intentions in the work plan, thereby improving work efficiency and work quality.

The autonomous mobile work device of the present invention is an autonomous mobile work device that performs work on a work surface while traveling autonomously according to a work plan, and is equipped with a work trajectory creation unit that creates a work trajectory of the work that has been completed and/or a virtual work trajectory of the work to be completed when working according to the work plan, a display control unit that overlays and displays the work trajectory and/or the virtual work trajectory on the work surface in an image acquired from an imaging unit, and a work plan correction unit that corrects the work plan when a change operation is received regarding the work trajectory and/or the virtual work trajectory, wherein the work trajectory creation unit creates the work trajectory based on the travel path from the start point of the work to the current point and the width of the work member , and creates the virtual work trajectory based on the travel path from the current point to the end point of the work and the width of the work member, and the travel path is offset due to changes made to the work trajectory and/or the virtual work trajectory displayed overlaid on the work surface in the image, and the work trajectory and/or the virtual work trajectory are re-displayed on the image.

The method of modifying a work plan of the present invention is a method of modifying a work plan of an autonomous mobile work device that performs work on a work surface while traveling autonomously, and includes the steps of creating a work trajectory of completed work and/or a virtual work trajectory of planned work when work is performed according to the work plan, a step of overlaying and displaying the work trajectory and/or the virtual work trajectory on the work surface in an image acquired from an imaging unit, and a step of modifying the work plan when a change operation is received regarding the work trajectory and/or the virtual work trajectory, in which in the creating step, the work trajectory is created based on the travel path from the start point of the work to the current point and the width of the work member , and the virtual work trajectory is created based on the travel path from the current point to the end point of the work and the width of the work member, and the travel path is offset due to changes made to the work trajectory and/or the virtual work trajectory displayed overlaid on the work surface in the image acquired , and the work trajectory and/or the virtual work trajectory are re-displayed on the image acquired.

According to these configurations, the operator is allowed to visually confirm the work trajectory and/or virtual work trajectory on the work surface in the captured image, and the operator's operation to change the work trajectory and/or virtual work trajectory is accepted. The work plan of the autonomous mobile work device is modified by the operation to change the work trajectory and/or virtual work trajectory, and the operator's intentions can be reflected in the work plan, improving work efficiency and work quality.

The above-mentioned autonomous mobile work device includes a work trajectory division unit that divides the completed work trajectory and/or the virtual work trajectory into a plurality of divided trajectories, the display control unit displays the plurality of divided trajectories superimposed on the work surface in the captured image, and the work plan correction unit corrects the work plan when a change operation is made to the divided trajectories. With this configuration, the operator's intentions can be reflected in the work plan in more detail.

The above-mentioned autonomous mobile work device learns a work plan according to manual operation, and performs work on the work surface while autonomously traveling according to the learned work plan, the display control unit displays the trajectory of the work performed on the work surface in the captured image after learning the work plan, and the work plan correction unit corrects the learned work plan when a change operation is received regarding the trajectory of the work performed. With this configuration, the operator's intentions can be reflected in the learned work plan without the autonomous mobile work device having to repeatedly learn by manual operation.

Another autonomous mobile work device of the present invention is an autonomous mobile work device that learns a work plan according to manual operation and performs work on a work surface while autonomously traveling according to the learned work plan, and is equipped with a work trajectory creation unit that creates a work trajectory that has already been completed when work is performed according to manual operation while learning the work plan and a virtual work trajectory of the planned work predicted from the work trajectory, a display control unit that displays the work trajectory and the virtual work trajectory superimposed on the work surface in the captured image acquired from the imaging unit while learning the work plan, and a work plan creation unit that creates a work plan for the virtual work trajectory, which is an unlearned portion, while learning the work plan.

With this configuration, the operator is allowed to visually check the virtual work trajectory on the work surface in the captured image while the work plan is being learned, and the operator is allowed to confirm whether the virtual work trajectory, which is an unlearned portion, is appropriate. This prevents the generation of a work plan that goes against the operator's intentions, and allows the operator's intentions to be reflected in the work plan for the virtual work trajectory without having to repeatedly learn the autonomous mobile work device through manual operation.

The program of the present invention is a program for an autonomous mobile work device that performs work on a work surface while traveling autonomously according to a work plan, and causes the autonomous mobile work device to execute the following steps: creating a work trajectory of completed work and/or a virtual work trajectory of planned work when working according to the work plan; overlaying and displaying the work trajectory and/or the virtual work trajectory on the work surface in an image obtained from an imaging unit; and modifying the work plan when a change operation is received regarding the work trajectory and/or the virtual work trajectory.In the creating step, the work trajectory is created based on the travel path from the start point of the work to the current point and the width of the work member , and the virtual work trajectory is created based on the travel path from the current point to the end point of the work and the width of the work member , and the travel path is offset by changes to the work trajectory and/or the virtual work trajectory displayed overlaid on the work surface in the image , and the work trajectory and/or the virtual work trajectory are re-displayed on the image. According to this configuration, by installing a program in the autonomous mobile work device, a function can be added that allows the operator to visually view the previous work trajectory and/or the virtual work trajectory on the work surface in the captured image, prompting the operator to revise the work plan.

According to the present invention, the operator's intentions can be reflected in the work plan, improving work efficiency and quality.

FIG. 2 is a perspective view of the cleaning device according to the embodiment. FIG. 2 is a schematic diagram of a cleaning device according to the present embodiment. FIG. 2 is a control block diagram of the main body of the apparatus according to the embodiment. FIG. 4 is a diagram showing an example of a display screen of the present embodiment. FIG. 13 is a diagram showing an example of a display screen before cleaning in the present embodiment. 11A to 11C are diagrams illustrating an example of a process for designating an intensive cleaning area according to the present embodiment. 11A to 11C are diagrams illustrating an example of a process for designating an intensive cleaning area according to the present embodiment. 13 is a diagram showing the positional relationship of the cleaning device after learning of the cleaning plan of the present embodiment. FIG. FIG. 13 is a diagram showing an example of a display screen after learning in the present embodiment. FIG. 13 is a diagram showing an example of a display screen during learning of a cleaning plan in the present embodiment. 13 is an example of a flowchart of a cleaning plan correction process according to the present embodiment. 13 is an example of a flowchart of a process for generating a cleaning plan during learning according to the present embodiment.

The cleaning device of this embodiment will be described below with reference to the drawings. FIG. 1 is a perspective view of the cleaning device of this embodiment. FIG. 2 is a schematic diagram of the cleaning device of this embodiment. In the following description, an autonomously traveling cleaning device will be used as an example of the autonomously traveling work device, but the autonomously traveling work device may be a work device that performs various tasks such as cleaning, polishing, polishing, and waxing on a work surface by autonomous traveling.

As shown in FIG. 1, the cleaning device 1 is configured to be capable of autonomously cleaning floor surfaces (work surfaces) F in commercial facilities, manufacturing plants, train station concourses, etc. Front wheels 12 as drive wheels and rear wheels 13 as auxiliary wheels are provided at the bottom of the device body 10 of the cleaning device 1. Cleaning members 16 such as cleaning pads and cleaning brushes are provided between the front wheels 12 and rear wheels 13, and a squeegee 17 is provided behind the cleaning members 16. A handle 28 is provided at the upper rear side of the device body 10, and a display unit 22 is provided in front of the handle 28. An imaging unit 21 is provided at the upper front side of the device body 10, and an obstacle sensor 26 is provided in a recess below the imaging unit 21.

As shown in FIG. 2, the device body 10 of the cleaning device 1 is provided with a running unit 11, a cleaning unit 15, a wireless communication unit 19, an imaging unit 21, a display unit 22, a measuring unit 25, an operation unit 27, a memory unit 50, a power supply unit 29, and a control unit 30. The running unit 11 is composed of the above-mentioned front wheels 12, rear wheels 13, a running motor (not shown), an encoder (not shown), etc. The running unit 11 drives the front wheels 12 with the running motor to run the cleaning device 1. Note that, although a front-wheel-drive cleaning device 1 is exemplified in this embodiment, only the rear wheels 13 may be driven by the running unit 11, or both wheels may be driven by the running unit 11.

The cleaning unit 15 cleans the floor surface F, for example, by wet cleaning, and is composed of the above-mentioned cleaning member 16 and squeegee 17, a cleaning motor (not shown), an actuator (not shown), a cleaning liquid supply unit (not shown), a wastewater recovery unit (not shown), etc. The cleaning unit 15 presses the cleaning member 16 against the floor surface F using the actuator, and rotates the cleaning member 16 relative to the floor surface F using the cleaning motor. At this time, cleaning liquid is sprayed from a cleaning liquid tank onto the floor surface F by the cleaning liquid supply unit, and wastewater collected by the squeegee 17 is collected in a wastewater tank by the wastewater recovery unit.

The wireless communication unit 19 connects the device main body 10 and the management terminal (not shown) via wireless communication. For example, the device main body 10 and the management terminal are connected via a wireless LAN such as WiFi (registered trademark). The imaging unit 21 is a dome-shaped camera capable of capturing an image of the floor surface F. The imaging unit 21 may be any camera capable of capturing an image of the floor surface F, and may be a spherical camera capable of capturing a panoramic image, or a fixed camera capable of capturing an image in front of the cleaning device 1. In this embodiment, the cleaning device 1 is equipped with the imaging unit 21, but the cleaning device 1 may also obtain an image of the floor surface F via wireless communication from a surveillance camera installed in a commercial facility or the like.

The display unit 22 has a touch panel type display screen. The display screen of the display unit 22 displays the captured image (mainly video) captured by the imaging unit 21, as well as an operation screen that accepts operations by the operator on the cleaning device 1. The operation screen allows the operation mode of the cleaning device 1 to be selected, and various settings to be input to the cleaning device 1. Note that a tablet terminal or a mobile terminal such as a smartphone that is detachable from the device main body 10 can also be used as the display unit 22. When a mobile terminal is used as the display unit 22, the wireless communication function of the mobile terminal may be used instead of the wireless communication unit 19.

The measuring unit 25 is equipped with the obstacle sensor 26 described above. The obstacle sensor 26 is, for example, configured with an LRF (Laser Range Finder) that measures the distance and angle from an obstacle or a wall surface. Laser light is emitted from the obstacle sensor 26 into the surroundings, and the distance and angle from the surrounding obstacle sensor 26 to an obstacle is measured by receiving reflected light from an object, making it possible to recognize the surrounding environment of the cleaning device 1. Furthermore, instead of detecting obstacles using laser light, the obstacle sensor 26 may perform obstacle detection using captured images of the surrounding environment.

The operation unit 27 receives manual operation by the operator and is composed of the above-mentioned handle 28 (see FIG. 1), throttle, etc. The operation unit 27 converts the steering amount of the handle 28 and the twist amount of the throttle into an electrical signal, which is output to the control board of the travel motor of the device main body 10 to drive the travel motor. The operation of the operation unit 27 adjusts the travel speed of the device main body 10 and allows left and right turns. In addition, a power switch (not shown) for turning the power ON/OFF and an emergency stop switch (not shown) for completely stopping the operation of the device main body 10 are provided near the operation unit 27 of the device main body 10.

The storage unit 50 is composed of one or more storage media such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), flash memory, etc. depending on the application. The ROM stores, for example, a program for controlling the cleaning device 1. The HDD and flash memory store, for example, a cleaning plan in which various cleaning conditions such as the travel path, travel speed, amount of cleaning liquid sprayed, and contact pressure of the cleaning member 16 are set, as well as other parameters. The power supply unit 29 is composed of a battery (not shown) and a charging circuit, etc. Power is supplied to each part of the device from the power supply unit 29.

The control unit 30 controls each part of the device. The various processes of the control unit 30 may be realized by software using a processor, or may be realized by a logic circuit (hardware) formed in an integrated circuit or the like. When a processor is used, the processor reads out and executes programs stored in the storage unit 50 to perform various processes. For example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) is used as the processor. The control blocks of the control unit 30 and the storage unit 50 will be described in detail later.

Such a cleaning device 1 is provided with three operating modes: manual mode, learning mode, and reproduction mode. In the manual mode, the cleaning device 1 is manually operated by an operator to clean the floor surface F. In the learning mode, the cleaning device 1 is manually operated by an operator to learn a cleaning plan. In the reproduction mode, the cleaning device 1 cleans while autonomously traveling within the cleaning area so as to reproduce the cleaning plan. In addition to cleaning according to the cleaning plan learned in the learning mode, the cleaning device 1 can also clean according to a cleaning plan created in advance.

However, a learned cleaning plan or a cleaning plan created in advance is not necessarily appropriate. For example, cleaning efficiency decreases if the distance from the cleaning trajectory of the cleaning device 1 to the wall is longer than the safe distance, or if the overlap width of adjacent cleaning trajectories is too large. Also, depending on the degree of dirt on the floor surface F, cleaning according to the cleaning plan alone may not be able to sufficiently remove the dirt. Therefore, the cleaning device 1 of this embodiment allows the operator to visually recognize the cleaning trajectory on the floor surface F in the captured image, and modifies the cleaning plan to reflect the operator's intentions so that cleaning efficiency and cleaning quality can be improved.

The detailed configuration of the control unit and storage unit of the device body will be described below. Figure 3 is a control block diagram of the device body of this embodiment. Note that the symbols in Figures 1 and 2 will be used appropriately in the description.

As shown in FIG. 3, the control unit 30 includes a learning cleaning control unit 31, a learning travel control unit 32, a SLAM control unit 33, a travel route creation unit 34, an environmental map creation unit 35, a cleaning plan creation unit 36, a reproduction cleaning control unit 37, a reproduction travel control unit 38, a previously cleaned track creation unit 39, a virtual cleaning track creation unit 40, an AR display control unit 41, a previously cleaned track calculation unit 42, a virtual cleaning track calculation unit 43, an uncleaned area calculation unit 44, a cleaning plan correction unit 45, and a cleaning track division unit 46. The memory unit 50 includes a cleaning condition memory unit 51, a travel route memory unit 52, an environmental map memory unit 53, a cleaning plan memory unit 54, a cleaning track memory unit 55, and a device setting memory unit 56.

The learning cleaning control unit 31 controls the manual cleaning of the cleaning device 1 according to the operation of the operator during the learning mode. The control contents by the learning cleaning control unit 31 are stored in the cleaning condition memory unit 51 as teaching data. This teaching data includes, for example, the amount of cleaning liquid sprayed, the contact pressure of the cleaning member 16, and the suction force of dirty water. The learning travel control unit 32 controls the manual travel of the cleaning device 1 according to the operation of the operator during the learning mode. The control contents by the learning travel control unit 32 are stored in the cleaning condition memory unit 51 as teaching data. This teaching data includes, for example, the travel speed.

The SLAM control unit 33 executes SLAM in real time to estimate the self-position of the cleaning device 1 and create a local map based on the distance and angle from obstacles around the cleaning device 1 measured by the measurement unit 25. The travel path creation unit 34 creates a travel path L (see FIG. 6(A)) for the cleaning device 1 by connecting multiple self-positions arranged in chronological order. The environmental map creation unit 35 creates an environmental map M (see FIG. 6(A)) by connecting multiple local maps arranged in chronological order. The travel path L is stored as teaching data in the travel path memory unit 52, and the environmental map M is stored as teaching data in the environmental map memory unit 53.

The measuring unit 25 may use a two-dimensional type LRF as the obstacle sensor 26, or a three-dimensional type LRF. Two-dimensional data may be acquired by a two-dimensional type LRF, or three-dimensional data may be acquired by the swinging of a three-dimensional type LRF or a two-dimensional type LRF. That is, the travel path creation unit 34 may create a two-dimensional travel path, or a three-dimensional travel path. The environmental map creation unit 35 may create a two-dimensional environmental map, or a three-dimensional environmental map.

The cleaning plan creation unit 36 creates a cleaning plan by associating various teaching data with each step at a predetermined time interval from the start to the end of the learning mode. In this case, the travel path L of the cleaning device 1 is reflected in the environmental map M, and a cleaning plan is created by setting the amount of cleaning liquid to be sprayed, the contact pressure of the cleaning member 16, the suction power of dirty water, and the travel speed at each point from the start point P1 to the end point P2 of the travel path L (see FIG. 6(A)). The cleaning plan is stored in the cleaning plan memory unit 54 and is used during the reproduction mode. In addition to the cleaning plan learned in the learning mode, the cleaning plan memory unit 54 also stores cleaning plans created in advance.

The reproduction cleaning control unit 37 reads a cleaning plan from the cleaning plan memory unit 54 during reproduction mode, and controls the automatic cleaning of the cleaning device 1 according to the cleaning plan. The reproduction travel control unit 38 reads a cleaning plan from the cleaning plan memory unit 54 during reproduction mode, and controls the autonomous travel of the cleaning device 1 according to the cleaning plan. While the cleaning device 1 autonomously travels along the travel path L, it sprays cleaning liquid onto the floor surface F and cleans the floor surface F with the cleaning members 16. In this way, the cleaning device 1 is controlled by the reproduction cleaning control unit 37 and the reproduction travel control unit 38, and the cleaning plan read from the cleaning plan memory unit 54 is reproduced.

The previously cleaned trajectory creation unit (work trajectory creation unit) 39 creates a previously cleaned trajectory T1 (see FIG. 4B) when cleaning is performed according to the cleaning plan, based on the travel path L from the cleaning start point P1 to the current point. In this case, the previously cleaned trajectory creation unit 39 reads the travel path L from the travel path memory unit 52 and reads the width of the cleaning member 16 from the device setting memory unit 56. Then, based on the travel path L from the cleaning start point P1 to the current point and the width of the cleaning member 16, the previously cleaned trajectory creation unit 39 creates a previously cleaned trajectory (worked trajectory) T1 for the floor surface F. The previously cleaned trajectory T1 is stored in the cleaning trajectory memory unit 55.

The virtual cleaning trajectory creation unit (work trajectory creation unit) 40 creates a virtual cleaning trajectory T2 (see FIG. 4B) for cleaning according to the cleaning plan, based on the travel path L from the current location to the cleaning end point P2 (see FIG. 6A). In this case, the virtual cleaning trajectory creation unit 40 reads the travel path L from the travel path memory unit 52 and the width of the cleaning member 16 from the device setting memory unit 56. Then, based on the travel path L from the current location to the cleaning end point P2 and the width of the cleaning member 16, the virtual cleaning trajectory creation unit 40 creates a virtual cleaning trajectory (virtual work trajectory) T2 for cleaning the floor surface F. The virtual cleaning trajectory T2 is stored in the cleaning trajectory memory unit 55.

The AR display control unit (display control unit) 41 causes the display unit 22 to display the previously cleaned trajectory T1 and/or the virtual cleaning trajectory T2 superimposed on the floor surface F in the captured image input from the imaging unit 21. For example, feature points of the environmental map M are matched with feature points of the captured image to identify the current position of the cleaning device 1 in the imaging coordinate system (camera coordinate system) of the captured image. The previously cleaned trajectory T1 and the virtual cleaning trajectory T2 based on the current position of the cleaning device 1 are superimposed on the floor surface F in the captured image. The previously cleaned trajectory T1 and the virtual cleaning trajectory T2 are not limited to being linear, and the previously cleaned trajectory T1 and the virtual cleaning trajectory T2 may be formed in an arc shape at the turning point of the cleaning device 1.

Note that the recognition of the floor surface F in the captured image is achieved by image segmentation (semantic segmentation). Image segmentation is a technique for distinguishing objects for each pixel of a captured image and displaying them in an identifiable manner, and determining which category each pixel of the image belongs to. Pixels of the same category are treated as the same label, and the floor surface F is recognized as being distinguished from the ceiling, wall surfaces, etc., and the already cleaned trajectory T1 and the virtual cleaning trajectory T2 are superimposed on the floor surface F. The already cleaned trajectory T1 and the virtual cleaning trajectory T2 on the floor surface F may be represented by colored areas, or each area may be represented by a different display color.

The cleaned trajectory calculation unit 42 reflects the cleaned trajectory T1 on the environmental map M, and calculates the distance and area from an obstacle such as a wall measured by the measurement unit 25 to the cleaned trajectory T1, and the lap width of the cleaned trajectory T1. The virtual cleaning trajectory calculation unit 43 reflects the virtual cleaning trajectory T2 on the environmental map M, and calculates the distance and area from an obstacle such as a wall measured by the measurement unit 25 to the virtual cleaning trajectory T2, the lap width of the virtual cleaning trajectory T2, and the lap width of the cleaned trajectory T1 and the virtual cleaning trajectory T2. The uncleaned area calculation unit 44 regards the gap in the cleaned trajectory T1 as an uncleaned area, and calculates the vertical and horizontal dimensions and area of this uncleaned area.

The cleaning plan correction unit (work plan correction unit) 45 corrects the cleaning plan when a change operation is received regarding the existing cleaning trajectory T1 and/or the virtual cleaning trajectory T2. For example, when a change operation is received regarding the distance from the wall surface to the cleaning trajectory or the lap width of the cleaning trajectory, the cleaning plan correction unit 45 offsets the travel path L of the cleaning plan. The travel path L is a set of X and Y coordinate values that indicate the self-position of the cleaning device 1 on the environmental map M. The set values of the distance from the wall surface to the cleaning trajectory and the lap width of the cleaning trajectory are converted into displacement amounts in the X and Y directions, and the X and Y direction displacement amounts are added to the X and Y coordinate values of the travel path L to offset the travel path L.

For example, when an operation to change the cleaning conditions for the cleaning trajectory is received, the cleaning plan correction unit 45 changes the cleaning conditions of the cleaning plan. By increasing the contact pressure of the cleaning member 16, etc., it is possible to focus on cleaning a portion of the floor surface F. Instead of changing the cleaning conditions, the cleaning plan correction unit 45 may set a reciprocating path for a portion of the travel path L of the cleaning plan. By repeating reciprocating cleaning by the cleaning device 1 on the reciprocating path, it is possible to focus on cleaning a portion of the floor surface F. By changing the travel path L and cleaning conditions, the cleaning plan is modified to reflect the operator's intentions.

The cleaning trajectory division unit (work trajectory division unit) 46 divides the cleaning trajectory T1 and/or the virtual cleaning trajectory T2 into multiple divided trajectories 70c (see FIG. 6(B)) consisting of straight trajectories and turning trajectories. In this case, the AR display control unit 41 displays the multiple divided trajectories 70c superimposed on the captured image, and the multiple divided trajectories 70c are individually selected by the operator's touch operation. By selecting the divided trajectories 70c, it is possible to change the cleaning conditions, etc. for each divided trajectory 70c. In addition, when a shrink operation is performed on the selected divided trajectory 70c, the divided trajectory 70c is shrinked, making it possible to change the cleaning conditions for a narrower range (see FIG. 7(B)).

This cleaning device 1 can use cleaning plans learned by manual operation and cleaning plans created in advance on a management terminal or the like. When learning a cleaning plan, while the cleaning device 1 is manually operated by an operator, SLAM is executed by the SLAM control unit 33, the self-position of the cleaning device 1 is estimated, and a local map is created. The self-position is connected by the travel path creation unit 34 to create a travel path L, and the local maps are connected by the environmental map creation unit 35 to create an environmental map M. The cleaning plan creation unit 36 reflects the travel path L in the environmental map M, and the cleaning conditions are associated with the travel path L to create a cleaning plan.

When creating a cleaning plan in advance, an environmental map M is created from drawing data prepared in advance. In addition, a zigzag travel path L is created that wraps the cleaning trajectory at a fixed width based on previously input settings such as the cleaning start point P1, end point P2, safety distance from the wall, and width of the cleaning member 16. The travel path L is reflected in the environmental map M, and the cleaning conditions input in advance are associated with the travel conditions to create a cleaning plan. Instead of using drawing data, the environmental map M may be created by executing SLAM while the cleaning device 1 is manually driven along the wall by an operator.

When cleaning using a learned cleaning plan or a cleaning plan created in advance, the cleaning device 1 moves according to the travel path L of the cleaning plan while estimating its own position using SLAM, thereby cleaning the floor surface F. At this time, a local map is created by SLAM, and the environmental map M may be updated using this local map. The display screen of the display unit 22 displays the previously cleaned trajectory T1 and/or the virtual cleaning trajectory T2 on the floor surface F in response to an operation by the operator. When the operator visually confirms the display screen and receives a change operation related to the previously cleaned trajectory T1 and/or the virtual cleaning trajectory T2, the cleaning plan is modified.

Here, the process of correcting the cleaning plan during cleaning will be described. Figure 4 shows an example of the display screen of this embodiment, where (A) is a display screen with AR display disabled, and (B) is a display screen with AR display enabled. Note that the cleaning plan may be a learned cleaning plan or a cleaning plan created in advance. Here, the explanation will be given using the symbols in Figures 2 and 3 as appropriate.

As shown in FIG. 4(A), the cleaning device 1 is temporarily stopped while cleaning according to the cleaning plan. When AR display is disabled, an image of the room in front of the cleaning device 1 captured by the imaging unit 21 is displayed on the display screen of the display unit 22. It is difficult to determine whether the cleaning plan is appropriate from the indoor image, and it is not possible to prompt the operator to modify the cleaning plan. For this reason, a switch button 73a with "AR display" written in white letters on a black background is displayed on the display screen. When the operator touches the switch button 73a, the AR display on the display screen is switched from disabled to enabled.

As shown in FIG. 4(B), when the AR display is enabled, the previously cleaned trajectory T1 and the virtual cleaning trajectory T2 are displayed superimposed on the floor surface F of the room image. Also displayed on the room image are setting values such as the lap width 73b of the previously cleaned trajectory T1, the lap width 73c of the virtual cleaning trajectory T2, the distance 73d from the right wall surface to the virtual cleaning trajectory T2, and the number of straight paths 73e. Of these setting values, the setting values for the previously cleaned trajectory T1 written in black on a white background cannot be changed, and the setting values for the virtual cleaning trajectory T2 written in white on a black background can be changed. The operator visually checks the display screen to determine whether or not to modify the subsequent cleaning plan.

By changing the setting value for the virtual cleaning trajectory T2, the travel path L (see FIG. 6(A)) is offset and the virtual cleaning trajectory T2 is redisplayed on the room image. The setting value change operation is repeated, and when the virtual cleaning trajectory T2 is in line with the operator's intention, the confirm button (not shown) is pressed and the offset travel path L is reflected in the cleaning plan. In this way, the cleaning plan is modified while the floor surface F is being cleaned. Note that after the setting value is confirmed, the setting value does not need to be displayed on the display screen. This allows the operator to recognize that the cleaning plan has been modified to match the operator's intention.

Next, the process of correcting the cleaning plan before cleaning will be described. FIG. 5 is a diagram showing an example of the display screen before cleaning in this embodiment, where (A) shows the display screen before the setting is changed, and (B) shows the display screen after the setting is changed. FIG. 6 is a diagram showing an example of the process of designating the focused cleaning area in this embodiment, where (A) shows the first stage of division and (B) shows the second stage of division. FIG. 7 is a diagram showing an example of the process of designating the focused cleaning area in this embodiment, where (A) shows the selection state of the division trajectory and (B) shows the reduced state of the division trajectory. The cleaning plan may be a learned cleaning plan or a cleaning plan created in advance. Here, the symbols in FIG. 2 and FIG. 3 will be used appropriately for explanation.

As shown in FIG. 5(A), when the AR display is enabled before cleaning the floor surface F, the virtual cleaning trajectory T2 is displayed superimposed on the floor surface F in the room image. Also displayed on the room image are setting values such as the lap width 73f of the virtual cleaning trajectory T2, the distance 73g from the right wall surface to the virtual cleaning trajectory T2, and the number of straight paths 73h. Also displayed on the room image is a designation button 73i with the words "Designate priority cleaning area" written in white letters on a black background. When the operator touches the designation button 73i, a priority cleaning area 72 is designated for the virtual cleaning trajectory T2, as shown in FIG. 5(B).

First, as shown in FIG. 6(A), the virtual cleaning trajectory T2 is divided into a linear trajectory collection area 70a and a rotating trajectory collection area 70b, and then, as shown in FIG. 6(B), the collection areas 70a and 70b are divided into a plurality of divided trajectories 70c, each of which is a linear trajectory and a rotating trajectory. The plurality of divided trajectories 70c are displayed on the display screen so as to be selectable by the operator's touch operation. An arbitrary divided trajectory 70c is selected from the plurality of divided trajectories 70c, and the selected divided trajectory 70c is designated as the priority cleaning area 72. Note that one divided trajectory 70c from the plurality of divided trajectories 70c may be designated as the priority cleaning area 72, or two or more divided trajectories 70c may be designated as the priority cleaning area 72.

As shown in FIG. 7(A), when the divided trajectory 70c is designated as the priority cleaning area 72, the cleaning conditions, such as the contact pressure of the cleaning member 16 associated with the travel path L of the priority cleaning area 72, are changed to be higher. Instead of changing the contact pressure, the cleaning device 1 may perform reciprocating cleaning of the travel path L of the priority cleaning area 72. In this case, the cleaning device 1 may perform a pivot turn at the end point of the travel path L corresponding to the priority cleaning area 72, and the direction of travel of the cleaning device 1 may be reversed to perform reciprocating cleaning. As shown in FIG. 7(B), the divided trajectory 70c may be reduced (shortened) by one step in response to a touch operation by the operator. This allows the priority cleaning area 72 to be designated for the reduced divided trajectory 70c.

The revised plan may also be modified by changing each setting value on the indoor image (see FIG. 5(B)). By changing each setting value, the travel path L is offset and the virtual cleaning trajectory T2 is redisplayed on the display screen. The offset travel path L is then reflected in the cleaning plan, and the revised plan is modified before cleaning floor surface F. Here, an example of modifying the cleaning plan before cleaning floor surface F has been described, but the cleaning plan may also be modified after cleaning floor surface F. The revised cleaning plan can be stored as a separate plan in the cleaning plan memory unit 54 and used for the next cleaning or later.

Next, the process of correcting the cleaning plan after learning will be described. Figure 8 is a diagram showing the positional relationship of the cleaning device after learning the cleaning plan of this embodiment. Figure 9 is a diagram showing an example of the display screen after learning of this embodiment, where (A) shows the display screen before the setting change and (B) shows the display screen after the setting change. Here, the symbols in Figures 2 and 3 will be used appropriately for explanation.

As shown in FIG. 8, after learning the cleaning plan, the cleaning device 1 is positioned at the end point P2 of cleaning. Because the cleaning device 1 faces the wall at the end point P2, the operator manually changes the direction of the cleaning device 1 so that it faces into the room. As shown in FIG. 9(A), when the AR display is enabled after the direction of the cleaning device 1 is changed, the previously cleaned trajectory T1 is displayed superimposed on the floor surface F of the room image. In addition, the distance 73j from the right wall surface to the previously cleaned trajectory T1 and the uncleaned area 73k are displayed on the room image. The display of the uncleaned area 73k makes the operator aware of any areas that have not been cleaned.

As shown in FIG. 9(B), when the operator changes the distance 73j from the right wall surface to the previously cleaned trajectory T1, the cleaning plan correction unit 45 offsets the travel path L (see FIG. 8) to narrow the path spacing. By offsetting the travel path L, the previously cleaned trajectory T1 is redisplayed superimposed on the floor surface F of the room image. When the operator confirms on the display screen that there are no more uncleaned areas, the offset travel path L is reflected in the cleaning plan, and the learned correction plan is corrected. The travel path L may also be offset by changing the lap width of the previously cleaned trajectory T1.

The travel path L may be partially offset by selecting a portion of the previously cleaned trajectory T1. In this case, the previously cleaned trajectory T1 is divided into a plurality of divided trajectories 70c (see FIG. 6B) consisting of a straight trajectory and a turning trajectory. The divided trajectory 70c is selected by the operator, and the travel path L corresponding to the divided trajectory 70c is directly offset. At this time, the travel path L corresponding to the unselected divided trajectory 70c may also be automatically offset. If the offset of the travel path L causes the distance 73j from the right wall surface to the previously cleaned trajectory T1 to become too large, a straight path may be added to the travel path L. The addition of a straight path is performed manually by the operator.

The above describes the process of correcting the cleaning plan, but the cleaning device of this embodiment is also capable of automatically generating a cleaning plan during learning. The process of generating a cleaning plan during learning is now described. Figure 10 shows an example of a display screen during learning of a cleaning plan in this embodiment, where (A) shows the display screen before the settings are changed, and (B) shows the display screen after the settings are changed. Note that the symbols in Figures 2 and 3 will be used appropriately in this description.

As shown in FIG. 10(A), when the cleaning device 1 is temporarily stopped while learning a cleaning plan and AR display is enabled, the AR display control unit 41 displays the already cleaned trajectory T1 and the virtual cleaning trajectory T2 superimposed on the floor surface F of the room image. While learning a cleaning plan, only the travel path L is created from the cleaning start point P1 (see FIG. 8) to the current point. In order to display the virtual cleaning trajectory T2, it is necessary to create the travel path L from the current point to the cleaning end point P2 (see FIG. 8). Therefore, the cleaning device 1 predicts the travel path L from the current point to the cleaning end point P2 from the travel path L from the cleaning start point P1 to the current point.

More specifically, the previously cleaned trajectory creation unit (work trajectory creation unit) 39 creates a previously cleaned trajectory T1 when cleaning is performed according to manual operations while learning the cleaning plan. At this time, the path interval and straight line length of the travel path L from the cleaning start point P1 to the current point are calculated. The virtual cleaning trajectory creation unit (work trajectory creation unit) 40 predicts the travel path L when cleaning is performed from the current point to the cleaning end point P2 from the path interval and straight line length of the travel path L. Then, a virtual cleaning trajectory T2 is created based on the travel path L from the current point to the cleaning end point P2. In this way, the virtual cleaning trajectory T2 is predicted from the previously cleaned trajectory T1.

In addition to the already cleaned trajectory T1 and the virtual cleaning trajectory T2, the indoor image also displays set values such as the lap width 73l of the already cleaned trajectory T1, the lap width 73m of the virtual cleaning trajectory T2, the distance 73n from the right wall to the virtual cleaning trajectory T2, and the number of straight paths 73o. The operator visually checks the display screen and determines whether the virtual cleaning trajectory T2 is appropriate based on the already cleaned trajectory T1, the virtual cleaning trajectory T2, and each set value. For example, it is determined whether the virtual cleaning trajectory T2 is appropriate based on a comparison of the lap width 73m of the virtual cleaning trajectory T2 with the allowable lap width, and a comparison of the distance 73n from the right wall to the virtual cleaning trajectory T2 with the safe distance.

If the virtual cleaning trajectory T2 is appropriate, the cleaning plan creation unit (work plan creation unit) 36 creates a cleaning plan for the previously cleaned trajectory T1 by associating the cleaning conditions implemented during manual operation with the travel path L from the cleaning start point P1 to the current point. The cleaning plan creation unit 36 also creates a cleaning plan for the virtual cleaning trajectory T2 by associating the cleaning conditions already implemented during manual operation with the travel path L from the current point to the cleaning end point P2. The cleaning plan for the previously cleaned trajectory T1 is then combined with the cleaning plan for the virtual cleaning trajectory T2 to complete the overall cleaning plan.

As shown in FIG. 10(B), if the virtual cleaning trajectory T2 is inappropriate, the travel path L is offset by changing the setting value for the virtual cleaning trajectory T2, and the virtual cleaning trajectory T2 is redisplayed. The setting value change operation is repeated, and when the virtual cleaning trajectory T2 matches the operator's intention, a confirmation button (not shown) is pressed. The cleaning plan creation unit 36 associates the cleaning conditions already implemented during manual operation with the offset travel path L to create a cleaning plan for the virtual cleaning trajectory T2. Then, the cleaning plan for the virtual cleaning trajectory T2 is combined with the cleaning plan for the already cleaned trajectory T1 to complete the overall cleaning plan.

The processing operation of the cleaning device will now be described. FIG. 11 is an example of a flowchart for the process of correcting a cleaning plan in this embodiment. FIG. 12 is an example of a flowchart for the process of generating a cleaning plan during learning in this embodiment. Note that the symbols in FIG. 2 and FIG. 3 will be used as appropriate in the explanation.

As shown in FIG. 11, an image of the room is captured by the imaging unit 21, and the image of the room is displayed on the display unit 22 (step S01). Next, the previous cleaning trajectory creation unit 39 creates a previous cleaning trajectory T1 when cleaning is performed according to the cleaning plan, and the virtual cleaning trajectory creation unit 40 creates a virtual cleaning trajectory T2 when cleaning is performed according to the cleaning plan (step S02). Next, when the AR display is enabled, the AR display control unit 41 displays the previous cleaning trajectory T1, the virtual cleaning trajectory T2, the setting values of the cleaning trajectory lap width, etc. superimposed on the room image (step S03). The operator visually checks the display screen and determines whether or not to modify the cleaning plan.

If no change operation is received for the setting values, etc., for the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 (No in step S04), the change operation is accepted until the operator disables the AR display (No in step S08). If a change operation is received for the setting values, etc., for the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 (Yes in step S04), the cleaning plan correction unit 45 offsets the travel path L. Then, the AR display control unit 41 redisplays the setting values, such as the already cleaned trajectory T1, the virtual cleaning trajectory T2, and the lap width of the cleaning trajectory, superimposed on the room image (step S05).

The processing of steps S04 and S05 is repeated until the operator visually checks the display screen and confirms the changes to the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 (No in step S06). When the changes to the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 are confirmed (Yes in step S06), the cleaning plan correction unit 45 reflects the changes in the cleaning plan and corrects the cleaning plan (step S07). Then, the processing of steps S04-S07 is repeated until the operator disables the AR display (No in step S08).

As shown in FIG. 12, while learning the cleaning plan, the imaging unit 21 captures an image of the room, and the display unit 22 displays the image of the room (step S11). Next, the cleaning trajectory creation unit 39 creates a cleaning trajectory T1 when cleaning according to manual operation (step S12). Next, the virtual cleaning trajectory creation unit 40 creates a virtual cleaning trajectory T2 for the cleaning schedule, which is predicted from the cleaning trajectory T1 (step S13). Next, when the AR display is enabled, the AR display control unit 41 displays the cleaning trajectory T1, the virtual cleaning trajectory T2, the setting values of the cleaning trajectory lap width, etc., superimposed on the room image (step S14). The operator visually checks the display screen and determines whether the virtual cleaning trajectory T2 is appropriate.

If no change operation is received for the setting values, etc., related to the virtual cleaning trajectory T2 (No in step S15), the change operation is accepted until the setting for the virtual cleaning trajectory T2 is confirmed by the operator (No in step S17). If a change operation is received for the setting values, etc., related to the virtual cleaning trajectory T2 (Yes in step S15), the travel path L corresponding to the virtual cleaning trajectory T2 is offset. Then, the AR display control unit 41 re-displays the setting values, such as the previously cleaned trajectory T1, the virtual cleaning trajectory T2, and the lap width of the cleaning trajectory, superimposed on the room image (step S16).

The processing of steps S15 and S16 is repeated until the operator visually checks the display screen and the settings for the virtual cleaning trajectory T2 are finalized (No in step S17). When the settings for the virtual cleaning trajectory T2 are finalized (Yes in step S17), the cleaning plan creation unit 36 creates a cleaning plan for the virtual cleaning trajectory T2, which is an unlearned area (step S18). The cleaning plan creation unit 36 then combines the cleaning plan for the already cleaned trajectory T1 and the cleaning plan for the virtual cleaning trajectory T2 to create an overall cleaning plan (step S19).

As described above, according to this embodiment, the operator is allowed to visually confirm the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 on the floor surface F in the captured image, and the operator's change operation regarding the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2 is accepted. The cleaning plan of the cleaning device 1 is modified by the change operation regarding the already cleaned trajectory T1 and/or the virtual cleaning trajectory T2, and the operator's intention can be reflected in the cleaning plan to improve cleaning efficiency and cleaning quality.

Furthermore, according to this embodiment, the operator is allowed to visually check the virtual cleaning trajectory T2 on the floor surface F in the captured image while the cleaning plan is being learned. By having the operator confirm whether the virtual cleaning trajectory T2, which is an unlearned portion, is appropriate or not, the generation of a cleaning plan that goes against the operator's intention is prevented. The operator's intention can be reflected in the cleaning plan for the virtual cleaning trajectory T2 without having the cleaning device 1 repeat learning through manual operation.

In this embodiment, a cleaning device that performs cleaning work is exemplified as the autonomous traveling work device, but the autonomous traveling work device is not limited to a cleaning device. The autonomous traveling work device may be a work device that performs other work such as dry cleaning work, polishing work, polishing work, wax application work, etc. In other words, the autonomous traveling work device may be a robotic cleaning machine, polisher, buffing machine, or wax application machine. Therefore, instead of cleaning members such as pads, cleaning brushes, polishing brushes, polishing brushes, and application members may be used in the autonomous traveling work device, and further, abrasives, polishing agents, and coating agents may be used as cleaning fluids.

In addition, in this embodiment, a floor surface is exemplified as the surface to be cleaned, but the surface to be cleaned may also be a wall surface, ceiling surface, window surface, mirror surface, etc.

In addition, in this embodiment, the environmental map is created using SLAM, but the environmental map may also be created using V-SLAM or LiDAR-SLAM, or the environmental map may be prepared in advance by the operator.

In addition, in this embodiment, the operator manually operates the cleaning device by holding the handle, but the operator may also manually operate the cleaning device remotely using a display terminal.

In addition, in this embodiment, when touching the already cleaned trajectory and/or the virtual cleaning trajectory on the floor surface in the captured image, the wall surface and the ceiling may be set to be unselectable, or the wall surface and the ceiling may be set to be selectable. If the wall surface is selectable, the cleaning device may have a function of spraying disinfectant or irradiating UV rays onto the wall surface. In response to the operator's touch operation on the wall surface, the cleaning device may notify the operator of the operator's intention by a visual or audio notification, and the operation may be switched from cleaning the floor surface to spraying disinfectant or irradiating UV rays onto the wall surface.

In addition, in this embodiment, both unchangeable and changeable setting values are displayed on the display screen, but it is also possible for only changeable setting values to be displayed on the display screen.

In addition, in this embodiment, a program may be installed in the cleaning device to add a function that allows the operator to visually recognize the already cleaned trajectory and/or the virtual cleaning trajectory on the floor surface in the captured image and prompts the operator to modify the cleaning plan. Also, a program may be installed in the cleaning device to add a function that allows the operator to visually recognize the virtual cleaning trajectory on the floor surface in the captured image while learning the cleaning plan and generates a cleaning plan for the virtual cleaning trajectory. These programs are stored in a storage medium. The storage medium is not particularly limited, but may be a non-transitory storage medium such as an optical disk, a magneto-optical disk, or a flash memory.

Although this embodiment has been described, other embodiments may be combinations of the above embodiments and variations in whole or in part.

The technology of the present invention is not limited to the above-described embodiments, and may be modified, substituted, or altered in various ways without departing from the spirit of the technical idea. Furthermore, if the technical idea can be realized in a different way due to technological advances or other derived technologies, it may be implemented using that method. Therefore, the claims cover all embodiments that may fall within the scope of the technical idea.

As explained above, the technology of the present invention is useful for industrial robots suitable for automating work in work areas, such as autonomous cleaning devices.

1: Cleaning device (autonomous mobile work device)
36: Cleaning plan creation department (work plan creation department)
39: Cleaned trajectory creation section (already worked trajectory creation section)
40: Virtual cleaning trajectory creation unit (virtual work trajectory creation unit)
41: AR display control unit (display control unit)
45: Cleaning plan modification section (work plan modification section)
46: Cleaning trajectory dividing section (work trajectory dividing section)
70c: Division locus F: Floor surface (worked surface)
L: Travel route P1: Start point P2: End point T1: Previously cleaned path (previously worked path)
T2: Virtual cleaning trajectory (virtual work trajectory)

Claims (6)

An autonomous mobile work device that performs work on a work surface while autonomously traveling according to a work plan,
a work trajectory creation unit that creates a work trajectory of completed work and/or a virtual work trajectory of planned work when work is performed according to the work plan;
a display control unit that displays the completed work trajectory and/or the virtual work trajectory superimposed on a work surface in a captured image acquired from an imaging unit;
a work plan correction unit that corrects a work plan when a change operation is received regarding the existing work trajectory and/or the virtual work trajectory,
the work trajectory creation unit creates the completed work trajectory based on a travel path from a work start point to a current point and a width of a work member , and creates the virtual work trajectory based on a travel path from the current point to a work end point and the width of the work member ;
An autonomous mobile work device, characterized in that the travel path is offset by changes to the previous work trajectory and/or the virtual work trajectory displayed superimposed on the work surface in the captured image, and the previous work trajectory and/or the virtual work trajectory are re-displayed on the captured image. A work trajectory division unit that divides the existing work trajectory and/or the virtual work trajectory into a plurality of divided trajectories,
The display control unit displays a plurality of the division trajectories in a superimposed manner on a work surface in a captured image,
2. The autonomous navigation and work device according to claim 1, wherein the work plan correction unit corrects the work plan when a change operation is received for the divided trajectory. It learns the work plan according to manual operation, and performs the work on the work surface while autonomously traveling according to the learned work plan.
The display control unit displays the completed work trajectory superimposed on a work surface in a captured image after learning the work plan,
3. The autonomous mobile work device according to claim 1, wherein the work plan correction unit corrects the learned work plan when a change operation related to the previous work trajectory is received. The change operation regarding the completed work trajectory is an operation of changing a distance from a wall surface to the completed work trajectory,
An autonomous mobile work device as described in any one of claims 1 to 3, characterized in that the change operation related to the virtual work trajectory is an operation to change at least one of a lap width of the virtual work trajectory, a distance from a wall surface to the virtual work trajectory, and a number of straight paths of a travel path. A method for correcting a work plan of an autonomous mobile work device that performs work on a work surface while traveling autonomously, comprising:
A step of creating a completed work trajectory and/or a virtual work trajectory of a planned work when work is performed according to the work plan;
A step of superimposing and displaying the already worked trajectory and/or the virtual work trajectory on a worked surface in a captured image acquired from an imaging unit;
and modifying the work plan when a change operation is received regarding the existing work trajectory and/or the virtual work trajectory,
In the creating step, the already-operated trajectory is created based on a travel path from a work start point to a current point and a width of the work member , and the virtual work trajectory is created based on a travel path from the current point to an work end point and the width of the work member ;
A method for modifying a work plan, characterized in that a travel path is offset by changes to the already worked trajectory and/or the virtual work trajectory displayed superimposed on the work surface in an captured image, and the already worked trajectory and/or the virtual work trajectory are re-displayed on the captured image. A program for an autonomous mobile work device that performs work on a work surface while autonomously traveling according to a work plan,
A step of creating a completed work trajectory and/or a virtual work trajectory of a planned work when work is performed according to the work plan;
A step of superimposing and displaying the already worked trajectory and/or the virtual work trajectory on a worked surface in a captured image acquired from an imaging unit;
and modifying a work plan when a change operation is received regarding the existing work trajectory and/or the virtual work trajectory.
In the creating step, the already-operated trajectory is created based on a travel path from a work start point to a current point and a width of the work member , and the virtual work trajectory is created based on a travel path from the current point to an work end point and the width of the work member ;
A program characterized by offsetting a travel path by making changes to the previously worked trajectory and/or the virtual work trajectory displayed superimposed on the work surface in an captured image, and re - displaying the previously worked trajectory and/or the virtual work trajectory on the captured image.

Images