JP7765345B2 - Work status management system, work status management method and program - Google Patents

Description

The present invention relates to a work status management system, a work status management method, and a program, and in particular to technology for supporting work using work machines and tools.

At civil engineering and construction sites, many tasks involve using work machinery and tools in specific locations for specific periods of time to achieve a specific purpose. Examples include compacting soil using rolling machines (also known as tamping rammers or plate compactors), compacting crushed stone under railroad tracks using tie tampers, tapping inspections of exterior walls using tapping rods, and filling or coating walls with insulation or fireproofing materials using spray guns. Even outside of civil engineering and construction, tasks such as spraying pesticides on fields using sprayers or drones, cleaning indoor carpets with cleaning machines, and brewers using oars all involve using specific work machinery and tools in specific locations for specific periods of time.

In these tasks, the position, range, time, angle, etc. of the work machinery and tools to be used may be instructed in advance, but in reality, these decisions are made based on the individual worker's intuition and experience. As a result, there is a tendency for variations to occur between workers, and even the same worker may produce different results each time. From the perspective of managing work quality, the challenge is to minimize such variations as much as possible.

Related technologies include Patent Documents 1 to 4. All of these relate to the compaction of fresh concrete using a vibrator, a type of work tool, and aim to objectively grasp the work situation.

Patent documents 1 to 3 describe a system that installs AR markers around the formwork, captures images of the AR markers and a vibrator during compaction work with a camera, and uses the images to estimate the position of the vibrator and identify the areas where compaction has been performed.

Patent document 4 describes a system that uses GPS to acquire the location of a vibrator and uses a head-mounted display worn by a worker to display an augmented reality image of the vibrator at that location.

Japanese Patent Application Laid-Open No. 2021-161845 Japanese Patent Application Laid-Open No. 2021-161846 Japanese Patent Application Laid-Open No. 2021-161847 Japanese Patent Application Laid-Open No. 2019-035265

However, the methods described in Patent Documents 1 to 3 require a process of installing a camera in advance in a position where it can simultaneously and continuously capture images of the AR marker and the work machine. This process requires a considerable amount of time, effort, and cost. Furthermore, with this method, the operating position of the work machine is estimated based on the parts visible in the image (for example, the state of the hose connected to the vibrator) and on the structure specific to that work machine. This estimation method, which is specialized for a specific work machine, cannot be generally applied to other work machines. Furthermore, this system does not have a configuration for instructing the worker on the position, range, time, angle, etc. at which the work machine should be operated, nor a configuration for reporting the results of the work to a manager.

The method described in Patent Document 4 obtains the position of work machinery and tools using a GPS unit. However, depending on the type of work machinery and tools, the GPS unit may have difficulty receiving signals from satellites, making it difficult to measure their position. It is said that GPS-based position detection can have an error of several meters, and there are limits to how accurately this method can reproduce the position of work machinery and tools. Furthermore, this system does not include any features for instructing workers on the position, range, time, angle, etc. at which the work machinery and tools should be operated, nor any features for reporting work results to managers.

The present invention was made to solve these problems, and aims to provide a work status management system, work status management method, and program for supporting work using work machines and tools.

According to one embodiment, a work status management system is a work status management system that includes a work machine and a worker terminal, wherein the work machine has an acting body that applies a predetermined action to an object, and a 3DAR marker, and the worker terminal has a work machine recognition unit that photographs the 3DAR marker with a camera, recognizes the position and attitude of the 3DAR marker based on the photographed image, and estimates the position of the acting body based on the position and attitude of the 3DAR marker, a work instruction unit that displays an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible, and a work status display unit that determines whether the position of the acting body estimated by the work machine recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display.
According to one embodiment, the work status display unit estimates the distance between the target and the position of the operating body, and displays the distance together with or instead of the determination result.
According to one embodiment, the work status display unit estimates an angle between the target and the position of the operating body, and displays the angle together with or instead of the determination result.
According to one embodiment, the work machine or tool further has a switch that activates or stops the acting body and notifies the worker terminal of whether the acting body is operating or not, the work machine or tool recognition unit determines whether the acting body is operating or not based on the notification from the switch, and the work status display unit estimates a work time based on the determination result of whether the acting body is operating or not and the distance estimation result, and displays the work time together with at least one of the determination result and the distance, or instead of the determination result and the distance.
According to one embodiment, a work status management system includes a work machine or tool and a worker terminal, wherein the work machine or tool has an acting body that applies a predetermined action to an object, and a self-location estimation device, the self-location estimation device estimates its own position and attitude and transmits it to the worker terminal, and the worker terminal has a work machine or tool recognition unit that estimates the position of the acting body based on the position and attitude received from the self-location estimation device, a work instruction unit that displays an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible, and a work status display unit that determines whether the position of the acting body estimated by the work machine or tool recognition unit is within the range indicated by the object, and displays the determination result on the transmissive display.
According to one embodiment, the work status management method includes a work machine/tool recognition step of capturing an image of a 3DAR marker equipped on a work machine/tool with a camera, recognizing the position and attitude of the 3DAR marker based on the captured image, and estimating the position of an acting body equipped on the work machine/tool for applying a predetermined action to an object based on the position and attitude of the 3DAR marker; a work instruction step of displaying an object indicating the position at which the acting body should act on a transmissive display so that it is superimposed on the object and visible; and a work status display step of determining whether the position of the acting body estimated by the work machine/tool recognition unit is within a range indicated by the object, and displaying the determination result on the transmissive display.
According to one embodiment, a program causes a computer to carry out the above method.

This invention provides a work status management system, work status management method, and program for supporting work using work machines and tools.

FIG. 1 is a diagram illustrating an example of a hardware configuration of a work status management system 1. 1 is a diagram showing an example of a working machine or implement 10. FIG. FIG. 2 illustrates an example of a worker terminal 20. FIG. 2 illustrates an example of an administrator terminal 30. FIG. 2 is a block diagram showing the functional configuration of a worker terminal 20 and a manager terminal 30. FIG. 2 is a block diagram showing the functional configuration of a worker terminal 20 and a manager terminal 30. 4 is a flowchart showing an example of the operation of the work status management system 1. 1 is a diagram showing an example of a working machine or implement 10. FIG. 4 is a flowchart showing an example of the operation of the work status management system 1.

First Embodiment
1 is a diagram showing the hardware configuration of a work status management system 1 according to a first embodiment of the present invention. The work status management system 1 includes a work machine or tool 10, a worker terminal 20, and a manager terminal 30.

Figure 2 shows an example of a work machine tool 10. The work machine tool 10 typically includes a housing having an acting body 11 and a switch 14, as well as a 3DAR marker 15.

The acting body 11 is a part or component that applies a predetermined action to an object. For example, the striking plate of a tamping rammer, the vibrating plate of a plate compactor, the beater of a tie tamper, the tip of a tapping rod, the nozzle of a spray gun, sprayer or carpet cleaner, the tip of a paddle, etc. are examples of acting body 11.

When switch 14 is in the ON state, it drives the work machine 10 to continue acting on the target, and when it is in the OFF state, it stops the operation of the work machine 10 and discontinues acting on the target. In addition, switch 14 has a so-called smart switch function and can notify the worker terminal 20 via Bluetooth (registered trademark) communication or the like of whether it is in the ON or OFF state. Note that if the work machine 10 is not powered (for example, if it is a diagnostic wand or paddle), there is no need to have switch 14.

A typical use of the work implement 10 will now be described.
The worker grasps the work machine/tool 10 and brings the acting body 11 into contact with, inserts, or holds it facing the target. When the worker turns on the switch 14 and drives a power source (not shown), the acting body 11 acts on the target. For example, it applies vibration or impact, or injects a filler or agent.

The 3DAR marker 15 is an AR marker with a three-dimensional shape, such as a hexahedron. The 3DAR marker 15 is attached in a position that is easily visible from the outside, such as on the outside of the housing of the work machine or tool 10. The 3DAR marker 15 is used by the worker terminal 20 to determine the position and orientation of the work machine or tool 10.

The 3DAR marker 15 may also have the following shapes. The 3DAR marker 15 is not limited to a hexahedron, and may be any three-dimensional shape, such as a sphere. Alternatively, a QR code (registered trademark) or the like may be attached to several surfaces (for example, four surfaces) around the work machine or tool 10, and these may be used as the 3DAR marker 15. Alternatively, the external shape of the work machine or tool 10 may be created and stored in advance as a 3D model, and when an object matching this 3D model is found in real space, that object (i.e., the work machine or tool 10 itself) may be used as the 3DAR marker 15 (object marker).

Figure 3 shows an example of a worker terminal 20. The worker terminal 20 is typically a head-mounted housing equipped with a camera 21, a depth sensor 22, an inertial sensor 23, a processing unit 24, a communication unit 25, and a transmissive display 26. It is a so-called HMD (Head Mounted Display), and examples include Hololens (registered trademark).

The camera 21 captures images of the real world. Typically, it captures an area corresponding to the field of view of the worker wearing the worker terminal 20 at a predetermined frame rate.

The depth sensor 22 measures the distance to an object by irradiating it with infrared light or the like and measuring the time it takes for the light to reflect back. The depth sensor 22 can acquire three-dimensional point cloud data by measuring (scanning) the distances to multiple points on the object. The three-dimensional point cloud data is information that indicates the shape of an object that exists in the real world.

The inertial sensor 23 is a sensor that measures acceleration, angular velocity, etc. The acceleration, angular velocity, etc. are used to calculate, for example, the amount of movement, movement direction, movement speed, and attitude (tilt, i.e., line of sight) of the worker terminal 20.

The processing unit 24 is an information processing device equipped with a CPU (Central Processing Unit) 241, memory 242, etc., and realizes predetermined functions by the CPU 241 executing programs stored in the memory 242.

The communication unit 25 communicates with external devices such as the work machine or tool 10 or the administrator terminal 30.

The transmissive display 26 displays the information output by the processing unit 24. A worker wearing the worker terminal 20 can view the real world through the transmissive display 26 while simultaneously viewing the information displayed on the transmissive display 26.

The basic functions of the worker terminal 20 will be described.
The worker terminal 20 has a function for setting a stationary coordinate system in the real space around the worker. The stationary coordinate system here refers to a coordinate system with an origin and coordinate axes fixed in real space (the origin and coordinate axes do not change even when the worker terminal 20 moves). The origin and coordinate axes of the stationary coordinate system can be defined, for example, by placing a reference object such as an AR marker in real space. In this case, the camera 21 captures an image of the AR marker placed in real space. The AR marker is typically a flat plate printed with a predetermined pattern, and the pattern encodes information indicating the position of the origin and the orientation of the coordinate axes. Alternatively, a three-dimensional object can be used as the AR marker (3D AR marker). The processing unit 24 recognizes and analyzes the AR marker included in the captured image and determines the origin and coordinate axes of the stationary coordinate system. Alternatively, the position of the worker terminal 20 at the time of startup can be used as the origin. The worker terminal 20 typically sets the stationary coordinate system at startup.

The worker terminal 20 can also constantly recognize its own position in real space. For example, using known techniques such as SLAM (Simultaneous Localization and Mapping), it can generate a map of the real world based on information acquired by the camera 21 or depth sensor 22, while simultaneously recognizing its own position. The recognized self-position can be output as coordinates in a stationary coordinate system.

The worker terminal 20 also has the function of superimposing and displaying any object in real space. The object here refers to an image generated by the processing unit 24 and includes, for example, text, two-dimensional images, and three-dimensional models. The processing unit 24 displays the object on the transmissive display 26. This allows a worker wearing the worker terminal 20 on their head to view the real world through the transmissive display 26 while simultaneously viewing the object displayed on the transmissive display 26. At this time, the processing unit 24 positions the object at predetermined coordinates in a stationary coordinate system. In other words, even if the worker wearing the worker terminal 20 on their head changes their position or posture, the processing unit 24 detects this change using SLAM, the inertial sensor 23, etc., and changes the display position of the object on the transmissive display 26 in response to this change. As a result, the worker sees the object as if it has always been positioned at a predetermined position in real space.

The worker terminal 20 also has the function of grasping the shape of an object existing in real space. Specifically, the depth sensor 22 scans real space and acquires three-dimensional point cloud data that indicates the shape of the object. The processing unit 24 forms a polygon mesh from the three-dimensional point cloud data, thereby recognizing and reproducing the surface of the object.

Figure 4 shows an example of an administrator terminal 30. The administrator terminal 30 is typically a so-called tablet computer that incorporates a processing unit 31, a communication unit 32, a touch-panel display 33, and the like in a flat-shaped housing.

The processing unit 31 is an information processing device equipped with a CPU (Central Processing Unit) 311, memory 312, etc., and realizes predetermined functions by the CPU 311 executing programs stored in the memory 312.

The communication unit 32 communicates with external devices such as the work machine or tool 10 or the worker terminal 20.

The display 33 displays the information output by the processing unit 31.

5A and 5B are block diagrams showing the functional configurations of the worker terminal 20 and the manager terminal 30. As shown in FIG.
The worker terminal 20 has a work machine/tool recognition unit 201, a work instruction unit 202, and a work status display unit 203. These components are logically realized by the CPU 241 of the processing unit 24 executing a program stored in the memory 242.

The work machine/tool recognition unit 201 recognizes the position and orientation (tilt) of the work machine/tool 10 existing in real space. The work machine/tool recognition unit 201 detects the 3DAR marker 15 fixed to the exterior of the work machine/tool 10 from the image captured by the camera 21, and recognizes the position and orientation of the 3DAR marker 15. The work machine/tool recognition unit 201 also stores in advance an offset indicating the relative position between the 3DAR marker 15 and the acting body 11. Therefore, the work machine/tool recognition unit 201 can calculate the coordinates of the acting body 11 based on the position and orientation of the 3DAR marker 15 and the offset indicating the relative position between the 3DAR marker 15 and the acting body 11.

Here, the work machine/tool recognition unit 201 is assumed to previously store information regarding the shape and size of the 3DAR marker 15 (for example, that it is a hexahedron with sides of 10 cm). When the work machine/tool recognition unit 201 detects a 3DAR marker 15 in an image captured by the camera 21, it recognizes it as a hexahedron with sides of 10 cm in the stationary coordinate system. In other words, it can recognize at what coordinates in the stationary coordinate system the hexahedron with sides of 10 cm exists, and at what angle. Therefore, if the offset indicating the relative position between the 3DAR marker 15 and the acting body 11 is known, the work machine/tool recognition unit 201 can also calculate the coordinates of the acting body 11 in the stationary coordinate system.

The work machine/tool recognition unit 201 also recognizes the operating status of the work machine/tool 10. The switch 14 of the work machine/tool 10 notifies the worker terminal 20 via Bluetooth (registered trademark) communication or the like whether it is in an ON or OFF state. The work machine/tool recognition unit 201 receives this notification via the communication unit 25 and determines whether the work machine/tool 10 is operating.

The work instruction unit 202 displays on the transmissive display 26 information indicating at which position on the work site the work machine or tool 10 should be applied. The work instruction unit 202 displays, for example, a circular object superimposed on the surface of the target, indicating the position where the work machine or tool 10 should be contacted, inserted, or directed. Note that the work instruction unit 202 is assumed to have stored in advance in memory 242 information indicating the position where the work machine or tool 10 should be applied. The work instruction unit 202 may also superimpose drawing data previously stored in memory 242 on the site. Superimposing drawing information and other objects on the site can be achieved using known technology, including Patent No. 6438995.

The work instruction unit 202 may also display on the transmissive display 26 information instructing the time or angle at which the work machine or tool 10 should be brought into contact with, inserted into, or directed toward the target. It is assumed that the work instruction unit 202 stores this information in advance in the memory 242.

The work status display unit 203 can generate information indicating the progress of the work (hereinafter simply referred to as the work status) and display it on the transmissive display 26. The work status can include, but is not limited to, information indicating the position of the acting body 11 (for example, information indicating the determination result of whether the acting body 11 is in contact with, inserted into, or directed at the indicated position, or information indicating the distance between the object and the acting body 11), information indicating the angle of the acting body 11 relative to the object, and information indicating the time when the acting body 11 is activated and in correct contact with, inserted, or directed. This information can be generated, for example, by the method shown below.

Position: The work instruction unit 202 displays, for example, a circular object superimposed on the surface of the target, indicating the position where the work machine/tool 10 should be contacted, inserted, or directed. The position where the work machine/tool 10 should be contacted, inserted, or directed may be displayed using a shape other than a circle, grid lines, or the like. The work machine/tool recognition unit 201 calculates the coordinates of the operating body 11 based on the position and orientation (tilt) of the 3DAR marker 15 and a predetermined offset. The work status display unit 203 determines whether the position of the operating body 11 is within the position indicated by the work instruction unit 202. Typically, it determines whether the coordinates obtained by perpendicularly projecting the coordinates of the operating body 11 onto the target surface are contained within the circular object. If the coordinates are contained within the circular object, for example, "OK" is output as a result; otherwise, "NG" is output. The position determination result is typically displayed as text information on the transmissive display 26.

Distance: The work machine/tool recognition unit 201 calculates and outputs the distance between the working body 11 and the target surface. Alternatively, it may calculate and output the distance from an object indicating the position where the work machine/tool 10 should be contacted, inserted, or directed, or the distance from the center point of that object. The work status display unit 203 typically displays the distance as text information on the transmissive display 26.

Angle: The work machine/tool recognition unit 201 calculates and outputs the angle between the working body 11 and the target surface. The work status display unit 203 typically displays the angle as text information on the transmissive display 26.

To perform this calculation, the work machine/tool recognition unit 201 is assumed to previously store the relative tilt of the operating body 11 with respect to the 3DAR marker. For example, if the operating body 11 is generally rod-shaped (such as the beater of a tie tamper, the tip of a tapping rod, the nozzle of a spray gun, sprayer or carpet cleaner, or the tip of a paddle), it is assumed to store information indicating the degree to which the line corresponding to the axis of the rod-shaped shape is tilted with respect to the 3DAR marker. Furthermore, if the operating body 11 acts on an object via a surface (such as the striking plate of a tamping rammer or the vibration plate of a plate compactor), it is assumed to store information indicating the degree to which the line corresponding to the normal to that surface is tilted with respect to the 3DAR marker. This allows the work machine/tool recognition unit 201 to indirectly calculate the tilt of the line corresponding to the axis or normal of the operating body 11 in the stationary coordinate system by detecting the attitude (tilt) of the 3DAR marker 15 in the stationary coordinate system. The work machine/tool recognition unit 201 calculates the angle between this line and the plane that corresponds to the target surface.

Time: The work status display unit 203 accumulates the time that work has been performed by the work machine or tool 10 at a specified location. The work status display unit 203 typically displays the accumulated time as text information on the transmissive display 26.

For example, when work is performed by contacting or inserting the operating body 11 into an object (such as the striking plate of a tamping rammer, the vibrating plate of a plate compactor, the beater of a tie tamper, the tip of a sounding rod, the nozzle of a carpet cleaner, or the tip of a paddle), the work status display unit 203 determines that the operating body 11 is in contact with the object surface, i.e., that work is being performed, when the distance between the operating body 11 and the object surface is equal to or less than a predetermined threshold. When the operating body 11 does not need to contact the object (such as the nozzle of a spray gun or sprayer), the work status display unit 203 determines that work is being performed when the operating status of the work machine or tool 10 notified by the work machine or tool recognition unit 201 is ON (operating). The work status display unit 203 repeatedly performs these determinations at regular intervals and accumulates and outputs the elapsed time during which it is determined that work is being performed. In addition to the above-mentioned conditions for contact determination or operation determination, it may also be possible to add, for example, that the position or angle meets certain requirements, and accumulate the elapsed time when all conditions are met.

The work status display unit 203 can visualize the work status of a task using colors, graphs, etc. In this example, the time portion of the work status is visualized using colors. The work instruction unit 202 is assumed to have stored in advance a specified value for work time at the location where work is to be performed (e.g., 8 seconds or more). If the cumulative work time at each location exceeds the specified value, the work status display unit 203 colors the horizontal circle object indicating that location in a color indicating "pass" (e.g., green). In other cases, the object indicating the insertion position is colored in a different color depending on the ratio of the cumulative work time to the specified value. For example, if the cumulative work time is more than 50% below the specified value, it is colored yellow, and if it is less than 50%, it is colored red. At this time, text indicating the cumulative work time may be displayed near the object.

The work status display unit 203 may also visualize work status related to distance, work time, angle, etc. using colors, graphs, etc.

The work status display unit 203 stores the acquired work status in a storage area within the memory 242. If necessary, statistical values of the work status may also be calculated and stored.

The work status display unit 203 may also store the visualized information displayed on the transmissive display 26 during work and the image of the real world captured by the camera 21 at that time. For example, the information expressing the work execution time using colors, as described above, can be saved as a set with the image of the real world at that time.

The manager terminal 30 has a work performance display unit 301. This component is logically realized when the CPU 311 of the processing unit 31 executes a program stored in the memory 312.

The work performance display unit 301 communicates with the worker terminal 20 via the communication unit 32 and acquires the work status stored in the storage area of the memory 242. In response to a request from the manager, the work performance display unit 301 displays the acquired work status on the display 33. For example, as described above, information that uses colors to represent the time the work is being performed can be displayed superimposed on an image of the real world at that time. In addition, various work statuses (distance, angle, time, etc.) at each work location may be displayed as text information on the display 33.

The operation of the work status management system 1 will be explained chronologically using the flowchart in Figure 6.

S101: Start-up The worker wears the worker terminal 20 on his/her head and starts up the program. The worker terminal 20 sets a stationary coordinate system in real space and starts the operation of the SLAM function.

S102: Reading and placing drawing data The work instruction unit 202 reads out drawing data stored in advance in the memory 242 and displays it superimposed on the work site. That is, the drawing data is placed in a stationary coordinate system. The work instruction unit 202 also places an object indicating the insertion position of the work machine or tool 10 in the stationary coordinate system. The placed drawing and object are displayed on the transmissive display 26. This allows the worker to visually recognize the drawing and object superimposed on the real space.

S103: Recognize 3DAR markers The work machine/tool recognition unit 201 acquires images captured by the camera 21, detects 3DAR markers 15 fixed to the exterior of the work machine/tool 10 from the images, and recognizes the position and orientation of the 3DAR markers 15.

S104: Estimation of position of acting body 11 The work machine/tool recognition unit 201 calculates the coordinates of the acting body 11 based on the position and orientation of the 3DAR marker 15 and an offset indicating the relative position between the 3DAR marker 15 and the acting body 11.

S105: Detect start of operation of acting body 11 The work machine/tool recognition unit 201 receives, for example, a signal indicating the state of the switch 14, and recognizes the operating status of the work machine/tool 10. If the acting body 11 is in operation, the process proceeds to S106.

Alternatively, the work machine/tool recognition unit 201 determines whether the operating body 11 is in contact with the target. If so, it is assumed that work is being performed by the operating body 11, and the process proceeds to S106.

S106: Display of Work Status The work status display unit 203 calculates and displays the angle of the working body 11 during work, the work time, and the like.

S107: Detect end of operation of acting body 11, save work result The work machine/tool recognition unit 201 receives, for example, a signal indicating the state of the switch 14 and recognizes that the work machine/tool 10 has stopped. Alternatively, the work machine/tool recognition unit 201 determines whether or not the acting body 11 is in contact with an object, and if not, considers that the work by the acting body 11 has ended. The work status display unit 203 saves the work status and work status statistics acquired in S106 in a storage area within the memory 242. Furthermore, for example, in response to a request from the manager terminal 30, the work status display unit 203 transmits the saved work status, etc. to the manager terminal 30.

S108: Viewing Work Results The work performance display unit 301 of the manager terminal 30 acquires the work status and the like from the worker terminal 20 and displays them on the display 33.

<Effects>
According to the first embodiment, a 3DAR marker 15 is attached to the work machine or tool 10, and the position of the acting body 11 is expressed as an offset from the 3DAR marker 15. The 3DAR marker 15 is installed in a position that is always visible to the worker terminal 20, so the position of the acting body 11 can be continuously tracked no matter where it is located. Furthermore, the worker terminal 20 can recognize the position and orientation of the 3DAR marker 15 using its inherent function, so it can always accurately estimate the position of the acting body 11, which is fixedly offset from the 3DAR marker 15.

According to this embodiment, the work instruction unit 202 of the worker terminal 20 displays instructions regarding the position, angle, time, etc. at which the work should be performed, as well as the work results, superimposed on real space on the transmissive display 26. This allows the worker to perform the work accurately, uniformly, and efficiently while checking the instructions and work status in real time.

According to this embodiment, the work performance display unit 301 of the manager terminal 30 acquires the work status accumulated by the worker terminal 20 and displays it as work performance on the display 33. This makes it possible to efficiently and automatically keep quality control records that were previously created manually by managers.

<Second Embodiment>
In the above-described first embodiment, the 3DAR marker 15 is attached to the work machine or tool 10, and the worker terminal 20 identifies the position of the operating body 11 by tracking the 3DAR marker 15.

In contrast, in embodiment 2, a self-position estimation device 17 is attached to the work machine or tool 10 instead of the 3DAR marker 15. The worker terminal 20 identifies the position of the operating body 11 by recognizing the position of the self-position estimation device 17.

The hardware configuration of the work status management system 1 according to the second embodiment is the same as that according to the first embodiment (see Figure 1). The work status management system 1 includes a work machine or tool 10, a worker terminal 20, and a manager terminal 30.

Figure 7 is a diagram showing an example of a work tool 10 according to embodiment 2. The work tool 10 typically includes a housing having an operating body 11 and a switch 14, as well as a self-position estimation device 17.

The acting body 11 and switch 14 are the same as in embodiment 1, so their explanation will be omitted.

The self-location estimation device 17 has a processor and memory, and the processor reads and executes programs stored in the memory to realize various functions. The self-location estimation device 17 also has a communication device for communicating with the worker terminal 20, various sensors (e.g., camera, depth sensor, acoustic sensor, etc.) used to detect the amount of movement of the device itself, and a tilt sensor (e.g., gyro, etc.) for detecting the device's posture. Typically, a smartphone or the like can be used as the self-location estimation device 17.

The self-location estimation device 17 sequentially estimates its own movement amount and outputs three-dimensional coordinates indicating its current position. Typical methods for calculating the movement amount include the SLAM (Simultaneous Localization and Mapping) method and the acceleration sensor method. In the SLAM method, the self-location estimation unit 17 senses the surrounding environment while moving using sensors (e.g., cameras, depth sensors, acoustic sensors, etc.) and creates a three-dimensional environmental map. At the same time, it sequentially estimates the movement amount of the self-location estimation device 17 and estimates its own position on the environmental map. In the acceleration sensor method, the self-location estimation device 17 estimates its own position by calculating the movement distance based on the acceleration obtained by the acceleration sensor. In either case, it is possible to improve the estimation accuracy by using information obtained from a tilt sensor (e.g., gyro, etc.).

The self-position estimation device 17 also uses a tilt sensor (e.g., a gyroscope) to measure and output its own attitude, i.e., its tilt relative to the coordinate axes.

The self-localization device 17 also has the function of setting a stationary coordinate system in the real space around it. The origin and coordinate axes of the stationary coordinate system can be defined, for example, by placing a reference object such as an AR marker in the real space. An AR marker is typically a flat plate with a predetermined pattern printed on it, and information indicating the position of the origin and the orientation of the coordinate axes is encoded in the pattern. Alternatively, a three-dimensional object can be used as an AR marker (3DAR marker). For example, a camera provided in the self-localization device 17 captures an AR marker placed in real space. Then, a processor recognizes and analyzes the AR marker included in the captured image to determine the origin and coordinate axes of the stationary coordinate system.

When the self-location estimation device 17 sets a stationary coordinate system, it can use the same AR markers (including 3DAR markers) that the worker terminal 20 uses when setting the stationary coordinate system. This allows the self-location estimation device 17 and the worker terminal 20 to use the same stationary coordinate system. In other words, they can share space.

The self-position estimation device 17 sequentially notifies the worker terminal 20 of its estimated self-position (coordinates in the stationary coordinate system) and attitude (tilt in the stationary coordinate system).

The hardware configuration and functional configuration of the worker terminal 20 and the administrator terminal 30 in embodiment 2 are basically the same as those in embodiment 1 (see Figures 3, 4, 5A, and 5B). Below, we will mainly explain the differences from embodiment 1, and omit the rest.

The work machine/tool recognition unit 201 of the worker terminal 20 receives the position and orientation from the self-position estimation device 17. The work machine/tool recognition unit 201 also stores in advance an offset indicating the relative position between the self-position estimation device 17 and the acting body 11. Therefore, the work machine/tool recognition unit 201 can calculate the coordinates of the acting body 11 based on the position and orientation of the self-position estimation device 17 and the offset indicating the relative position between the self-position estimation device 17 and the acting body 11.

The operation of the work status management system 1 according to the second embodiment will be explained in chronological order using the flowchart in Figure 8.

S201: Startup The worker wears the worker terminal 20 on his/her head and starts up the program. He/she also starts up the self-location estimation device 17. The worker terminal 20 and self-location estimation device 17 use the same AR marker to set a stationary coordinate system in real space and start operation of the SLAM function.

S202: Reading and placing drawing data The work instruction unit 202 reads out drawing data stored in advance in the memory 242 and displays it superimposed on the work site. That is, the drawing data is placed in a stationary coordinate system. The work instruction unit 202 also places an object indicating the insertion position of the work machine or tool 10 in the stationary coordinate system. The placed drawing and object are displayed on the transmissive display 26. This allows the worker to visually recognize the drawing and object superimposed on the real space.

S203: Recognize position of self-position estimation device 17 The self-position estimation device 17 estimates its own position and orientation in a stationary coordinate system using the SLAM function. The work machine/tool recognition unit 201 receives the position and orientation from the self-position estimation device 17.

S204: Estimate position of acting body 11 The work machine/tool recognition unit 201 calculates the coordinates of the acting body 11 based on the position and attitude of the self-position estimation device 17 and an offset indicating the relative position between the self-position estimation device 17 and the acting body 11.

S205: Detect start of operation of acting body 11 The work machine/tool recognition unit 201 receives, for example, a signal indicating the state of the switch 14, and recognizes the operating status of the work machine/tool 10. If the acting body 11 is in operation, the process proceeds to S206.

Alternatively, the work machine/tool recognition unit 201 determines whether the operating body 11 is in contact with the target. If so, it is assumed that work is being performed by the operating body 11, and the process proceeds to S206.

S206: Display of Work Status The work status display unit 203 calculates and displays the angle of the working body 11 during work, the work time, etc.

S207: Detect end of operation of acting body 11, save work result The work machine/tool recognition unit 201 receives, for example, a signal indicating the state of the switch 14, and recognizes that the work machine/tool 10 has stopped. Alternatively, the work machine/tool recognition unit 201 determines whether or not the acting body 11 is in contact with an object, and if not, considers that the work by the acting body 11 has ended. The work status display unit 203 saves the work status and work status statistics acquired in S206 in a storage area within the memory 242. Furthermore, for example, in response to a request from the manager terminal 30, the work status display unit 203 transmits the saved work status, etc. to the manager terminal 30.

S208: Viewing Work Results The work performance display unit 301 of the manager terminal 30 acquires the work status and the like from the worker terminal 20 and displays them on the display 33.

<Effects>
According to the second embodiment, a self-position estimation device 17 is attached to the work machine or tool 10, and the self-position and orientation estimated by the self-position estimation device 17 are transmitted as needed to the worker terminal 20. In the first embodiment described above, the camera 21 of the worker terminal 20 was required to constantly capture the 3DAR marker 15. However, with the above configuration of the second embodiment, the worker terminal 20 can grasp the position and orientation of the self-position estimation device 17 even when the camera 21 is not capturing the work machine or tool 10. Furthermore, since the processing load on the worker terminal 20 is smaller than in the first embodiment, the self-position estimation device 17 and the operating body 11 can be tracked more smoothly.

Although the above describes an embodiment of the present invention, the present invention is not limited to this, and components and various processes can be modified, deleted, or added as appropriate without departing from the spirit of the present invention. For example, each processing means constituting the present invention may be configured using hardware, or any process may be realized by having a CPU execute a computer program. Furthermore, computer programs can be stored using various types of temporary or non-temporary computer-readable media and supplied to a computer. Temporary computer-readable media include, for example, electromagnetic signals supplied to a computer via wire or wirelessly.

REFERENCE SIGNS LIST 1 work status management system 10 work machine/tool 11 action body 14 switch 15 3DAR marker 17 self-position estimation device 20 worker terminal 21 camera 22 depth sensor 23 inertial sensor 24 processing unit 241 CPU
242 Memory 25 Communication unit 26 Transmissive display 30 Administrator terminal 31 Processing unit 311 CPU
312 Memory 32 Communication unit 33 Display 201 Work machine/tool recognition unit 202 Work instruction unit 203 Work status display unit 301 Work performance display unit

Claims (7)

A work status management system including a work machine and a worker terminal,
The working machine or tool has an action body that applies a predetermined action to a target and a 3DAR marker,
The worker terminal includes:
a work machine/tool recognition unit that captures an image of the 3DAR marker with a camera, recognizes a position and orientation of the 3DAR marker based on the captured image, and estimates a position of the operating body based on the position and orientation of the 3DAR marker;
a work instruction unit that displays an object indicating a position where the acting body should act on a transmissive display so that the object is superimposed on the target and visible;
a work status display unit that determines whether the position of the operating body estimated by the work machine/tool recognition unit is within a range indicated by the object, and displays the determination result on the transmissive display. The work status management system according to claim 1 , wherein the work status display unit estimates a distance between the target and the position of the operating body, and displays the distance together with or instead of the determination result. The work status management system according to claim 1 , wherein the work status display unit estimates an angle between the target and the position of the operating body, and displays the angle together with or instead of the determination result. the work machine or tool further includes a switch that activates or stops the acting body and notifies the worker terminal of whether the acting body is operating,
the work machine/tool recognition unit determines whether the operating body is operating based on the notification from the switch;
The work status management system of claim 2, wherein the work status display unit estimates a work time based on a determination result of whether the acting body is operating or not and a result of the distance estimation, and displays the work time together with at least one of the determination result and the distance, or instead of the determination result and the distance. A work status management system including a work machine and a worker terminal,
the work machine or tool includes an action body that applies a predetermined action to an object and a self-position estimation device;
the self-position estimation device estimates its own position and orientation and transmits the estimates to the worker terminal;
The worker terminal includes:
a work machine/tool recognition unit that estimates the position of the working body based on the position and orientation received from the self-position estimation device;
a work instruction unit that displays an object indicating a position where the acting body should act on a transmissive display so that the object is superimposed on the target and visible;
a work status display unit that determines whether the position of the operating body estimated by the work machine/tool recognition unit is within a range indicated by the object, and displays the determination result on the transmissive display. a work machine/tool recognition step of photographing a 3DAR marker provided on the work machine/tool with a camera, recognizing the position and orientation of the 3DAR marker based on the photographed image, and estimating the position of an acting body provided on the work machine/tool for applying a predetermined action to an object based on the position and orientation of the 3DAR marker;
a work instruction step of displaying an object indicating a position where the acting body should act on the target on a transmissive display so that the object is superimposed on the target and visible;
a work status display step of determining whether the position of the operating body estimated by the work machine/tool recognition unit is within a range indicated by the object, and displaying the determination result on the transmissive display. A program for causing a computer to execute the method described in claim 6.