JP7608239B2 - Technology transfer tracing system and technology transfer tracing method - Google Patents

Description

The present invention relates to a system and method for tracing the transfer of skills of workers in specific tasks, such as at construction sites.

For example, when unskilled workers perform various tasks at a construction site, it is desirable for them to work under the direct guidance of an experienced worker. However, experienced workers are not always present at the actual site, and there may be situations where unskilled workers must work alone.

For example, Patent Document 1 discloses a motion teaching device that uses a head-mounted display device that allows the user to simultaneously view an image of the external world superimposed on an artificial image. In Patent Document 1, a video of a specific motion performed by an expert is prepared in advance, and the video can be superimposed on the user's image of the external world on the head-mounted display device.

Patent Document 2 also discloses a work instruction system that uses a computer to instruct an unskilled worker on the actions of a skilled worker. In Patent Document 2, the actions of a skilled worker performing a specified task using a tool are detected by a sensor and transferred to a processing device as action data. The processing device displays the action data three-dimensionally as a 3DCG virtual image on an output device. This allows an unskilled worker to perform the specified task while looking at the virtual image.

Japanese Patent Application Publication No. 7-333550 JP 2001-393188 A

However, the images of the expert prepared in advance did not necessarily match the actual visual field image of the worker, and from the perspective of passing on the skills of the expert to future generations, there was room for improvement in accuracy.

The present invention has been made in consideration of the above points, and aims to provide a technology transfer system that can transfer the skills of experts to future generations with greater accuracy.

The technology transfer tracing system of one embodiment of the present invention is a technology transfer tracing system that supports the transfer of skills of an expert to a worker, and includes a state detection device that detects the posture and state of a subject, an imaging device that images the subject's surrounding environment, and an information processing device that generates motion model data related to a specified task of the expert based on outputs previously acquired from the state detection device and the imaging device during the specified task of the expert. The information processing device generates actual motion data corresponding to the actual motion of the worker based on outputs acquired from the state detection device and the imaging device during the specified task of the worker, compares the motion model data with the actual motion data, converts the motion model data into a format corresponding to the actual motion data, and displays it on a display device.

In addition, a technique transfer tracing method according to one aspect of the present invention is a technique transfer method for supporting the transfer of an expert's technique to a worker, and includes the steps of: generating motion model data for a prescribed task performed by an expert based on outputs acquired in advance from a state detection device that detects the posture and state of the subject and an imaging device that images the subject's surrounding environment; generating actual motion data corresponding to the actual motion of the worker based on outputs acquired from the state detection device and the imaging device during the prescribed task performed by the worker; comparing the motion model data with the actual motion data; and converting the motion model data into a format corresponding to the actual motion data based on the comparison result between the motion model data and the actual motion data, and displaying the converted data on a display device.

The present invention makes it possible to pass on the skills of experts to future generations with greater precision.

1 is a schematic configuration diagram showing an example of a technology transfer system according to an embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of the head mounted display device according to the embodiment. FIG. 13 is a conceptual diagram illustrating an example of a predetermined task according to the embodiment. FIG. 11 is a diagram illustrating a flow of creating an expert motion model according to the embodiment; 1 is a diagram illustrating an image display flow of a head mounted display device according to an embodiment. FIG. 1A to 1C are diagrams illustrating an example of an image display of a head mounted display device according to an embodiment. 1A to 1C are diagrams illustrating an example of an image display of a head mounted display device according to an embodiment. FIG. 2 is a diagram illustrating a hardware configuration of a computer for realizing an information processing device and a server device according to the embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. Note that the same reference numerals are used to designate corresponding elements in multiple drawings.

Figure 1 is a schematic diagram showing an example of a technology transfer tracing system according to an embodiment. The technology transfer tracing system 100 according to this embodiment (hereinafter simply referred to as system 100) enables a worker at a specific work site, such as a construction site, to easily trace the work of an expert when performing a specific task. For example, system 100 supports the transfer of skills of an expert worker (which may also be called an expert) to a relatively unskilled worker (hereinafter simply referred to as a worker). Note that, in this embodiment, a specific task at a construction site is described as an example, but this is not limited to this and can be changed as appropriate. For example, the work site may be a construction site such as road construction or a care site at a care facility.

As shown in FIG. 1, the system 100 includes a head mounted display device 10 (hereinafter referred to as HMD device 10) that can be worn by a specific worker, and a server device 20 that systematically stores and manages motion model data related to specific tasks performed by skilled workers. The HMD device 10 and the server device 20 can communicate with each other via a network. The HMD device 10 and the server device 20 may communicate with a network (such as the Internet) via wired and/or wireless (e.g., LTE (Long Term Evolution), NR (New Radio), Wi-Fi (registered trademark), etc.).

The HMD device 10 is composed of a display device that is worn on a person's head. The HMD device 10 may be a non-transparent type that completely covers the eyes, or a transparent type display device that overlays an image (a virtual image such as a video) through a screen onto the real visual field. The HMD device 10 may also be called a wearable terminal. Details of the HMD device 10 will be described later.

In addition, subjects who wear the HMD device 10 include workers who work at construction sites, factories, and other such sites, as described above, as well as caregivers who care for people in need of care (care recipients) who live in care facilities. In this embodiment, the subject may also be called a worker or an expert.

The server device 20 systematically stores motion model data related to specific tasks performed by skilled workers at a work site by task type and time series. The motion model data will be described later. The functions of the server device 20 may be built into the HMD device 10.

Fig. 2 is a block diagram showing an example of the functional configuration of a head mounted display device according to an embodiment. As shown in Fig. 2, the HMD device 10 includes a state detection device 1, an imaging device 2, an information processing device 3, a display device 4, and a helmet 5 (see Fig. 3). The various components of the state detection device 1, the imaging device 2, the information processing device 3, and the display device 4 are mounted on the helmet 5 worn by the worker.

The state detection device 1 detects the state and posture of a worker wearing the HMD device 10. The "state" of the worker represents the state of a specific movement (e.g., walking), and the "posture" of the worker represents the current posture of the worker or changes thereto (e.g., standing posture, sitting posture, etc.).

Specifically, the condition detection device 1 includes a sensor 11, such as a gyro sensor or acceleration sensor, that outputs detection values corresponding to the subject's movements, and a memory 12. While the subject is working, the detection values output from the sensor 11 are repeatedly stored in the memory 12.

The condition detection device 1 also outputs (transmits) the detection value of the sensor 11 to, for example, the imaging device 2 or the information processing device 3. The detection value may be output (transmitted) continuously or at predetermined time intervals. The detection value output from the sensor 11 is used to estimate the condition and posture of the worker, as will be described in detail later.

The imaging device 2 captures images of the worker's surrounding environment and is attached to the worker's body or any other location than the worker's body. For example, when the imaging device 2 is attached to the worker's body, the imaging device 2 is attached to the helmet 5 worn by the worker, as described above. When the imaging device 2 is attached to any other location than the subject's body, the imaging device 2 may be installed at any location within a construction site, factory, or care facility, or may be mounted on an unmanned mobile object such as a drone or robot.

The imaging device 2 also includes a camera 21 and a memory 22. When the imaging device 2 is attached to the subject's body, the camera 21 is an action camera, a 360-degree camera, or the like, which captures images (still images or moving images) in the subject's field of view or around the subject, and stores the captured images in the memory 22. When the imaging device 2 is installed in any location other than the subject's body, the camera 21 is a fixed surveillance camera that is installed in advance at a construction site, factory, or nursing facility, or a camera equipped on an unmanned mobile object, which captures images of the construction site, factory, or nursing facility, including the subject, and stores the captured images in the memory 22.

The imaging device 2 reads out the image captured by the camera 21 from the memory 22, and outputs (transmits) the read image to the information processing device 3. Note that the output (transmission) of the image may be performed continuously or at predetermined time intervals.

The information processing device 3 is composed of a computer that generates predetermined data based on the output of the above-mentioned state detection device 1 and imaging device 2, and controls the display of the display device 3, which will be described later. The information processing device 3 estimates the state, posture, etc. of the worker, and displays the estimated state and posture as an image superimposed on the screen. Specifically, the information processing device 3 includes, as functional blocks, an extraction unit 31, an estimation unit 32, an integration unit 33, and a comparison unit 34. The extraction unit 31, the estimation unit 32, the integration unit 33, and the comparison unit 34 may be collectively referred to as a control unit. Note that the information processing device 3 is not limited to these functional blocks, and may also have other functional blocks.

The extraction unit 31 extracts and analyzes a specific object in an image transmitted from the imaging device 2 based on the image. Here, the object refers to a structure contained in the image, such as a "floor," "pillar," "stairs," and "handrail" shown in FIG. 3. The extraction unit 31 may also generate three-dimensional environmental data corresponding to the extracted object. The three-dimensional environmental data is three-dimensional data that indicates the environment around the worker during a specific task, and includes data in which the above-mentioned objects are three-dimensionalized. The extraction unit 31 may extract objects that respectively correspond to a "floor," "pillar," "stairs," and "handrail" in an arbitrary three-dimensional space, and generate three-dimensional environmental data corresponding to each object for each time series at a specific time.

The estimation unit 32 generates three-dimensional skeleton data indicating the movement (posture) of the worker at a specified time based on the detection values for a specified time transmitted from the state detection device 1 and the images for a specified time transmitted from the imaging device 2. That is, the estimation unit 32 estimates the state and posture of the worker at a specified time. Here, the three-dimensional skeleton data is three-dimensional data indicating the skeleton of the worker and the trajectory of the skeleton when the worker is performing a specified task in an arbitrary three-dimensional space. The three-dimensional skeleton data may also be called three-dimensional movement data.

FIG. 3 is a conceptual diagram showing an example of a predetermined task according to the embodiment. As shown in FIG. 3, for example, a worker at a construction site is assumed to be performing a task of welding a handrail of a staircase as a predetermined task. In this case, the estimation unit 32 generates 3D skeleton data showing the motion of the worker while crouching and performing welding. The estimation unit 32 may integrate data based on the state of the worker (state data) and data based on the posture of the worker (posture data) to generate 3D skeleton data for each time series at a predetermined time. Note that, in FIG. 3, welding of a handrail of a staircase is described as an example of a predetermined task, but this is not limited thereto and can be changed as appropriate. Other examples of the predetermined task include various tasks such as screwing and cutting sheet metal.

The integration unit 33 integrates the above-mentioned object (or environmental 3D data) and skeletal 3D data to generate motion model data. The motion model data is generated based on the output of the state detection device 1 and the imaging device 2 by having a skilled person wearing the HMD device 10 perform a specified task in advance. The generated motion model data may be stored in a storage area within the information processing device 3, or may be systematically stored (saved) for each task in a database of the server device 20 described later. The integration unit 33 also generates actual motion data based on the output obtained from the state detection device 1 and the imaging device 2 during the specified task performed by the worker.

The comparison unit 34 compares and collates the expert's motion model data with the worker's actual motion data, and converts the expert's motion model data generated as described above into a format corresponding to the actual motion data. For example, the comparison unit 34 converts the motion model data into mixed reality data for display on the display device 4 using MR (Mixed Reality) technology.

Note that each functional configuration of the information processing device 3 may be realized by the server device 20 described below.

The display device 4 is configured as a display provided at the bottom front of the helmet 5. The display device 4 is configured as, for example, a transmissive display, and is configured to superimpose the above-mentioned motion model data (mixed reality data) on the actual visual field image of the worker. The display device 4 may also display the image captured by the imaging device 2 as a video (video). Note that in this embodiment, the display device 4 is described as being part of a so-called head-mounted display device mounted on the helmet 5, but is not limited to this and can be modified as appropriate. For example, the display device 4 may be configured as a mobile device such as a smartphone or tablet terminal, or a personal computer.

As described above, the server device 20 has a database that systematically stores motion model data relating to predetermined tasks of skilled workers at a work site by task type and time series.
For example, the motion model data for a specific task of an expert is
(1) Name of the work (e.g. welding, screwing, part positioning, etc.)
(2) Work order (Step 1, Step 2, etc.)
(3) Working time (*** hours, *** minutes, *** seconds, etc.)
The items are set for each predetermined task and for each time series.

This database systematically stores motion model data corresponding to each task for multiple tasks. This motion model data is used as reference data (thresholds) to determine whether or not a task is being performed according to the correct procedure when an actual worker is performing a specified task. Note that the motion model data may also include, in addition to the above (1)-(3), information on tips for the specified task and points requiring attention.

When unskilled workers perform various tasks at construction sites, etc., it is desirable for them to work under the direct guidance of an experienced worker. However, experienced workers are not always present at actual work sites, and situations may arise where unskilled workers must work alone.

For example, until now, it was common for workers to learn tasks through trial and error, either by looking at a paper manual or by listening to verbal instructions from an expert during on-the-job training (OJT).

Nowadays, on-site workers can share the video they are watching with a veteran (experienced) operator in a remote location, and work while receiving instructions. There are also efforts being made to record the work process on video, etc., and to understand what the workers are doing by watching the video.

However, with conventional technology, it takes time for a novice worker to reach the same level of work, speed, and quality as an experienced worker. In addition, rework (redoing work) due to incorrect work execution is a factor that affects subsequent processes (schedules).

The inventors of the present invention therefore came up with the aim of enabling even relatively inexperienced workers to effectively learn tasks without receiving direct instruction from an expert. For example, in this embodiment, an expert is made to wear the HMD device 10 in advance and perform a specified task, and motion model data of the expert is generated and stored in a database. The actual worker then performs the task while wearing the HMD device 10. At this time, the HMD device 10 estimates the worker's own posture and motion (state) several seconds later from camera footage and sensor data, and displays an image (virtual data) of the expert's correct motion on the display based on the motion model data of the expert.

This allows an actual worker to see the movements (correct movements) of an expert in a particular task as an image superimposed on the real work space, allowing the worker to check the correct procedure of the expert while working. As a result, efficient and accurate work can be achieved. Furthermore, by having the worker actually trace the movements of the expert while working, the efficiency of learning the task and the speed of skill acquisition can be increased, and even relatively unskilled workers can achieve work of the same quality as an expert. In other words, it is possible to pass on the skills of an expert to future generations with greater precision.

More specifically, the system 100 according to this embodiment supports the transfer of the skills of an expert to a worker, and includes a state detection device 1 that detects the posture and state of a subject, an imaging device 2 that images the subject's surrounding environment, and an information processing device 3 that generates motion model data related to a specified task of the expert based on the output previously acquired from the state detection device 1 and the imaging device 2 during the specified task of the expert. The information processing device 3 generates actual motion data corresponding to the actual motion of the worker based on the output acquired from the state detection device 1 and the imaging device 2 during the specified task of the worker, compares the motion model data with the actual motion data, converts the motion model data into a format corresponding to the actual motion data, and displays it on the display device 4. This allows the worker to work while checking the motion of the expert superimposed on the screen, and improves the learning efficiency of the worker.

In addition, in this embodiment, the information processing device 3 compares the actual movement data with the converted movement model data to determine whether the specified work of the worker is correct. With this configuration, the difference between the actual movement of the worker and the movement of an expert can be clearly recognized, and the learning efficiency of the worker can be further improved.

In addition, in this embodiment, the information processing device 3 extracts objects related to the surrounding environment from the captured image of the subject's surrounding environment, generates 3D skeletal data related to the subject's skeleton based on the subject's posture and state, and integrates the objects and 3D skeletal data to generate motion model data or actual motion data. With this configuration, it is possible to generate data that estimates the actual motion of the worker from multiple data.

In addition, in this embodiment, the state detection device 1, the imaging device 2, the information processing device 3, and the display device 4 constitute an HMD device 10 (wearable terminal) that can be worn by the subject. By realizing these configurations in a single device, it is possible to learn the movements of an expert with a simple configuration without interfering with the work of the worker.

In addition, in this embodiment, the system 100 further includes a server device 20 that systematically stores operation model data for each of a number of predetermined tasks. This configuration makes it possible to systematically manage all tasks at a work site.

Next, the technology transfer tracing method according to the present embodiment will be described with reference to Figs. 4 to 7. Fig. 4 is a diagram illustrating an example of an expert's behavior model creation flow according to the embodiment. Fig. 5 is a diagram illustrating an example of an image display flow of the HMD device according to the embodiment. Figs. 6 and 7 are diagrams illustrating example image displays of the HMD device according to the embodiment. In the flow shown below, unless otherwise specified, the subject of the operation is the information processing device 3 (control unit). Note that the flow shown below is merely an example, and other processes may be added as necessary. Also, the order of each process (step) can be changed as long as no contradiction occurs.

The technique transfer tracing method according to the present embodiment is a technique transfer method for supporting the transfer of techniques from an expert to a worker, and includes the following steps.
(1) A step of generating motion model data for a specified task of an expert based on outputs acquired in advance from a state detection device 1 that detects the posture and state of a subject and an imaging device 2 that images the subject's surrounding environment (steps ST101-104 in FIG. 4).
(2) A step of generating actual action data corresponding to an actual action of a worker based on outputs obtained from the state detection device 1 and the imaging device 2 during a predetermined task performed by the worker (steps ST201-204 in FIG. 5).
(3) A step of comparing the behavior model data with the actual behavior data (step ST205 in FIG. 5).
(4) A step of converting the behavior model data into a format corresponding to the actual behavior data based on the result of comparison between the behavior model data and the actual behavior data, and displaying the converted behavior model data on a display device (step ST206 in FIG. 5).
(5) A step of comparing the actual action data with the converted action model data to determine whether or not the predetermined action of the worker is correct (step ST207 in FIG. 5).

First, the flow for creating an expert movement model will be described. In this flow, expert movement model data corresponding to a specified task is generated and prepared in advance in the server device 20. In this case, the expert performs the specified task while wearing the HMD device 10.

As shown in FIG. 4, first, in step ST101, the information processing device 3 acquires various data. Specifically, the information processing device 3 acquires data related to the posture and state of the skilled worker during a specified task from the state detection device 1, and acquires image data (captured image) corresponding to the surrounding environment of the skilled worker from the imaging device 2.

In the next step ST102, the information processing device 3 extracts a predetermined object (A) corresponding to the work site environment from the images captured by the imaging device 2 in chronological order. For example, the information processing device 3 focuses on both the moving image of the skilled worker's body recorded by the imaging device 2 and the moving image showing the skilled worker's surroundings, and extracts which objects (e.g. pillars, beams, tools, etc.) are involved during the work and at what point in time. Note that when extracting objects, it is possible to utilize an AI model that has previously learned each object at the work site.

In the next step ST103, the information processing device 3 generates three-dimensional skeletal data (B) that estimates the state and posture of the skilled worker during the specified task. For example, the information processing device 3 focuses on both the moving image of the skilled worker's body recorded in the imaging device 2 and the moving image showing the skilled worker's surroundings, and estimates the skilled worker's three-dimensional posture (e.g. standing upright, crouching, lying down, etc.). The information processing device 3 also focuses on the previous moving image and data from the sensor 11, and estimates the skilled worker's three-dimensional state (e.g. walking, ascending and descending, falling, etc.).

In the next step ST104, the information processing device 3 integrates the object (A) and the three-dimensional skeleton data (B) to generate motion model data (A+B) of an expert. The generated motion model data is stored in a predetermined storage area (the information processing device 3 or the server device 20).

Next, the screen display flow will be described. Note that steps ST201-204 in FIG. 5 are the same as steps ST101-104 in FIG. 4, except that "expert" is replaced with "operator" and "action model data" is replaced with "actual action data."

As shown in FIG. 5, first, in step ST201, the information processing device 3 acquires various data. Specifically, the information processing device 3 acquires data related to the posture and state of the worker during a specified task from the state detection device 1, and acquires image data (captured image) corresponding to the surrounding environment of the worker from the imaging device 2.

In the next step ST202, the information processing device 3 extracts a predetermined object (C) corresponding to the work site environment from the images captured by the imaging device 2 in chronological order. For example, the information processing device 3 focuses on both the moving image of the worker's body recorded by the imaging device 2 and the moving image showing the worker's surroundings, and extracts which objects (e.g. pillars, beams, tools, etc.) are involved during the work and at what point in time. Note that when extracting objects, it is possible to utilize an AI model that has previously learned each object at the work site.

In the next step ST203, the information processing device 3 generates three-dimensional skeletal data (D) that estimates the state and posture of the worker during the specified task. For example, the information processing device 3 focuses on both the moving image of the worker's body recorded by the imaging device 2 and the moving image showing the worker's surroundings, and estimates the worker's three-dimensional posture (e.g. standing upright, crouching, lying down, etc.). The information processing device 3 also focuses on the previous moving image and data from the sensor 11, and estimates the worker's three-dimensional state (e.g. walking, ascending and descending, falling, etc.).

In the next step ST204, the information processing device 3 integrates the object (C) and the three-dimensional skeleton data (D) to generate actual movement data (C+D) of the worker.

In the next step ST205, the information processing device 3 reads out the movement model data of the expert from the server device 20 and compares and collates it with the actual movement data of the worker.

In the next step ST206, the information processing device 3 converts the expert's motion model data according to the worker's state (actual motion data) and displays it superimposed on the screen. The work sites of the expert and the worker do not necessarily coincide, and by reflecting the actual worker's motion in the expert's motion model data, it is possible to improve the reproducibility of the teaching data (motion model data after conversion) in real space.

In the next step ST207, the information processing device 3 superimposes on the screen the comparison result between the converted action model data and the actual action data. This provides feedback on whether the worker's actions match those of an expert, allowing the worker to visually recognize the accuracy of the worker's actions. For example, the information processing device 3 may determine whether the work is based on correct actions based on whether the amount of deviation on the screen between the converted action model data and the actual action data is within a predetermined threshold.

According to such a screen display flow, for example, as shown in FIG. 6 and FIG. 7, an image based on the actual movement data of the worker is displayed on the screen with a solid line, and the converted movement model data of the expert is displayed with a two-dot chain line. The work order and work time are displayed in the upper left of the screen. Also, as shown in FIG. 7, if the deviation amount on the screen between the converted movement model data and the actual movement data is within a predetermined threshold, the work is based on a correct movement and a message "OK!" is displayed. For example, if the deviation amount on the screen between the converted movement model data and the actual movement data exceeds a predetermined threshold, the work is not based on a correct movement and a message "NG!" may be displayed. Such a message (alert) may not only be displayed on the screen, but may also be notified by voice. Also, the information processing device 3 may control the image display so that when the work in Step 1 in FIG. 6 is judged to be OK, the process proceeds to Step 2 shown in FIG. 7.

As described above, according to this embodiment, the learning efficiency of actual workers can be improved by converting previously acquired motion model data for a specific task performed by an expert to match the visual space of the actual worker and displaying it on the screen.

Figure 8 is a diagram showing an example of a computer for implementing the information processing device 3 or the server device 20. For ease of explanation, the information processing device 3 and the server device 20 will be described using the same reference numerals.

The computer 70 shown in FIG. 8 includes a processor 71, a memory 72, a storage device 73, a reader 74, a communication interface 75, and an input/output interface 76. The processor 71, the memory 72, the storage device 73, the reader 74, the communication interface 75, and the input/output interface 76 are connected to each other via a bus 77.

The processor 71 is configured with a single processor, a multiprocessor, a multicore, etc. The processor 71 also reads out a program stored in the storage device 73 and uses the memory 72 to execute a program that describes the procedures of the flowcharts shown in Figures 4 and 5, thereby providing some or all of the functions of the control unit (e.g., the extraction unit 31, the estimation unit 32, the integration unit 33, and the control unit 34).

Memory 72 may be a semiconductor memory or the like, and may include a RAM (Random Access Memory) area and a ROM (Read Only Memory) area.

The storage device 73 is a semiconductor memory such as a hard disk or flash memory, or an external storage device.

The reader 74 accesses the removable storage medium 78 according to instructions from the processor 71. The removable storage medium 78 is realized by a semiconductor device, a medium where information is input and output by magnetic action, a medium where information is input and output by optical action, etc. An example of a semiconductor device is a USB (Universal Serial Bus) memory, etc. An example of a medium where information is input and output by magnetic action is a magnetic disk, etc. An example of a medium where information is input and output by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc (Blu-ray is a registered trademark), etc.

The communication interface 75 communicates with other devices according to instructions from the processor 71. For example, the information processing device 3 collects information from the state detection device 1, the imaging device 2, and the server device 20 via the communication interface 75.

The input/output interface 76 is, for example, an interface between an input device and an output device. The input device is a device such as a keyboard, mouse, or touch panel that accepts instructions from a subject or an administrator who checks the mixed reality data, augmented reality data, or virtual reality data. The output device is a display device such as a display, or an audio device such as a speaker.

Each program according to the embodiment is provided to the information processing device 3 in the following form, for example.
(1) It is pre-installed in the storage device 73.
(2) Provided by a removable storage medium 78.
(3) Provided from a server such as a program server.

The hardware configuration of the computer 70 for realizing the information processing device 3 or the server device 20 described with reference to FIG. 8 is an example, and the embodiment is not limited to this. For example, part of the above-mentioned configuration may be deleted, or new configuration may be added. In another embodiment, for example, part or all of the functions of the above-mentioned information processing device 3 may be implemented as hardware such as an FPGA, a System-on-a-Chip (SoC), an Application Specific Integrated Circuit (ASIC), or a PLD.

The above describes the embodiments. However, the embodiments are not limited to the above embodiments, and should be understood to include various modified and alternative forms of the above embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments.

100 Technology transfer tracing system 1 State detection device 2 Imaging device 3 Information processing device 4 Display device 5 Helmet 10 Head-mounted display device (wearable terminal)
11 Sensor 12 Memory 21 Camera 22 Memory 31 Extraction section 32 Estimation section 33 Integration section 34 Comparison section

Claims (6)

A technology transfer tracing system that supports the transfer of skills of an expert to a worker,
A state detection device that detects the posture and state of a subject;
an imaging device that captures an image of the subject's surroundings;
an information processing device for extracting objects related to the expert's surrounding environment from an image of the subject's surrounding environment previously acquired from the imaging device during a predetermined task by the expert, generating three-dimensional skeleton data related to the skeleton of the expert based on the posture and state of the subject previously acquired from the state detection device, and integrating the objects related to the expert's surrounding environment and the three-dimensional skeleton data related to the skeleton of the expert to generate motion model data related to the predetermined task by the expert,
the information processing device extracts objects related to the worker's surrounding environment from an image of the subject's surrounding environment acquired from the imaging device during the specified task of the worker, generates three-dimensional skeletal data related to the worker's skeleton based on the posture and state of the subject acquired from the state detection device, integrates the objects related to the worker's surrounding environment and the three-dimensional skeletal data related to the worker's skeleton to generate actual movement data corresponding to the actual movement of the worker, compares the movement model data with the actual movement data, converts the movement model data into a format corresponding to the actual movement data, and displays it on a display device. 2. The technology transfer tracing system according to claim 1 , wherein the information processing device compares the actual action data with the converted action model data to determine whether the predetermined work of the worker is correct. The technology transfer tracing system according to claim 1 or 2 , wherein the state detection device, the imaging device, the information processing device, and the display device constitute a wearable terminal that can be worn by a subject person. 4. The technology transfer tracing system according to claim 1 , further comprising a server device that systematically stores said behavior model data for each of said plurality of predetermined operations. A technique transfer tracing method for supporting the transfer of a technique of an expert to a worker, comprising the steps of:
extracting an object related to the surrounding environment of the expert from a captured image of the surrounding environment of the expert , the captured image having been acquired in advance from an imaging device that captures an image of the surrounding environment of the expert, during a predetermined task performed by the expert;
generating three-dimensional skeleton data relating to the skeleton of the expert based on a posture and a state of the expert, the posture and the state of the expert being acquired in advance from a state detection device that detects a posture and a state of the expert during a predetermined task performed by the expert;
A step of integrating objects related to the surrounding environment of the expert and three-dimensional skeleton data related to the skeleton of the expert to generate motion model data related to the predetermined task of the expert;
extracting an object related to a surrounding environment of the worker from a captured image of the surrounding environment of the subject acquired from the imaging device during the predetermined task of the worker;
generating three-dimensional skeleton data regarding a skeleton of the worker based on a posture and a state of the subject acquired from the state detection device during the predetermined task of the worker;
generating actual movement data corresponding to an actual movement of the worker by integrating objects related to the surrounding environment of the worker and three-dimensional skeleton data related to the skeleton of the worker ;
comparing the motion model data with the actual motion data;
a step of converting the behavior model data into a format corresponding to the actual behavior data based on a comparison result between the behavior model data and the actual behavior data, and displaying the converted behavior model data on a display device;
The above-mentioned method for tracing technology transfer is executed by an information processing device . 6. The technology transfer tracing method according to claim 5 , further comprising the step of: comparing the actual action data with the converted action model data to determine whether the predetermined work performed by the worker is correct.

Images