JP7753028B2 - Occupational training support system and content creation program for the occupational training support system - Google Patents

Description

The present invention relates to technology for efficiently educating workers engaged in work at construction sites, etc., and relates to a work training support system that is particularly suitable for situations where it is difficult for an instructor to provide direct instruction, as well as a content creation program used in this system.

Recently, workplaces have been experiencing a shortage of experienced and mid-career workers with the know-how, knowledge, and skills required for specific tasks. Furthermore, in a climate where infection control measures such as avoiding contact with others as much as possible are being advocated, it has become difficult to implement regular on-the-job training (OJT), making it difficult to accumulate and pass on work know-how and important points to bear in mind (such as how to do things, posture, and the work environment).

In this current situation, many workers lack basic knowledge of on-site work, and because they are unable to observe work sites, they often are unable to understand the atmosphere associated with different types of work (for example, single work versus collaborative work). Even if they attend general work courses, they often fail to acquire practical skills.

In this context, technology such as that disclosed in Patent Document 1, which combines virtual reality (VR) technology with a simulated tool equipped with an actuator, is attracting attention, allowing users to safely experience dangerous movements (the impact of kickback) that occur when using handheld rotary tools.

Patent Documents 2 and 3 also disclose techniques for easily and efficiently creating VR content, as a way to broaden the scope of application of VR technology. For example, Patent Document 2 discloses the configuration of a program that automatically creates VR content by applying predetermined VR actions to video files uploaded by users. Patent Document 3 also discloses a VR content management system that creates a desired virtual space by placing pre-created VR object data in the virtual space.

JP 2013-114066 A Japanese Patent Application Laid-Open No. 2018-190177 JP 2019-168934 A

The experience device disclosed in Patent Document 1 allows users to safely learn about the dangers of specific tasks. However, the device disclosed in Patent Document 1 is a device specialized for specific tasks, and if users want to experience other technologies or tasks, they will need separate systems.

Furthermore, the VR content management and creation systems disclosed in Patent Documents 2 and 3 are configured to apply data under specific conditions in order to simplify the creation of VR content. As a result, in order to create VR content that takes into account different movements, line of sight, etc. for each task, it becomes necessary to define the movements and reactions of the content for each pattern, which requires work by someone with specialized knowledge.

The present invention aims to provide an occupational training support system that can be adapted to various tasks, as well as a content creation program for the occupational training support system that can easily create VR content for use with this system.

To achieve the above objective, the work training support system of the present invention is an education support system that allows inexperienced workers engaged in on-site work to learn the skills of experienced workers, and is characterized by comprising: a measuring device that detects physical quantity data and eye movement data of the virtual experiencer's movements and posture for a specific task; a display device that displays a field of view image in a virtual space; and a processing device that outputs the field of view image to the display device and, if the virtual experiencer is an experienced worker, records the physical quantity data and eye movement data as training data; if the virtual experiencer is an inexperienced worker, compares the physical quantity data and eye movement data with the training data to calculate a difference, and performs processing to determine the level of proficiency corresponding to the difference based on a predetermined reference value.

Furthermore, in an occupational training support system having the above-described characteristics, the measuring device may preferably include a target or camera that detects the movements of the virtual participant, and a sensor that detects the virtual participant's line of sight and physical quantities based on the movements.

Furthermore, in an occupational training support system having the above-mentioned features, it is desirable that the display device include an HMD (Head Mounted Display) worn by the virtual participant, and that various operational instructions be shown to the virtual participant via the HMD. By having such features, the virtual participant can become immersed in the VR space, giving the virtual work a sense of realism.

Furthermore, in an occupational training support system with the above-described features, when the proximity distance between objects falls below a predetermined value in the display of the field of view image, the appropriate joint state defined between the objects can be expressed and displayed. These features make it possible to grasp small objects, which was previously difficult to express, and to express the accurate joint state between objects.

Furthermore, to achieve the above-mentioned objective, the content creation program for an occupational training support system according to the present invention comprises a measuring device that detects physical quantity data and gaze movement data of the virtual experiencer's movements and posture for a specific task; a display device that displays a field of view image in a virtual space; and a processing device that outputs the field of view image to the display device and, if the virtual experiencer is an experienced worker, records the physical quantity data and gaze movement data as training data; if the virtual experiencer is an unskilled worker, compares the physical quantity data and gaze movement data with the training data to calculate a difference and determines a proficiency level corresponding to the difference based on a predetermined reference value. This content creation program is applicable to an occupational training support system that teaches unskilled workers engaged in on-site work the skills of experienced workers, and is characterized in that it creates a video of work experience content by selecting and adding additional information about the unskilled worker recorded in the processing device to video captured during the virtual experience by the experienced worker, and then converts the video of the work experience content into a DataSheet appropriate for the program and outputs it.

The occupational education support system with the above-mentioned features makes it possible to create and use occupational experience content corresponding to a variety of occupations. Furthermore, the content creation program with the above-mentioned features allows users to simply select and add additional information along with the video, and a DataSheet corresponding to the playback program is output, making it easy for anyone, even those without specialized knowledge, to create occupational experience content.

1 is a block diagram showing a configuration of an occupational training support system according to an embodiment. FIG. 2 is a flow diagram showing the overall flow of a work experience using the work training support system according to the embodiment. FIG. 10 is a flow chart showing details of a work experience by a skilled worker. FIG. 10 is a flow diagram showing a flow when editing work experience content. FIG. 10 is a diagram showing the movement of the eyes of a skilled worker during slinging work. FIG. 10 is a diagram showing the movement of the line of sight of an unskilled worker during slinging work. FIG. 10 is a diagram illustrating a comparison of approximation models based on feature amounts of eye movement during transportation work. FIG. 10 is a diagram showing a state at the start of work, among the field of view images during work experience by an unskilled worker. FIG. 10 is a diagram showing a state in which a wrench is lifted up in a field of view image during a work experience by an unskilled worker. FIG. 10 is a diagram showing a state in which a nut, which is a work target, is displayed in a visual field image during work experience by an unskilled worker. FIG. 10 is a diagram showing a state in which an unskilled worker is gazing intently at a nut that is the work target, in a visual field image during a work experience by the unskilled worker. FIG. 10 is a view showing a state in which an instruction is given to place a wrench on a nut to be worked on, from among the visual images during the work experience by the unskilled worker. FIG. 10 is a view showing a state in which a wrench is placed on a nut (lower side) that is the target of the work, from among the visual images during the work experience by an unskilled worker. FIG. 10 is a view showing a state in which an instruction is given to place a second wrench on the nut (upper side) that is the work target, in the field of view image during the work experience by the unskilled worker. FIG. 10 is a view showing a state in which the second wrench is placed on the nut (upper side) that is the work target, in the field of view image during the work experience by the unskilled worker. FIG. 10 is a view showing a state in which a wrench attached to an upper nut is rotated counterclockwise, from among the visual images during the work experience by an unskilled worker. FIG. 10 is a view showing a state in which the wrench is returned to the work tool palette, from among the field of view images during the work experience by the unskilled worker. FIG. 10 is a diagram showing a state in which an unskilled worker presses an experience end button in a field of view video during a work experience by the unskilled worker. FIG. 10 is a diagram illustrating an example of a radar chart showing the proficiency and work level of an unskilled worker based on skills and test results. FIG. 10 is a diagram showing an example of a DataSheet. FIG. 21 is an enlarged view of a portion of the DataSheet shown in FIG. 20. FIG. 22 is a diagram for explaining what a command based on a part of the DataSheet shown in FIG. 21 is like.

The following describes in detail embodiments of the occupational training support system and content creation program for the occupational training support system of the present invention, with reference to the drawings. Note that the embodiments described below are some of the preferred embodiments for implementing the present invention, and even if some of the configuration is changed, they can be considered part of the present invention as long as the effects are achieved.

[composition]
As shown in FIG. 1, the occupational training support system 10 according to this embodiment includes at least a measuring device 12, an external input device 14, a display device 16, an external output device 18, and a processing device.

The measuring device 12 is an element used to acquire data on the work skills of participants in the work training support system 10, such as skilled and unskilled workers. Specifically, it may be a camera or sensor (such as a general camera, a target sensor for capturing movements, or a sensor for detecting the position of the gaze) for acquiring image data of the participant's line of sight and body movements (movements), or a sensor (such as an acceleration sensor) for detecting hand and body movements during work as physical quantities.

The external input device 14 includes elements for detecting operations and sounds during work, as well as elements for creating and editing content, which will be described in detail below. Specific examples include a controller for operating the screen displayed on the display device 16 during work, a microphone for voice input, a PC or tablet device, mouse, and keyboard as data input means, as well as a distance sensor such as a Time of Flight (TOF) for detecting and inputting the amount of movement (for example, to detect the amount of movement of the target mentioned above).

The display device 16 is an element that shows the participant the field of view during work, as well as important work precautions and instruction information. Specific examples include a head-mounted display (HMD) worn by the participant, or a fixed or portable display (monitor).

The external output device 18 is an element that allows the participant to experience the physical vibrations and recoil that occur during work. Its configurations vary widely; a simple example would be a vibration that notifies the participant when the angle or time during work reaches a predetermined value. In the future, it may be possible to realize so-called full-body tracking by using sensors called trackers attached to the worker's limbs to physically transmit to the worker the vibrations, recoil, impact, resistance, etc. that occur during actual work. The external output device 18 also includes a speaker for outputting audio, etc. Furthermore, in comparison with the elements that make up the measuring device 12 and the external input device 14 described above, elements that simply have opposing functions (input and output) may be treated as a single component. For example, an HMD can be equipped with a variety of functions, such as a camera function, a sensor function for detecting line of sight, a field of view image display function, and audio input/output functions.

The processing device 20 is an element that creates content based on information such as task skill data acquired via the measuring device 12, and instruction data, voice data, and operation data input via the external input device 14, and outputs the content to the display device 16 and the external output device 18. Specifically, it may include an input unit 20a, an output unit 20b, a control device 20c, a computing device 20d, memory 20e, a voice processing device 20f, an image processing device 20g, and a large-capacity storage device 20h.

The input unit 20a and output unit 20b are elements that act as an interface, a conversion unit that exchanges data between the measuring device 12, external input device 14, display device 16, external output device 18, etc. In the case of the occupational training support system 10 according to the embodiment, the measuring device 12 and external input device 14 are connected to the input unit 20a, and the display device 16 and external output device 18 are connected to the output unit 20b.

The control device 20c is an element that transmits various data input via the input unit 20a to the calculation device 20d and image processing device 20g, and also transmits data input from the calculation device 20d to the image processing device 20g and output unit 20b.

The calculation device 20d is an element that transmits and receives various data between the control device 20c and memory 20e, and executes calculation processing in response to commands from various programs, etc.

Memory 20e is a temporary storage area for temporarily expanding various data when the calculation device 20d exchanges various information with the audio processing device 20f, the mass storage device 20h, the input unit 20a, and the output unit 20b.

The audio processing device 20f is an element for editing audio data temporarily stored in memory 20e.

The image processing device 20g is an element for editing image data recorded in a large-capacity storage device and image data transmitted from the control device 20c.

The large-capacity storage device 20h is a long-term storage area that stores various data input via the input unit 20a, and enables data to be read in response to requests from the memory 20e, image processing device 20g, output unit 20b, etc. Specific configurations include HDDs and SSDs.

[Action: Work education support]
Next, a work experience using the work training support system 10 according to the embodiment will be described with reference to Figures 2 to 4. First, when using (experienced) the work training support system 10 according to the embodiment, participants such as skilled workers and unskilled workers log in via the external input device 14 or the like (S100: Experiencer Login).

Next, each participant puts on VR equipment such as an HMD and sensors. Here, the VR equipment may be a complex sensory device equipped with various data input/output functions, such as the measuring device 12, external input device 14, display device 16, and external output device 18 (S110: Put on VR equipment). After putting on the VR equipment, the participant selects an experience category. Here, the experience category can be selected as "experienced worker" or "unskilled worker," but the "unskilled worker" category cannot be used for the work experience unless at least one training data (work experience content) from an experienced worker has been completed. In other words, before work experience content based on the work experience of an experienced worker is created, "unskilled worker (= beginner)" cannot be selected when selecting the experience category (S120).

When "Experienced Worker" is selected in S120, the user is prompted to select an item for the work experience (for creating work experience content) (S130: Select experience item). When an experience item is selected in S130, a function is activated to acquire data on the skilled worker's line of sight, posture, and movements while working. Specifically, the control device 20c and calculation device 20d in the processing device 20 are activated, and a voice recognition function for acquiring audio and video via the measuring device 12 and external input device 14, and a recording function are activated (S140: Activate content creation function). After the content creation function is activated, the skilled worker performs the work, completing the acquisition of data during the work (S150: Experienced worker's work experience).

During the work experience of an expert, work equipment is called up, as shown in detail in Figure 3. Work equipment can be called up by voice input or by selection from items displayed on the display device 16, such as an HMD (S150a: Calling of work equipment by experienced worker). When a work equipment is called up by the experienced worker, the processing device 20 expands the work equipment database stored in the mass storage device 20h into memory 20e via the control device 20c and calculation device 20d. The processing device then identifies the work equipment corresponding to the call from the work equipment database, and displays the object of the corresponding work equipment in the field of view image displayed on the display device 16, such as an HMD (S150b: Reading of equipment object).

After the work equipment corresponding to the call in the field of view image is displayed, the skilled worker calls up the work tool. As with calling up work equipment, the work tool can be called up by voice input or by selection from items displayed on the display device 16 such as an HMD (S150c: Calling of work tool by skilled worker). When the work tool is called up by the skilled worker, the processing device 20, as with calling up work equipment, identifies the work tool corresponding to the call and displays the object of the corresponding work tool in the field of view image displayed on the display device 16 such as an HMD (S150d: Reading of tool object).

Once the work equipment and tools are called, the skilled worker performs the simulated work. Instructions are given to perform the simulated work, with the details of the work explained step by step (S150e: Perform simulated work (experience the work in order while talking about it)). The simulated work progresses with repeated confirmation of comments, objects, and actions for each task. Specifically, the skilled worker first explains the work actions to be performed. The explanation of the work actions is given orally, and the processing device 20 collects this explanation as audio comments using the microphone of the external input device 14 and automatically registers them in the mass storage device 20h via memory 20e (S150f: Automatic registration of action comments (Phase 01)). When comments regarding the work actions are registered, the processing device 20 automatically recognizes objects related to the work (e.g., wrenches and nuts) and registers them as objects used in the work. Object recognition may be achieved by detecting objects displayed on the display device 16, or by recognizing the work equipment and work tools called up in S150a or S150c. (S150g: Automatic registration of object). Then, information that changes as the skilled worker performs the task is acquired and registered. Specifically, the skilled worker's motion information is detected via the measurement device 12 and external input device 14, and is calculated as the object's movement amount, rotation angle, number of motions, etc. via the control device 20c and calculation device 20d, and recorded in the large-capacity storage device 20h (S150h: Automatic registration of post-motion coordinates, angle, number of motions, etc.).

The operations and registration from S150e to S150h are repeated for each work operation. After that, when all operations corresponding to the work experience have been completed, the simulated work is declared to be stopped (ended) or selected, and the work experience by the skilled worker ends (S150i: Stop).

Here, the work training support system 10 according to this embodiment defines contact points and applies a suction action between objects when contact points come close to each other, making it possible to achieve contact between tiny parts and the gripping of tiny parts in VR, something that was previously difficult.

When the end of the simulated work is declared, the various pieces of information registered in S150f through S150h are associated and recorded as a single module (S150j: Save as Module). The recorded information on the skilled worker's work experience, stored as a module, is then converted into a DataSheet corresponding to a program for creating content for unskilled workers and output (S150k: Output as DataSheet). After S150k, the skilled worker is prompted to select either "Experience Complete" to end the experience or "Experience Another Item" to try another work item. A DataSheet is a table containing commands for a specific program, such as those shown in Figure 20. Figure 21 shows an excerpt from a DataSheet. In Figure 21, column A indicates the command name, column B indicates the control number, column C indicates the stage ID, column D indicates the flag number, column E indicates the definition name A, column F indicates the definition name B, column G indicates the X coordinate, column H indicates the Y coordinate, and column I indicates the Z coordinate. Of the command lines, line a instructs the reading of a 3D model of the nut, and line b instructs the reading of a 3D model of the wrench (see Figure 22 (A)). Line c is a command instructing the user to move the wrench closer to the nut (see Figure 22 (B)), and line d is a command to set the wrench on the nut and rotate the X axis around the Y axis by 12 degrees with the center of the nut as the center of rotation (see Figure 22 (C)) (S160).

Afterwards, if necessary, the skilled workers who have completed the work experience can be given a survey about their work experience. The survey content can be, for example, whether there are any points to be careful about at each work site, or whether there are any points to be careful about in the actual work that cannot be expressed in VR. This is for use in improving the work training support system 10 and explaining points to be careful about to content editors, details of which will be described later (S170: Conduct Survey).

The work experience by the skilled worker ends with the completion of the questionnaire. Once the base of the work experience content (DataSheet S150k) is saved and output during the work experience by the skilled worker, content editing by a content editor becomes possible. Content editing for creating content for the work experience by an unskilled worker will be explained below with reference to Figure 4.

[Editing work experience content]
The content editor accesses the work training support system 10 via the external input device 14 or the like and logs in. Here, the content editor does not need to be a person with advanced knowledge of general system program editing. Specifically, it is sufficient for the content editor to be a person (mid-level to experienced worker) who has learned the content editing flow in the work training support system 10 according to the embodiment, general editing operations, and basic work knowledge in the field (S400: Content editor logs in).

A content editor who accesses the work training support system 10 launches the content editing program. When the content editing program is first launched, an option such as "Load DataSheet" is displayed on the content editing screen. After selecting the "Load DataSheet" option, the content editor selects the content to be edited from among the DataSheets (hereinafter also referred to as "base DataSheets") of work content created by skilled workers through work experience. The DataSheet recorded in the mass storage device 20h is read via the control device 20c or the calculation device 20d and displayed on the display device 16 (S410: Selection of "Load DataSheet").

Next, the "Play Video" item is displayed on the content editing screen, and the content editor can select this to play the work video recorded in the DataSheet loaded in S410. The video is played by outputting the video data recorded in the mass storage device 20h via the image processing device 20g (S420: Select "Play Video"). If there is a section of the video being played that requires editing, select "Stop" to temporarily halt the video playback (S430: Select "Stop").

After stopping the video, select "Edit" or "Edit." "Edit" is the option for editing previously edited comments, etc. Therefore, the first task is to select "Edit" (S440).

Editing involves adding comments from experienced workers to the video, supplementary images, videos, and text, and setting the coordinates of objects, etc. Comments are selected by calling up a list of comments automatically registered by the experienced worker during the work experience (Phase 01 comments in S150f) and selecting from this list (S450: Selecting Phase 01 Comments). Supplementary images, videos, and text can be added by adding images and videos of the actual work site. Regarding text, it can be points of caution raised in a questionnaire administered to the experienced worker during the work experience. Supplementary images and videos can be selected by recording pre-taken images and videos of the site in a large-capacity storage device, reading them via the control device 20c or the computing device 20d, and displaying them on the display device 16. Regarding text, it can be questionnaire responses, etc., recorded as text information (S460: Adding Images, Videos, and Text). Furthermore, setting the coordinates of objects, etc. is the act of determining the position on the display device 16 of the comments, images, videos, text, etc. selected and added in S450 and S460. Coordinates can be set by inputting the coordinates of the object's placement position, or by positioning the object by dragging and dropping it on the display screen (S470: Setting coordinates in Phase 03).

After setting the object coordinates, you can choose whether or not to add other comments, etc. The options available are, for example, "End Editing" and "Continue Editing." If you select "Continue Editing," steps S450 to S470 are repeated (S480).

If "End Editing" is selected in S480, a choice (Yes or No) as to whether or not to play the video again is made (S490). If "No" is selected here to not play the video, "Stop" is selected in S430. If there are any corrections to be made to the comments, etc. selected and added in S450 to S470, "Edit" is selected in S440 after video playback has stopped. Following the steps S450 to S470, the correction items include selecting a comment "Comment for Phase 01," correcting the comment, and selecting coordinates (coordinates for Phase 03) (S440a to S440c). When S440c is completed, the process returns to S450, where images, videos, and text are added (S460), and the coordinates of the added images, etc. are set (S470).
When "Yes" is selected regarding video playback in S490 and editing of the entire video is completed, a DataSheet (hereinafter also referred to as a content DataSheet) including comments, additional images, etc. is output (S500: Output of DataSheet).

[Working model creation]
When the content editor finishes editing the work experience content, a work model of an experienced worker is created. Here, the work model numerically represents the characteristics of the worker's line of sight and movements, and is an element that facilitates comparison with the movements of an unskilled worker during the work experience.

The first step in creating a task model is to capture the motion and gaze data of the task participant. The motion and gaze data of the task participant is data associated with the data recorded in the content DataSheet, and is stored in the large-capacity storage device 20h of the processing device 20, which can then be read into memory 20e (S510: Capture task participant motion and gaze data).

Next, area division processing of movements and gaze is performed according to predetermined conditions. Area division processing is a preprocessing step for, for example, when it is desired to acquire gaze information, in which objects displayed within the field of view (work equipment, work tools, workpieces, hoisting equipment, etc.) and other workers are divided into separate areas, and the location of the gaze within each divided area is detected (S520: Area division processing of movements and gaze).

After the area division process is completed, the number, duration, and position of gazes (if gaze information is acquired) directed within each area, as well as gaze movements (sequence of gaze movement), are counted and quantified for each area. Figure 5 is a diagram showing the gaze shifts, gaze duration, and number of gazes of an experienced worker during rigging operations, broken down by area, with the size of the circle indicating the length of gaze time (the larger the circle, the longer the gaze time) (S530: Quantification process within each area). After the quantification process within each area is completed, feature values for each region are extracted. Here, feature values refer to the distance and speed of gaze movement, dwell time, and number of dwells for each area (S540: Extraction of feature values for each area).

After extracting the features, an approximate model showing the numerical trend is created, as shown in Figure 7. In the example of the approximate linear model shown in Figure 7, the horizontal axis shows the items in the segmented area, and the vertical axis shows the gaze time for each item (S550: Create approximate model). After the approximate model is created, if all editing work related to content creation is complete, select "Editing complete" to end the editing work.

[Work experience for unskilled workers]
Once the work experience content for an unskilled worker has been constructed through the above-described process of acquiring data based on the work experience of an experienced worker, editing the content based on the acquired data, and creating a work model, the work experience content for an unskilled worker can be implemented. The work experience by an unskilled worker will be described below.

First, the inexperienced worker logs in to the system as an experiencer (S100: Experiencer Login) and puts on the VR equipment (S110: Put on VR Equipment), just like an experienced worker would.

After equipping various equipment, the experience level is selected. As mentioned above, the experience level is divided into "experienced worker" and "unskilled worker," and in this case, "unskilled worker (= beginner)" is selected (S120).

When the "Unskilled Worker" item is selected in the experience category, the work experience content to be experienced is explained on the HMD display screen or through audio guidance. Here, the explanation of the work experience content may be given while displaying a character unique to the system. This is because a sense of familiarity and empathy can help improve the level of work mastery (S200: Explanation of experience content).

After the explanation of the work experience content, an explanation of how to operate the system via the VR equipment will be given. As with the explanation of the work experience content, this explanation may simply be given through visual or audio means, or may be given while a character is displayed (S210: Explanation of system operation).

After the explanation of how to operate the system is completed, the initial movements of the inexperienced worker are detected. Differences in the body shape of the person experiencing the work (inexperienced worker) can result in a large error in the optimal conditions for the work when compared with the training data (work model obtained through the work experience of an experienced worker). Therefore, by detecting the initial movements, errors with the optimal conditions that arise due to differences in body shape can be corrected, allowing for accurate comparison with the training data (S220: Detection of initial movements of the person experiencing the work).

After errors associated with the detection of initial movements have been corrected, if the inexperienced worker is ready, they can select "Start" to begin the work experience. On the other hand, if they are unsure about operating the system or would like to hear another explanation, they can select "Listen to operation explanation again" to return to S210 and listen to an explanation of how to operate the system (S230).

When the work experience begins, the basic knowledge related to the work to be experienced is first explained. The explanation is given visually and/or audibly via the HMD (S240: Basic Knowledge Explanation). Next, the basic procedures related to the work to be experienced are explained. As with the explanation of basic knowledge, the explanation of the work procedures is also given visually and/or audibly via the HMD (S250: Basic Work Procedure Explanation).

Once the basic knowledge and explanation of the work procedures have been completed, the work procedures are guided and explained via the HMD. Here, the guidance and explanation of the work procedures are given for each action involved in the work (S260: Guidance and explanation for each work procedure).

After receiving guidance and explanations about the work procedures, the inexperienced worker performs the actions (work experience) in accordance with the instructions. At this time, the measurement device detects the inexperienced worker's movements, posture, and eye movements. Based on the various detected data, the processing device 20 creates a work model, similar to the work experience performed by an experienced worker. For example, Figure 6 shows the gaze shifts, gaze duration, and frequency of an inexperienced worker during rigging work, broken down by area. Note that the size of the circle in Figure 6 indicates the length of gaze duration (the larger the circle, the longer the gaze duration). The processing device 20 also extracts feature values from the detected data and creates a work model such as that shown in Figure 7.

An example of a work experience (basic work) by an inexperienced worker will now be described with reference to Figures 8 to 18. In the following example, the basic work that the inexperienced worker experiences is loosening the upper nut of a double nut part. First, a visual field image including the work site and tools (two wrenches), as shown in Figure 8, is displayed on a display device 16 such as an HMD worn by the inexperienced worker. In this state, the inexperienced worker is given instructions, such as "Select and lift the H20 wrench," via text display or audio guidance.

Upon receiving the instruction, the unskilled worker reaches out for the wrench in the visual field image and lifts it. At this time, the program of the work training support system 10 according to the embodiment is configured so that when the unskilled worker's hand approaches the wrench, the wrench is automatically placed in the worker's hand (see Figure 9).

Next, the unskilled worker is shown the nut to be worked on (see Figure 10), and is instructed via text, voice, or other means to "visually check the nut for at least three seconds." The unskilled worker follows the instruction and visually inspects the nut to be worked on (see Figure 11). After the gaze period has elapsed, the unskilled worker is instructed to "attach the H20 wrench to the nut (bottom)" in the image displayed in Figure 12. The unskilled worker follows the instruction and brings the H20 wrench close to the nut to be worked on. In the program of the work training support system 10 according to the embodiment, when the distance between the objects (e.g., the wrench and the nut) is within a predetermined proximity distance, the display automatically unfolds to ensure correct assembly (see Figure 13).

The unskilled worker is then instructed to attach the lifting wrench to the nut (upper side) for the other wrench (the H26 wrench), and follows the instructions to attach the H26 wrench to the upper nut (see Figures 14 and 15). The unskilled worker is then instructed to "hold the H20 wrench in place and turn the H26 wrench counterclockwise." The unskilled worker follows the instructions and rotates only the H26 wrench counterclockwise a specified angle (e.g., 30 degrees) (see Figure 16).

In the program of the work training support system 10 according to the embodiment, the display is expanded so that the connection between the work tool and the work object is automatically released when the work is completed (for example, when the wrench has rotated 30 degrees). The unskilled worker is then instructed to "return the wrench to the work tool palette," and the unskilled worker performs the movement of returning the wrench to the work tool palette in accordance with the instruction (see FIG. 17). After the series of work experiences are completed, the unskilled worker ends the work experience by pressing the experience end button (see FIG. 18) displayed on the display screen (S270: Basic work experience for each work procedure).

Once the basic work experience is complete, a selection screen appears, allowing the inexperienced worker to choose whether to "end" or "retry" the work experience. If the inexperienced worker selects "retry," the process returns to S260, where guidance and explanations for each work procedure begin again. On the other hand, if "end" is selected, the work experience involving movements ends (S280).

When the inexperienced worker completes the work experience, a proficiency assessment is performed based on the inexperienced worker's movements. The proficiency assessment is performed by comparing the work model of the skilled worker included in the work experience content with the work model of the inexperienced worker (see Figure 7 for an example).

The task model is compared with an approximation model (a numerical line in this embodiment), and a level is assigned depending on the degree of angular deviation for each parameter. For example, if the deviation from the training data approximation model of an experienced worker is 5 degrees or less (θ≦5 degrees), the task level is determined as "Level 1." If the deviation is greater than 5 degrees but less than 15 degrees (5 degrees < θ≦15 degrees), the task level is determined as "Level 2." If the deviation is greater than 15 degrees but less than 30 degrees, the task level is determined as "Level 3." If the deviation is greater than 30 degrees, i.e., if it does not meet Level 3, a determination result is output encouraging re-learning (S290: Determining the Experienced Worker's Proficiency Based on Actions).

After the proficiency assessment based on movements is completed, the inexperienced worker undergoes a test based on their knowledge of on-site work and points to note regarding the work they have experienced. The inexperienced worker's work level is certified based on the test results (S300: Test taking/work level certification).

Once the inexperienced worker's proficiency level has been assessed and their work level has been certified, the work training support system 10 displays the worker's work level resulting from the work experience in a visually recognizable form. Here, a visually recognizable form includes, for example, a graph like the one shown in FIG. 19, as well as simple numerical or score displays, and displays using different colors (S310: Visualization of the experience worker's work level).

[effect]
The above-described work training support system makes it possible to create work experience content corresponding to various types of work. Furthermore, editing work experience content (VR content) in the program related to this system does not require extensive programming knowledge, so even non-experts can edit the content, making it relatively easy to create various types of work experience content.

10... Occupational training support system, 12... Measuring device, 14... External input device, 16... Display device, 18... External output device, 20... Processing device, 20a... Input section, 20b... Output section, 20c... Control device, 20d... Arithmetic device, 20e... Memory, 20f... Audio processing device, 20g... Image processing device, 20h... Large-capacity storage device.

Claims (5)

An educational support system that allows unskilled workers engaged in on-site work to learn the skills of skilled workers,
a measuring device for detecting physical quantity data of the movements and postures of the virtual experiencer performing a specific task, and data of the movement of the line of sight;
a display device that displays a field of view image in a virtual space;
a processing device that outputs the visual field image to the display device, and records the physical quantity data and gaze movement data as training data if the virtual experiencer is an experienced worker, and compares the physical quantity data and gaze movement data with the training data if the virtual experiencer is an inexperienced worker, calculates a difference, and calculates a proficiency level corresponding to the difference based on a predetermined reference value;
an external input device for creating and editing content;
The processing device is provided with a large-capacity storage device that stores work equipment and work tools to be displayed in the virtual experience,
The work equipment and the work tools are displayed in the virtual space when called by the skilled worker during the virtual experience,
A work training support system characterized in that the teacher data is created by editing and creating a work model by a content editor who accesses the training support system via the external input device . The occupational training support system described in claim 1, characterized in that the measurement device includes a target or camera that detects the virtual experiencer's movements, and a sensor that detects the virtual experiencer's line of sight and physical quantities based on the movements. The occupational training support system described in claim 1 or 2, characterized in that the display device includes an HMD (Head Mounted Display) worn by the virtual participant, and various operational instructions are displayed to the virtual participant via the HMD. The occupational training support system described in any one of claims 1 to 3, characterized in that, when the proximity distance between objects in the display of the field of view image falls below a predetermined value, the object is displayed in a specified appropriate joint state. a display device that displays a field of view image in a virtual space; a processing device that outputs the field of view image to the display device, and records the physical quantity data and gaze movement data as training data if the virtual user is a skilled worker, and compares the physical quantity data and gaze movement data with the training data if the virtual user is an unskilled worker, calculates a difference, and determines a proficiency level corresponding to the difference based on a predetermined reference value ; and an external input device, said content creation program being applicable to a work training support system that enables unskilled workers engaged in on-site work to learn the skills of skilled workers, said content creation program comprising: a measuring device that detects physical quantity data and gaze movement data of a virtual user performing a specific task;
The processing device stores the work equipment and work tools to be displayed in the virtual experience, and a content editing program that is activated by accessing the external input device,
When the skilled worker calls for a virtual experience, the work equipment and the work tools are displayed in the virtual space, and
creating a video of work experience content by selecting and adding additional information about the unskilled worker recorded in the processing device to a video taken during the virtual experience by the skilled worker via the content editing program ;
A content creation program for an occupational education support system, characterized in that it is capable of executing a process of converting an image of the occupational experience content into a DataSheet according to the program and outputting it.