JP7619158B2 - MANUAL ANALYSIS DEVICE, MANUAL PROVIDING DEVICE, MANUAL READING SYSTEM, METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a manual analysis device, a manual provision device, a manual reading system, a method, and a program.

Conventionally, various manuals (also called instruction manuals or user guides) have been used in various situations (Patent Document 1).

For example, one of the maintenance tasks of customer engineers in the manufacturing industry is to repair products at customer sites. One particularly common task is replacing product parts. If a customer engineer does not know the replacement procedure, they will replace the part in question while referring to the manual. This type of work is often done in a small space and the customer engineer's hands often get dirty. For this reason, there is a need for voice confirmation that allows customer engineers to refer to the manual without having to change their posture significantly or touch it with dirty hands.

However, in addition to the text explaining the procedure, the manual contains images with superimposed marks indicating the relative positions of the target part and surrounding parts, and the direction in which the target part should be moved or removed. Also, the surrounding parts shown in the images may not be mentioned in the text explaining the procedure. As a result, simply reading out the text of the manual lacks information regarding the relative positions of the target part and surrounding parts, and information regarding the direction in which the target part should be moved or removed, and it is not possible to provide sufficient information to carry out the work.

Therefore, one embodiment of the present invention aims to make it possible to understand a manual even without an image of the manual.

An apparatus according to one embodiment of the present invention includes a manual analysis unit that acquires training data, performs machine learning using the training data, and generates a machine learning model that outputs work procedure information when image data in a manual and text data in the manual are input as input data and text data generated based on the image data is added to the text data in the manual to obtain output data, and outputs work procedure information.

In one embodiment of the present invention, a manual can be understood even without an image of the manual.

FIG. 1 is a diagram for explaining an overview of the present invention. 1 is an overall configuration diagram according to an embodiment of the present invention; 1 is a hardware configuration diagram of a manual analysis device, a manual providing device, and a user terminal according to an embodiment of the present invention. 1 is a functional block diagram of a manual analysis device according to an embodiment of the present invention. 1 is a functional block diagram of a manual providing device according to an embodiment of the present invention; 1 is a flowchart of a manual analysis process according to an embodiment of the present invention. 11 is a flowchart of a manual providing process according to an embodiment of the present invention.

Each embodiment will be described below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

<Terminology>
In this specification, a "manual (also called an instruction manual or a user guide)" refers to an electronic manual that includes text data and image data. For example, the manual is a manual for repairing precision equipment such as a multifunction printer, but is not limited to this and may be any manual. For example, a customer engineer or the like can replace parts of a precision equipment such as a multifunction printer in order to repair the precision equipment such as a multifunction printer based on the manual.

<Summary>
1 is a diagram for explaining an overview of the present invention. In the present invention, it is possible to present work procedure information that is supplemented by adding text data based on image data in a manual to the text data in the manual.

For example, suppose the manual describes the procedure as shown in Figure 1. In other words, the manual includes text data such as "1. Open the front cover [A]. 2. While pressing down the stopper, slide the front cover [A] to the right to remove it.", and image data. The image data has superimposed thereon a mark indicating "[A]", a mark indicating the "main body", a mark indicating the "stopper", a mark indicating the direction in which to move the front cover, a mark indicating the direction in which to remove the front cover, and a mark indicating the direction in which to move the stopper.

In the manual, the target part (here, A) is indicated on the image data, and is described in the text data as "Front Cover [A]". Note that there may be cases where there is no explanation in the text data (i.e., only instructions by marks on the image data). The marks on the image data have multiple meanings and contain a lot of information, such as the direction in which the part should be moved, the direction in which the part should be removed, instructions for the target part described in the text data, and the order of each procedure. In the present invention, machine learning is performed using training data in which image data containing such various information (image data in the manual) and text data in the manual are input data, and text data that completely expresses the contents of the work (hereinafter also referred to as work procedure information) is output data. Then, the work procedure information can be presented using a machine learning model generated by machine learning.

Below is an example of work instruction information.

Before (i.e., text data in the manual)
1. Open the front cover [A].
2. While pressing down on the stopper, slide the front cover [A] to the right to remove it.

After (i.e., work procedure information supplemented by adding text data based on images in the manual to the text data in the manual)
1. Open the front cover of the device towards you.
2. While pressing down on the stopper located slightly left of the center where the main body and front cover join, slide the front cover to the right to remove it.

As described above, it can be seen that the work procedure information is presented in a state where the positions of the target parts and surrounding parts, the direction of operation, etc. are supplemented to the before text data. Note that the word [A] used when checking simultaneously with the image as in this example may be deleted from the work procedure information, or may be replaced with the wording indicated by [A].

<Overall configuration>
Fig. 2 is an overall configuration diagram according to an embodiment of the present invention. As shown in Fig. 2, the manual reading support system 1 can include a manual analysis device 10, a manual provision device 20, and a user terminal (customer engineer terminal) 30. The user terminal 30 can transmit and receive data to and from the manual provision device 20 via any network. Note that in this specification, the manual analysis device 10 and the manual provision device 20 are described as two devices, but the manual analysis device 10 and the manual provision device 20 may be implemented in one device.

The manual analysis device 10 analyzes a computerized manual that includes text data and image data so that the manual can be understood from the text data alone. Specifically, the manual analysis device 10 acquires training data in which the image data and text data in the manual are input data, and work procedure information supplemented by adding text data generated based on the image data in the manual to the text data in the manual is output data. The manual analysis device 10 also performs machine learning using the training data to generate a machine learning model that outputs work procedure information when the image data and text data in the manual are input.

In response to a request from a user terminal 30, the manual providing device 20 presents work procedure information that has been supplemented so that an electronic manual that includes text data and image data can be understood from the text data alone.

The manual providing device 20 may present pre-generated work procedure information (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

When presenting pre-generated work procedure information, the manual providing device 20 acquires information (e.g., image data) for identifying an object (e.g., a part of a precision device such as a multifunction printer) from the user terminal 30. The manual providing device 20 also identifies at least one of image data and text data in the manual that corresponds to the information for identifying the object. The manual providing device 20 also identifies pre-generated work procedure information linked to at least one of image data and text data in the manual. The manual providing device 20 also outputs the work procedure information to the user terminal 30.

When generating and presenting work procedure information in response to a request from the user terminal 30, the manual providing device 20 acquires information (e.g., image data) for identifying an object (e.g., a part of a precision device such as a multifunction printer) from the user terminal 30. The manual providing device 20 also identifies image data and text data in the manual that correspond to the information for identifying the object. The manual providing device 20 also inputs the image data and text data in the manual into a machine learning model to output work procedure information. The manual providing device 20 also outputs the work procedure information to the user terminal 30.

The user terminal (customer engineer terminal) 30 is a terminal used by a person who uses a manual according to one embodiment of the present invention (e.g., a customer engineer who repairs precision equipment such as a multifunction printer). Specifically, the user terminal 30 transmits information (e.g., image data) for identifying an object to the manual providing device 20. The user terminal 30 also receives work procedure information for the object from the manual providing device 20. For example, the user terminal 30 is a smart device, smart glasses, etc.

The user terminal (customer engineer terminal) 30 may use the work procedure information stored in the user terminal (customer engineer terminal) 30.

<Hardware Configuration>
FIG. 3 is a diagram showing the hardware configuration of the manual analysis device 10, the manual providing device 20, and the user terminal 30 according to an embodiment of the present invention.

As shown in FIG. 3, the manual analysis device 10, manual provision device 20, and user terminal 30 are constructed by computers, and as shown in FIG. 3, are equipped with a CPU 1001, a ROM 1002, a RAM 1003, a HD 1004, a HDD (Hard Disk Drive) controller 1005, a display 1006, an external device connection I/F (Interface) 1007, a network I/F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I/F 1015.

Of these, the CPU 1001 controls the overall operation of the manual analysis device 10, the manual provision device 20, and the user terminal 30. The ROM 1002 stores programs used to drive the CPU 1001, such as IPL. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data such as programs. The HDD controller 1005 controls the reading or writing of various data from the HD 1004 according to the control of the CPU 1001. The display 1006 displays various information such as a cursor, menu, window, character, or image. The external device connection I/F 1007 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 1001 shown in FIG. 3.

The keyboard 1010 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls the reading and writing of various data from a DVD-RW 1012, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I/F 1015 controls the reading and writing (storing) of data from a recording medium 1014, such as a flash memory.

<Function block>
Hereinafter, the functional blocks of the manual analysis device 10 will be described with reference to FIG. 4, and the functional blocks of the manual provision device 20 will be described with reference to FIG.

Figure 4 is a functional block diagram of a manual analysis device 10 according to one embodiment of the present invention. As shown in Figure 4, the manual analysis device 10 includes an in-manual image data storage unit 101, an in-manual text data storage unit 102, and a manual analysis unit 103. The manual analysis device 10 also functions as the manual analysis unit 103 by executing a program. Each of these is described below.

The manual image data storage unit 101 stores image data included in the manual. For example, the image data is electronic data such as photographs and illustrations showing the procedure of the work. The image data may be superimposed with marks indicating the target part (the part that is the subject of the work, e.g., the part to be replaced by a customer engineer), marks indicating peripheral parts, marks indicating the direction in which the part should be moved, marks indicating the direction in which the part should be removed, etc.

The manual text data storage unit 102 stores the text data contained in the manual.

The manual analysis unit 103 executes a manual analysis process. Specifically, the manual analysis unit 103 acquires image data stored in the in-manual image data storage unit 101 and text data stored in the in-manual text data storage unit 102. The manual analysis unit 103 also acquires complemented text data (text data complemented by text data generated based on image data in the manual). The manual analysis unit 103 also associates "image data in the manual and text data in the manual" with "complemented text." The manual analysis process will be described in detail with reference to FIG. 6.

Figure 5 is a functional block diagram of a manual providing device according to one embodiment of the present invention. As shown in Figure 5, the manual providing device 20 includes an object data input unit 201, a work procedure information selection unit 202, a work procedure information output unit 203, an in-manual image data storage unit 101, and an in-manual text data storage unit 102. Furthermore, the manual providing device 20 functions as the object data input unit 201, the work procedure information selection unit 202, and the work procedure information output unit 203 by executing a program. Each of these is described below.

The manual providing device 20 may use the in-manual image data storage unit 101 and the in-manual text data storage unit 102 of the manual analysis device 10.

The object data input unit 201 acquires information (e.g., image data, audio, text, etc.) for identifying an object (e.g., a part of a precision device such as a multifunction printer) from the user terminal 30.

The work procedure information selection unit 202 may present work procedure information that has been generated in advance (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

When presenting pre-generated work procedure information, the work procedure information selection unit 202 identifies at least one of image data and text data in the manual that corresponds to the information for identifying the object acquired by the object data input unit 201. In addition, the work procedure information selection unit 202 identifies pre-generated work procedure information linked to at least one of image data and text data in the manual.

When generating and presenting work procedure information in response to a request from the user terminal 30, the work procedure information selection unit 202 identifies image data and text data in the manual that correspond to information for identifying the object acquired by the object data input unit 201. In addition, the work procedure information selection unit 202 inputs the image data and text data in the manual into a machine learning model to output work procedure information.

The work procedure information output unit 203 outputs the work procedure information to the user terminal 30. For example, the work procedure information output unit 203 transmits audio data of the work procedure information to the user terminal 30.

<Processing method>
The manual analysis process will be described below with reference to FIG. 6, and the manual provision process will be described with reference to FIG.

Figure 6 is a flowchart of the manual analysis process according to one embodiment of the present invention.

In step 11 (S11), the manual analysis unit 103 acquires image data stored in the in-manual image data storage unit 101 and text data stored in the in-manual text data storage unit 102. This image data and text data become training data.

In step 12 (S12), the manual analysis unit 103 acquires the complemented text data (text data complemented by text data generated based on image data in the manual). This complemented text data becomes training data. For example, a system administrator or the like creates the complemented text data.

In step 13 (S13), the manual analysis unit 103 associates the "image data in the manual and the text data in the manual" with the "complemented text."

Specifically, the manual analysis unit 103 acquires training data in which the image data and text data in the manual acquired in S11 are input data, and the complemented text data acquired in S12 (i.e., work procedure information complemented by adding text data generated based on image data to the text data in the manual) is output data, and performs machine learning using the training data. Then, the manual analysis unit 103 generates a machine learning model that outputs work procedure information when the image data and text data in the manual are input. For example, the manual analysis unit 103 can use VQA (visual question answering), which is an example of a multimodal (image and text) method, as a machine learning method.

The following is an example of text data generated based on image data in a manual. For example, the text data generated based on image data in a manual relates to marks (symbolic information) included in the image data in the manual, such as the positional relationship between the target part (the part that is the target of the work, e.g., the part that the customer engineer replaces) and surrounding parts, the direction in which the target part is moved, and the direction in which the target part is removed.
- From the customer engineer's perspective, what aspect does the target part fit?
On which side of the surface is it?
-What color?
・Where do you start moving from (what do you do)?
In which direction should it move?
Other - What kind of shape is it?
-How big is it?
-How many are there?

Figure 7 is a flowchart of the manual provision process according to one embodiment of the present invention.

In step 21 (S21), the object data input unit 201 acquires information (e.g., image data) for identifying the object. Specifically, the object data input unit 201 acquires information (e.g., image data) for identifying the object from the user terminal 30. For example, the object is an object on which a customer engineer works (e.g., a part that the customer engineer replaces). Smart glasses worn by the customer engineer or the like may capture an image and transmit it to the manual providing device 20, or a smart device or the like may capture an image and transmit it to the manual providing device 20 via an interface such as a chatbot or an input box in a web browser.

As described above, the work procedure information selection unit 202 may present work procedure information that has been generated in advance (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

When presenting work procedure information that has been generated in advance, in step 22 (S22), the work procedure information selection unit 202 acquires at least one of image data and text data in the manual. Specifically, the work procedure information selection unit 202 identifies at least one of image data and text data in the manual that corresponds to the information for identifying the object in S201. For example, the work procedure information selection unit 202 identifies the object included in the image data in S201, and acquires at least one of image data and text data in the manual that describes the object.

When generating and presenting work procedure information in response to a request from the user terminal 30, in step 22 (S22), the work procedure information selection unit 202 acquires image data and text data in the manual. Specifically, the work procedure information selection unit 202 identifies image data and text data in the manual that correspond to the information for identifying the object in S201. For example, the work procedure information selection unit 202 identifies the object included in the image data in S201, and acquires image data and text data in the manual that describe the object.

In step 23 (S23), the work procedure information selection unit 202 completes the text data.

When presenting pre-generated work procedure information, in S23, the work procedure information selection unit 202 identifies pre-generated work procedure information linked to at least one of the image data in the manual identified in S22 and the text data in the manual.

When generating and presenting work procedure information in response to a request from the user terminal 30, in S23, the work procedure information selection unit 202 inputs the image data and text data in the manual identified in S22 into a machine learning model (the machine learning model generated by the manual analysis process in Figure 6) and outputs the work procedure information.

In one embodiment of the present invention, the work procedure information (i.e., the text data after completion) contains the part name obtained from the text data in the manual (i.e., the text data before completion), so by comparing "that part name" with "the part name obtained from the information for identifying the object in S21 (by captioning, voice recognition, etc.)", it is possible to select the work procedure information without going through the text data in the manual.

In step 24 (S24), the work procedure information output unit 203 outputs the work procedure information. Specifically, the work procedure information output unit 203 outputs the work procedure information to the user terminal 30. For example, the work procedure information output unit 203 outputs the work procedure information by voice. Note that the work procedure information output unit 203 may display the work procedure information on a smart device, display the work procedure information in AR (Augmented Reality) on smart glasses, etc.

＜Effects＞
Thus, in one embodiment of the present invention, a customer engineer or the like can hear audio of work procedure information in which text based on images in the manual is added to the text in the manual. Therefore, the customer engineer or the like can understand the manual only from the text information even without images. As a result, the customer engineer or the like can work while understanding the repair procedures for precision equipment such as a multifunction printer without changing his/her position to look at the manual or holding the manual with dirty hands.

Each function of the above-described embodiments can be realized by one or more processing circuits. In this specification, the term "processing circuit" includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), and conventional circuit modules designed to execute each function described above.

Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

1 Manual reading support system 10 Manual analysis device 20 Manual provision device 30 User terminal (customer engineer terminal)
101: Manual image data storage unit 102: Manual text data storage unit 103: Manual analysis unit 201: Object data input unit 202: Work procedure information selection unit 203: Work procedure information output unit

JP 2002-269147 A

Claims (6)

acquiring training data, the training data including image data in a manual and text data in the manual being input data, and work procedure information supplemented by adding text data generated based on the image data to the text data in the manual being output data;
Perform machine learning using the training data;
An apparatus including a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are input. The device according to claim 1, wherein the text data generated based on the image data indicates the positional relationship between the target part and surrounding parts, the direction in which the target part is to be moved, and the direction in which the target part is to be removed. 3. The apparatus according to claim 1, wherein the manual is a manual for a multifunction peripheral. A manual reading support system including a manual analysis device, a manual provision device, and a user terminal,
The manual analysis device includes:
acquiring training data, the training data including image data in a manual and text data in the manual being input data, and work procedure information supplemented by adding text data generated based on the image data to the text data in the manual being output data;
Perform machine learning using the training data;
a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are input;
The manual providing device includes:
an object data input unit that acquires information for identifying an object from the user terminal;
a work procedure information selection unit that identifies image data and text data in a manual that correspond to information for identifying the object, and inputs the image data and the text data in the manual into the machine learning model to output work procedure information;
a work procedure information output unit that outputs the work procedure information to the user terminal. 1. A computer-implemented method comprising:
acquiring training data, the input data being image data in a manual and text data in the manual, and the output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual;
A step of performing machine learning using the training data;
generating a machine learning model that outputs work procedure information when image data and text data in the manual are input;
The method includes: a computer acquires training data, the training data being input data for image data in a manual and text data in the manual, and output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual;
Perform machine learning using the training data;
A program for functioning as a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are input.

Images