JP7795995B2 - Work support device, work support method, and work support system - Google Patents

Description

The present invention relates to a work support device, a work support method, and a work support system. In particular, the present invention relates to a work support device that can be suitably used to extract tables from image data representing drawings.

Labor shortages are becoming more serious due to population decline and an aging society with a declining birthrate. There is a shortage of highly skilled experts in manufacturing sites, and the situation where only certain people can perform certain tasks, known as personalization, is becoming a problem. When passing on advanced technology, the successor also needs to have a certain level of skill, but due to a shortage of mid-career engineers in this position, the current situation is that technology transfer is not progressing well in many companies. In order to promote the transfer of technology to younger skilled workers, it is necessary to create a system in which explicit knowledge such as work procedures and know-how can be compiled into manuals and work can be carried out regardless of skill level.
In response to this situation, attention is being focused on activities to digitize business operations using wearable devices such as electronic paper.

Patent Document 1 discloses a frame line recognition device. In this frame line recognition device, an intersection extraction unit extracts intersections between ruled lines extracted from an input image, and a memory unit stores intersection information including arm vectors along each ruled line belonging to the intersection, each moving away from or approaching the intersection, and a flag indicating whether the arm vector has been used in a route search. The frame line extraction unit performs a route search starting from a specified intersection and according to the direction of the arm vector. When a closed route is detected, the unit extracts the closed route as a frame line and updates the flag corresponding to the arm vector used to detect the closed route to "used." When all flags have been updated to "used," the device terminates the route search.

JP 2012-64098 A

However, when the shape of the intersections is calculated based on the coordinate information of all lines, as in conventional technology, the table area may not be detected correctly if there are objects unrelated to the table lines, such as circuit symbols or diagram symbols.
An object of the present invention is to provide a task support device, a task support method, and a task support system that can correctly analyze a table structure even if an object unrelated to the table lines exists.

In order to solve the above problems, the present invention provides a work support device comprising: a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines that constitute figures that describe circuit components using geometric information, into second circuit drawing data created in a second data format configured to represent figures that describe circuit components using pixels; a line detection unit that detects lines from at least one of the first circuit drawing data and the second circuit drawing data; a character detection unit that detects characters from at least one of the first circuit drawing data and the second circuit drawing data; a circuit symbol detection unit that detects circuit symbols excluding conductor lines from an image area of the second circuit drawing data; a graphical symbol detection unit that detects graphical symbols based on the detected lines and characters; a conductor line detection unit that detects, from the lines detected by the line detection unit, circuit symbols and lines that contact the graphical symbols as conductor lines; a table object detection unit that detects the remaining parts of the lines detected by the line detection unit excluding the circuit symbols, graphical symbols, and conductor lines as table objects that constitute a table; a first table area detection unit that detects a table area that is an area of a table in a drawing based on the table objects; and a table cell detection unit that detects cells within the table area.

The line detection unit can classify the detected lines by line type, which is the type of line, and in this case, it is possible to determine whether the lines are the same cell or not by the line type.
Furthermore, the line detection unit can classify the detected lines into a group representing a line and a group representing a point based on a first threshold value set for the line length, thereby easily classifying the lines into a group representing a line and a group representing a point.
Furthermore, the line detection unit can set a second threshold value for each interval in the direction in which the detected lines extend, and classify the groups representing the lines into solid lines and dashed lines based on the second threshold value. In this case, it is possible to easily classify the lines into solid lines and dashed lines.
Furthermore, the table cell detection unit can determine that cells adjacent to each other across a dashed line are the same cell, thereby improving the accuracy of cell detection.
Furthermore, the first table area detection unit can detect a table area after converting the shape of the intersections of the lines that make up the table, thereby improving the accuracy of table area detection.
Furthermore, the first table area detection unit can convert the shape of the intersections by excluding lines whose length is equal to or less than a predetermined threshold, thereby further improving the accuracy of table area detection.
Furthermore, after converting the shape of the intersections by excluding lines with a length equal to or less than a predetermined threshold, the first table area detection unit can further convert the shape of the intersections for the steps of the cells that make up the table using a predetermined conversion table, thereby further improving the accuracy of table area detection.
The first table area detection unit can convert the shape of the intersection when the table comes into contact with the outline of the drawing, thereby detecting the table area even when the table comes into contact with the outline of the drawing.
In addition, the table cell detection unit can traverse the outer frame of the table to extract the intersections where the outer frame intersects with the ruled lines that make up the table, and detect cells based on the extracted intersections. In this case, cells can be easily detected.
Furthermore, the table cell detection unit can extract the ruled lines that form the cells by repeating the process of extracting straight lines that touch the intersections, then extracting straight lines that touch the extracted straight lines, and then extracting straight lines that touch the extracted straight lines until no new straight lines can be extracted, thereby improving the accuracy of ruled line extraction.

The present invention also includes a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines that make up a figure describing a circuit component using geometric information into second circuit drawing data created in a second data format configured to represent a figure describing a circuit component using pixels; a line detection unit that detects lines from at least one of the first circuit drawing data and the second circuit drawing data; a character detection unit that detects characters from at least one of the first circuit drawing data and the second circuit drawing data; and a character detection unit that detects characters derived from an image area of the second circuit drawing data. This work support device includes a circuit symbol detection unit that detects circuit symbols excluding lines, a graphic symbol detection unit that detects graphic symbols based on the detected lines and characters, a conductor line detection unit that detects circuit symbols and lines that contact graphic symbols from the lines detected by the line detection unit as conductor lines, a table object detection unit that detects the remaining lines detected by the line detection unit excluding circuit symbols, graphic symbols, and conductor lines as table objects that make up tables, a second table area detection unit that determines the table type and separates the table area from the drawing, and an intra-table cell detection unit that detects cells within the table area.

Here, the second table area detection unit is a task support device described in claim 12, which separates tables and other components using lines shared by them as boundaries, converts the shape of the intersections, and then detects table areas. In this case, table areas can be separated with high accuracy.

Furthermore, the present invention provides a work support method in which a processor executes software stored in memory to convert first circuit drawing data created in a first data format configured to represent lines constituting figures describing circuit components using geometric information into second circuit drawing data created in a second data format configured to represent figures describing circuit components using pixels; detect lines from at least one of the first circuit drawing data and the second circuit drawing data; detect characters from at least one of the first circuit drawing data and the second circuit drawing data; detect circuit symbols excluding conductor lines from the image area of the second circuit drawing data; detect graphic symbols based on the detected lines and characters; detect circuit symbols and lines contacting the graphic symbols from the detected lines as conductor lines; detect the remaining parts of the detected lines excluding the circuit symbols, graphic symbols, and conductor lines as table objects constituting a table; detect a table area, which is the area of the table in the drawing, based on the table object; and detect cells within the table area.

Furthermore, the present invention comprises a work terminal on which work is performed by handwriting in cells of a table in a drawing, and a work support device that detects cells and analyzes the content handwritten in the cells. The work support device includes a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines constituting a figure describing a circuit component using geometric information into second circuit drawing data created in a second data format configured to represent the figure describing the circuit component using pixels, a line detection unit that detects lines from at least one of the first circuit drawing data and the second circuit drawing data, and a line detection unit that detects lines from at least one of the first circuit drawing data and the second circuit drawing data. The work support system also includes a character detection unit that detects characters, a circuit symbol detection unit that detects circuit symbols excluding conductor lines from the image area of the second circuit drawing data, a symbol detection unit that detects symbols based on the detected lines and characters, a conductor line detection unit that detects circuit symbols and lines that contact the symbols from the lines detected by the line detection unit as conductor lines, a table object detection unit that detects the remaining parts of the lines detected by the line detection unit excluding the circuit symbols, symbols, and conductor lines as table objects that make up a table, a first table area detection unit that detects a table area, which is the area of a table in the drawing, based on the table object, and a table cell detection unit that detects cells within the table area.

The present invention provides a work support device, work support method, and work support system that can correctly analyze table structures even when objects unrelated to table borders are present.

10A and 10B are diagrams showing examples of tables that are used when using the work support system of this embodiment. 1A is a diagram showing an example of a drawing including unnecessary objects, and FIG. 1B is a diagram showing a table when unnecessary objects are removed from the drawing. FIG. 1 is a block diagram showing a functional configuration of a work support system according to a first embodiment. 10A and 10B are diagrams illustrating a process in which a line detection unit classifies detected lines according to line type. FIG. 10 is a diagram showing a method for detecting cells using line types. FIG. 10 is a diagram showing the result of removing unnecessary objects from a drawing. 10A and 10B are diagrams illustrating cases where the circuit symbol detection unit detects circuit symbols using a template matching method, a deep learning (object detection model) method, and a deep learning (object detection + object identification method). 10A to 10D are diagrams showing examples of graphic symbols. 10A and 10B are diagrams illustrating a method in which a first table area detection unit detects a table area. (A) is a diagram showing a table to be detected. (B) is an example of extracting intersections arranged in the Y direction from the table shown in (A). (C) is an example of extracting intersections arranged in the X direction from the table shown in (A). 10A to 10E are diagrams showing a method in which the table cell detection unit extracts intersections of table ruled lines. 1A shows a case where two tables are adjacent to each other, and FIG. 1B shows a case where an L-shaped table is treated as a rectangular table and cells are extracted. 10A to 10D are diagrams showing a method for detecting cells in a rectangular table with four corners missing. 4 is a flowchart showing the operation of the work support system according to the first embodiment. FIG. 10 is a diagram showing an example of adding information to circuit components in a drawing. 10A is a diagram showing a case where the shape of an intersection cannot be detected correctly, and FIG. 10B is a diagram showing a minute line. FIG. 10 is a diagram showing a conversion table for converting the shape of an intersection when there is a small step between adjacent cells. 1A is a diagram showing a case where the outline of a drawing and the outer frame of a table come into contact, and 1B and 1C are diagrams showing a method for converting the shape of the intersection when the first table area detection unit comes into contact with the outline of a drawing. 10 is a flowchart illustrating the operation of a first table area detection unit in the second embodiment. 1A shows a case where a table is in contact with another element, and 1B shows a case where the outer frame of the table is formed by the outline of the drawing and the four corners cannot be distinguished. FIG. 10 is a block diagram showing the functional configuration of a work support system according to a third embodiment. 10 is a flowchart illustrating a process of determining a table type and detecting a table area based on the table type. FIG. 23 is a diagram showing a method in which the second table area detection unit determines the table type in step S301 of FIG. 22. 23 is a flowchart illustrating special table area detection performed by a second table area detection unit in step S303 of FIG. 22. 24(A) to 24(C) are diagrams illustrating steps S401 to S403 in more detail. 10 is a flowchart showing the operation of the work support system according to the third embodiment.

<Overall Description of Work Support System 1>
1A and 1B are diagrams showing examples of tables that are used when using the work support system 1 of this embodiment.
1A shows table H1 before the work of red-marking the table, and FIG. 1B shows table H2 after the work of red-marking the table. In this case, when the contents of each cell of table H1 shown in FIG. 1A are consistent, the worker handwrites over table H1 and red-marks it.

Because handwritten data includes time-series data, using a wearable device makes it possible to obtain information that cannot be obtained from paper work, such as work procedures and work times.In addition, the work progress rate can be calculated from the total number of check items and the total number of items checked.
However, this handwritten data is merely point cloud data, and is not directly linked to the table. To digitize the results of the vermilion painting work, it is necessary to identify which cells have been checked by comparing the coordinates of the handwritten data on the drawing with the coordinates of the table cells.

Data describing a drawing, such as CAD drawing data, may be described using unique information within the data (e.g., circuit symbol type, identifier, coordinates of figures that make up a table or circuit symbol, etc.). When checking table cells by hand for data created in this format, it is relatively easy to identify the table cells on the handwritten path. This is because it is sufficient to identify the coordinates of the handwritten path and compare them with the coordinates of the cell in the table.

On the other hand, terminal devices such as wearable devices used by workers at work sites may not have the processing power to display drawings created in this data format. In such cases, the drawing data must be converted into a data format that the terminal device can display. Examples of data formats that terminal devices can display include raster data, which represents shapes using pixels. When checking cells in a table by hand on a drawing in this data format, as shown in Figure 1(B), it is not easy to identify the cells in the table. This is because the coordinate information of the table is lost when the data format is converted, and the data becomes mere pixel information.

Furthermore, to identify cells within a table, it is necessary to detect the borders that make up the table, but there are many unnecessary objects in the drawing other than the table borders. These objects hinder the detection of the table. We will explain unnecessary objects below.

FIG. 2A is a diagram showing an example of a drawing including unnecessary objects.
As shown in the figure, there are cases where circuits are written inside the table, wiring is written outside the table, and symbols such as caution marks are in contact with the table. These can cause false detection of ruled line intersections.
FIG. 2B is a diagram showing the table obtained when unnecessary object Ob1 is removed from FIG. 2A.
By removing unnecessary objects, the intersections of lines become clearer, making it less likely that false positives will occur, thereby improving the accuracy of table detection.

The work assistance system 1 according to this embodiment will be described below using the first to third embodiments.

[First embodiment]
First, a description will be given of a first embodiment of the task support system 1. In the first embodiment, a case will be described in which unnecessary objects are removed from a drawing and table cells are detected.

FIG. 3 is a block diagram showing the functional configuration of the work support system 1 according to the first embodiment.
The illustrated work support system 1 comprises a work support device 2 that detects table cells in a drawing and analyzes the content handwritten in the cells, and a work terminal 3 on which a worker performs work by handwriting in the table cells in the drawing.
The work support device 2 is a device that extracts handwritten portions from a drawing that has been handwritten by a worker using the work terminal 3. The work terminal 3 is a terminal device on which the worker works on the drawing, and is, for example, a wearable terminal used when applying red lacquering as shown in FIG.

The work support device 2 includes a drawing analysis unit 10 that analyzes drawing data, which is data representing a drawing, and detects table cells present in the drawing; a handwritten data analysis unit 20 that analyzes handwritten data created by a worker on a work terminal 3; a communication unit 30 that communicates with the work terminal 3; and a DB (Data Base) 40.

The illustrated work support system 1 operates generally as follows.
After the worker handwrites on the drawing at the work terminal 3, the drawing data is sent to the work support device 2. The sent drawing data is received by the communication unit 30 of the work support device 2 and sent to the handwritten data analysis unit 20.
Meanwhile, the drawing analysis unit 10 of the work support device 2 analyzes the drawing data before the worker overwrites it and detects each cell of the table present in the drawing. Then, the handwritten data analysis unit 20 of the work support device 2 extracts the parts handwritten by the worker in each cell. This corresponds to the parts shaded in red in Figure 1 (B). Then, the analysis results of the handwritten parts are saved in the DB 40.

<Description of Drawing Analysis Unit 10>
Next, the drawing analysis unit 10 will be described in detail.
The drawing analysis unit 10 includes a conversion unit 11 that converts the format of image data, a drawing analysis unit 12 that detects table objects that exist in the drawing and constitute a table, a first table area detection unit 13 that detects table areas that are areas of tables that exist in the drawing, and a table cell detection unit 14 that detects cells within a table.

The conversion unit 11 converts first circuit drawing data created in a first data format F1 into second circuit drawing data created in a second data format F2. In this case, the first data format F1 is a data format configured to represent lines constituting a figure describing circuit components using geometric information. The first data format F1 is, for example, CAD drawing data. The second data format F2 is a data format configured to represent a figure describing circuit components using pixels. The second data format F2 is, for example, raster format data.
The drawing analysis unit 12 detects table objects, which are objects that make up a table, from the converted drawing data. Table objects are, for example, ruled lines.
The first table area detection unit 13 detects a table area, which is a table area in a drawing, based on a table object.
The table cell detection unit 14 detects cells in a table area, that is, the table cell detection unit 14 detects cells separated by ruled lines in a table.

<Detailed Description of Drawing Analysis Unit 12>
Next, the drawing analysis unit 12 will be described in detail.
The drawing analysis unit 12 also includes a circuit symbol detection unit 121 , a line detection unit 122 , a character detection unit 123 , a conductor detection unit 124 , a diagram symbol detection unit 125 , and a table object detection unit 126 .

The circuit symbol detection unit 121 detects circuit symbols excluding conductors from the image area of the second circuit drawing data.
The line detection unit 122 detects lines from at least one of the first circuit diagram data and the second circuit diagram data.
The character detection unit 123 detects characters from at least one of the first circuit diagram data and the second circuit diagram data.
Conductor detection unit 124 detects, from the lines detected by line detection unit 122, lines that contact circuit symbols and graphic symbols as conductors.
The graphic symbol detection unit 125 detects graphic symbols based on the detected lines and characters.
The table object detection unit 126 detects the remaining parts, excluding circuit symbols, diagram symbols, and conducting wires, from the lines detected by the line detection unit 122 as table objects that make up a table.

The line detection unit 122 detects lines from the image data and classifies the detected lines according to their line types.
FIG. 4 is a diagram showing the process in which the line detection unit 122 classifies the detected lines by line type.
The line detection unit 122 classifies detected lines into a group representing lines and a group representing points, based on a first threshold value set for the line length. That is, if the detected line length is equal to or greater than the first threshold value, it is classified into the line group. Conversely, if the detected line length is less than the first threshold value, it is classified into the point group. The first threshold value is set to a length that allows dotted lines created with drawing software to be detected as points.

Next, the line detection unit 122 sets a second threshold for the spacing between each detected line in the direction it extends, and classifies the group representing the lines into solid lines and dashed lines based on the second threshold. In other words, if similar lines exist in the direction it extends and the spacing between them exceeds the second threshold, they are separate lines, and the detected lines are classified as solid lines. If the spacing is equal to or less than the second threshold, they are classified as dashed lines.

Furthermore, when a detected line is classified as a dashed line, the line detection unit 122 classifies the type of dashed line based on the number of points between the detected lines. That is, if the number of points is 0, it is classified as a normal dashed line. If the number of points is 1, it is classified as a chain line (dash-dotted line). If the number of points is 2, it is classified as a two-dot chain line.

The line detection unit 122 then sets a third threshold for the spacing between each group of points in the direction in which the detected lines extend, and classifies them into points and dotted lines based on the third threshold. In other words, if similar lines exist in the direction in which the detected lines extend and the spacing between them exceeds the third threshold, these are separate points, and the detected line is classified as a point. If the spacing is equal to or less than the third threshold, the line is classified as a dotted line.

FIG. 5 is a diagram showing a method for detecting cells using line types.
The table consists of horizontal and vertical lines.
By analyzing horizontal and vertical lines, excluding diagonal lines that are unrelated to the table, the processing speed and accuracy of the analysis when detecting cells are improved.
The dotted lines in table H3 shown in Figure 5 are not lines that divide cells, but lines that align characters. In this case, it is more convenient to consider rows 1 through 4 as one cell. In this case, the table cell detection unit 14 considers cells that are adjacent to each other on either side of the dashed line to be the same cell. By distinguishing and using line types, table information can be digitized correctly as intended by the drawing creator.

FIG. 6 shows the result of removing unnecessary objects from a drawing.
The input drawing Z0 indicates that characters, lines, circuit symbols, diagram symbols, and conductors exist as unnecessary objects in the drawing.
In this case, in this embodiment, first, the character detection unit 123, the line detection unit 122, and the circuit symbol detection unit 121 detect characters, lines, and circuit symbols, respectively.
Drawing Z1 shows the result of detecting characters in the diagram by the character detection unit 123. Drawing Z2 shows the result of detecting lines in the diagram by the line detection unit 122. Drawing Z3 shows the result of detecting circuit symbols in the diagram by the circuit symbol detection unit 121.

Furthermore, in this embodiment, the graphic symbol detection unit 125 detects graphic symbols based on the detected characters and lines. Then, in this embodiment, the conductor detection unit 124 detects, as conductors, circuit symbols and lines that contact graphic symbols from the lines detected by the line detection unit 122. Drawing Z4 shows the results of the detection of graphic symbols in the drawing by the graphic symbol detection unit 125. Furthermore, drawing Z5 shows the results of the detection of conductor lines in the drawing by the conductor detection unit 124.
The table object detection unit 126 then removes these unnecessary objects from the input drawing Z0. Drawing Z6 shows a case where the table object detection unit 126 has removed the unnecessary objects and detected, as the remainder, a table object Ob2 that constitutes a table.

To detect circuit symbols, methods such as template matching, deep learning (object detection model) and deep learning (object detection + object identification) can be used.
FIG. 7 is a diagram showing a case where the circuit symbol detection unit 121 detects circuit symbols by a template matching method, a deep learning (object detection model) method, and a deep learning (object detection + object identification method).
(Method A) The template matching method detects circuit symbols by performing template matching using a template image prepared in advance with the input drawing Z0. The template matching method is easy to implement, but is not good at scaling or rotating the image.
(Method B) Deep learning (object detection model) method detects circuit symbols using a detector trained using object detection models such as R-CNN, YOLO, and SSD. Because it is a single model, it is difficult to train it to eliminate detection errors.
(Method C) Deep learning (object detection + object identification method) detects circuit symbols using a detector trained with an object detection model as well as an object identification model such as ResNet, DenseNet, AmoebaNet, or EfficientNet. This allows the identification model to reject detection errors, improving the accuracy of circuit symbol detection.

8A to 8D are diagrams showing examples of the graphic symbols.
In this case, in addition to the attention mark shown in Fig. 8(A), the arrow shown in Fig. 8(B), and the brackets shown in Fig. 8(C), simple character strings may also be treated as graphic symbols, as shown in Fig. 8(D). In this case, if lines touch the left and right or top and bottom of a character area and these lines are on the same straight line, it is detected that the character is being treated as a symbol. These graphic symbols can be detected on a rule-based basis.

<Detailed Description of First Table Area Detection Unit 13>
Next, the first table area detection unit 13 will be described in detail.
The first table area detection unit 13 detects a table area as follows.
9A and 9B are diagrams showing a method in which the first table area detection unit 13 detects a table area.
The first table area detection unit 13 detects table areas by circulating the outer frame of the table based on the extracted table object.

The rotation direction at this time is shown as the (A-1) rotation direction in Fig. 9A. That is, there are two ways, clockwise and counterclockwise, and either one is adopted.
Furthermore, when the robot reaches an intersection where the lines that make up the table intersect during the rotation, the direction of travel is determined by the shape of the intersection. The shape of the intersection is where the horizontal and vertical lines that make up the table intersect. For example, the horizontal and vertical lines intersect in a shape that forms a ""[ shape. Note that there are four shapes for this orientation, rotated 90 degrees each. Also, the horizontal and vertical lines intersect in a shape that forms a "T" shape. Note that there are four shapes for this orientation, rotated 90 degrees each. Furthermore, the horizontal and vertical lines intersect in a shape that forms a "+".
The rules for the direction of travel when an intersection is reached are shown as (A-2) Direction of Travel in Figure 9(A). Here, the rules for the direction of travel when circulating the outer frame of the table are shown for both clockwise and counterclockwise directions.

An actual example of circulating the outer frame of a table will be described with reference to FIG. 9(B).
First, the starting point is one of the intersections on the outer frame of the table as shown in the figure. In this case, the starting point is the intersection located in the upper left corner as shown in (a). The shape of the intersection at this point corresponds to the "upper left" in the clockwise direction of (A-2). The direction of travel at this intersection is given priority in the direction of No. 1, and if it is not possible to travel in the direction of No. 1, the direction is changed to No. 2. In this case, since it is not possible to travel upward, travel will proceed to the right.

Thereafter, if the shape of the intersection does not reach any of the intersections of "upper left", "upper right", "lower right", or "lower left" in (A-2), this rule will be maintained, but if it does reach any of them, the rule will be changed to the rule described in (A-2).
Therefore, the second intersection shown in (b) is not an intersection of "upper left,""upperright,""lowerright," or "lower left," so the "upper left" rule is maintained. In this case, No. 1 is the upward direction, and progress can be made in that direction, so progress is made upward.
The third intersection shown in (c) is the "upper left" intersection, so this rule is maintained and the process proceeds to the right of No. 2.
The fourth intersection shown in (d) is not an "upper left,""upperright,""lowerright," or "lower left" intersection, so the "upper left" rule is maintained. In this case, the path proceeds to the right of No. 2.
Since the fifth intersection shown in (e) is an "upper right" intersection, the rule is changed to "upper right." In this case, since it is not possible to proceed to the right of No. 1, it proceeds downward of No. 2.
This process is repeated from (f) to (h) and returns to the original starting point at (h). This allows the table area to be detected as shown in (i).

<Detailed Description of Table Cell Detection Unit 14>
Next, the table cell detection unit 14 will be described in detail.
The table cell detection unit 14 detects cells in a table as follows.
10A is a diagram showing table H4 to be detected. As shown in the figure, table H4 is formed by a plurality of lines (ruled lines) that make up table H4, which are arranged vertically (up and down, Y direction in the figure) and horizontally (left and right, X direction in the figure) and intersect with each other. In this embodiment, cells in table H4 are detected by focusing on the intersections of the ruled lines.
Figure 10(B) is an example of extracting intersections aligned in the Y direction from Table H4 shown in Figure 10(A). In this case, the intersections are indicated by "●" or "◯". At this time, four intersections indicated by Y1 to Y4 are detected in the Y direction. The thick lines connecting these intersections in the Y direction are the ruled lines. As a result, it can be seen that the cell arrangement is three rows. In this way, the number of cell rows can be defined by the number of intersections aligned in the Y direction.
Also, Figure 10(C) is an example of extracting intersections aligned in the X direction from Table H4 shown in Figure 10(A). In this case, the intersections are indicated by "●" or "◯". At this time, five intersections indicated by X1 to X5 are detected in the X direction. The thick lines connecting these intersections in the X direction are the ruled lines. As a result, it can be seen that the cell arrangement is four columns. In this way, the number of cell columns can be defined by the number of intersections aligned in the X direction.

However, the detected table object is not necessarily a table border, and the detected intersections are not necessarily table border intersections.
11A to 11E are diagrams showing a method in which the table cell detection unit 14 extracts the intersections of the ruled lines of a table.
FIG. 11A shows a case where table H5 contains lines other than ruled lines. In this case, the L-shaped table object Ob3 is not a ruled line of table H5. Furthermore, the intersections indicated by "★" in FIG. 11B are not intersections of ruled lines of table H5. Note that the intersections of ruled lines of table H5 are indicated by "●" here.
In this embodiment, the table cell detection unit 14 excludes the intersections indicated by "★" in the following manner.
11C, the table cell detection unit 14 traverses the outer frame of table H5 to detect the intersections between the outer frame and the ruled lines that make up table H5. The starting point for traversing the outer frame of table H5 is, for example, the upper left corner, and the outer frame of table H5 is traced clockwise or counterclockwise. When the table cell detection unit 14 returns to the original starting point, it determines this as the end point.
The table cell detection unit 14 then detects cells based on the extracted intersections. Specifically, as shown in Figure 11 (D), the table cell detection unit 14 extracts lines that touch these intersections. The table cell detection unit 14 then extracts lines that touch the extracted straight lines. The table cell detection unit 14 then repeats the process of extracting lines that touch the extracted straight lines until no new lines can be extracted. In this way, the table cell detection unit 14 extracts the ruled lines that form the cells.
As shown in FIG. 11(E), the table cell detection unit 14 detects the intersections on the extracted lines as intersections of the ruled lines of table H5.
According to this method, the L-shaped table object Ob3 is not extracted, and the intersections indicated by "★" are excluded.

When detecting cells in a table that are not rectangular, the table cell detection unit 14 can do so by the following method.
12A shows a case where two tables are adjacent to each other, one being an L-shaped table H6 and the other being a rectangular table H7.
In the method shown in Figure 10, the L-shaped table H6 may be treated as a rectangular table, as shown in Figure 12(B), and cells may be extracted. In the example shown, the areas of the two tables H6 and H7 may overlap. It is rare for tables to be deformed when used, and tables with shapes other than rectangular usually have one or more of the four corners missing.

13A to 13D are diagrams showing a method for detecting cells in a rectangular table with missing corners.
FIG. 13A shows an example of a rectangular table H8 with the four corners cut off.
In this case, the areas around the four corners of the rectangle are not table areas, but non-table areas, as shown in Fig. 13B. Here, the non-table areas are detected by the following method.
First, as shown in Fig. 13C, the coordinates of the four corners are calculated from the combination of maximum and minimum values in the X and Y directions of the table area. Here, the four corners are designated as corners 1 to 4.
Next, as shown in Figure 13(D), among the calculated coordinates of the four corners, the four corners that are not included in the outer frame of table H8 are set as starting points. In this case, all four corners are set as starting points. Here, these are shown as starting points 1 to 4.
The distance from each starting point to the nearest intersection in the horizontal direction is defined as the width, and the distance to the nearest intersection in the vertical direction is defined as the height. In this case, as shown in Figure 13(D), widths 1 to 4 and heights 1 to 4 are defined for starting points 1 to 4, respectively. The four rectangles with widths 1 to 4 and heights 1 to 4 become non-table areas.

14 is a flowchart showing the operation of the task support system 1 according to the first embodiment. Note that this flowchart describes the processing performed after the conversion unit 11 converts image data from the first data format F1 to the second data format F2.
First, the line detection unit 122 detects lines from at least one of the first circuit diagram data and the second circuit diagram data (step S101).
Next, the character detection unit 123 detects characters from at least one of the first circuit drawing data and the second circuit drawing data (step S102).
Furthermore, the circuit symbol detection unit 121 detects circuit symbols excluding conductors from the image area of the second circuit drawing data (step S103).
Furthermore, the conductor detection unit 124 detects, from the lines detected by the line detection unit 122, lines that contact the circuit symbols and graphic symbols as conductors (step S104).
Then, the graphic symbol detection unit 125 detects graphic symbols based on the detected lines and characters (step S105).

Next, the table object detection unit 126 detects the remaining parts, excluding the circuit symbols, diagram symbols, and conductors, from the lines detected by the line detection unit 122 as table objects that make up a table (step S106).
The first table area detection unit 13 detects a table area, which is an area of a table in the drawing, based on the table object (step S107). Here, the detection of the table area is illustrated as first table area detection.
Then, the table cell detection unit 14 detects cells in the table area (step S108).

Meanwhile, the communication unit 30 sends the image data to the work terminal 3 (step S109).
At the work terminal 3, the worker performs the vermilion painting work as shown in FIG. 1 (step S110).
Then, the handwritten data by the worker is sent to the work support device 2 and received by the communication unit 18 (step S111).
Next, the handwritten data analysis unit 20 analyzes the handwritten data handwritten in the cells (step S112). At this time, the information on the cells detected in step S108 is used to analyze the handwritten data for each cell.
The analysis results are then registered in the DB 40 (step S113).
Thereafter, the worker is notified that the work has been completed (step S114).

FIG. 15 is a diagram showing an example of adding information to circuit components in a drawing.
For example, an electronic component such as a CPU may have a table inside it that shows the wiring connections. Table H9 inside the CPU on the left side of the drawing in Figure 15 shows that the IF1 signal of the CPU is connected to terminals 1 and 2. It also shows that the IF2 signal of the CPU is connected to terminals 3 and 4.
15 shows the case where the ammeter is in contact with Table H10. The ammeter on the left is for the power supply, and the ammeter on the right is for the standby power supply. By adding the information from Tables H9 and H10 to the circuit components in this way, the diagram can be digitized in more detail.

Second Embodiment
Next, a description will be given of a second embodiment of the task support system 1. In the second embodiment, the first table area detection unit 13 detects a table area after converting the shape of the intersections of the lines that make up the table.

FIG. 16A is a diagram showing a case where the shape of the intersection cannot be detected correctly.
FIG. 16A shows an intersection K1 at a corner of a table. The intersection K1 is originally shaped like "", but because the table object protrudes from the corner, it is detected as a "T" shape, as shown in the speech bubble. In such a case, it becomes impossible to cycle as shown in FIG. 9, making it difficult to correctly detect the table area.

In this embodiment, the first table area detection unit 13 calculates the length of lines in each direction with intersection K1 as the origin, and determines the shape of the intersection by ignoring lines that are less than the threshold. That is, if the length of the "△" shown in the left diagram of Figure 16(B) is a very small line that is less than this threshold, this part is ignored, and the shape is determined to be " " as shown in the speech bubble in the right diagram of Figure 16(B). This can also be said to mean that the first table area detection unit 13 converts the shape of the intersection by ignoring lines that are less than a predetermined threshold in length.

However, as shown in Figure 16, if the shape of the intersection is determined while ignoring lines of very small length, the intersection of the table may not be detected correctly if there is a small step between adjacent cells in the table.
FIG. 17 shows a conversion table for converting the shape of an intersection when there is a small step between adjacent cells.
The first table area detection unit 13 converts the shape of the intersections by assuming that there are no lines with a length less than a predetermined threshold, and then further converts the shape of the intersections using a predetermined conversion table for the steps of the cells that make up the table.
For example, in the case shown in (1), the shape of the intersection is a combination of upper left and upper right, and a slight step occurs between adjacent cells C1 and C2. The first table area detection unit 13 then shows the conditions under which it determines that such a step occurs, and the conversion method. In other words, because the vertical line L1 of this minute step portion is of a very small length, the intersection shape, which is actually a "T", is judged to be a "" shape. Therefore, a process is performed to convert the intersection shape from the "" shape to the original "T" shape.

Furthermore, if a table contacts a contour line in the drawing, the table area cannot be detected correctly.
FIG. 18A shows a case where the outline Lr of the drawing and the outer frame of the table H11 are in contact with each other.
In this case, the detected shape of the intersection at the top right corner of table H11 is different from the correct answer because it touches the contour line Lr. Similarly, the detected shape of the intersection at the right side of table H11 is also different from the correct answer because it touches the contour line Lr.

18B and 18C are diagrams showing a method in which the first table area detection unit 13 converts the shape of the intersection when the table H11 contacts the contour line Lr of the drawing.
First, the first table area detection unit 13 detects the contour line Lr. Here, as shown in Fig. 18(B), the first table area detection unit 13 detects the longest horizontal and vertical lines closest to the origin (the lower left corner of the rectangular contour line Lr) and the longest horizontal and vertical lines farthest from the origin as the contour line Lr. Alternatively, the first table area detection unit 13 may detect a line forming a rectangular area with a large area as the contour line Lr.
18B, the first table area detection unit 13 converts the shapes of the T and cross in accordance with the contour line Lr. For example, an upward-facing T that touches the bottom frame is deleted, and a cross is converted into a downward-facing T.
As shown in Fig. 18C, the first table area detection unit 13 starts from one of the four corners of the table H11 and moves clockwise or counterclockwise around the outer frame of the table H11. When the first table area detection unit 13 comes into contact with the contour line Lr, it converts the shape of the intersection based on the determination table shown at the bottom of Fig. 18C. The determination table shown in Fig. 18C indicates the starting point, the rotation direction, and the converted shape of the intersection when it comes into contact with the contour line Lr.

FIG. 19 is a flowchart illustrating the operation of the first table area detection unit 13 in the second embodiment.
First, the first table region detection unit 13 adjusts the lengths of the lines that make up the table (step S201). Specifically, if the gap between a horizontal line and a vertical line is equal to or smaller than a threshold, the lines are extended to eliminate the gap.
Next, the first table area detection unit 13 detects the intersections of the lines that make up the table (step S202). The coordinates of the intersections can be detected from the coordinate relationship between the start and end points of the horizontal and vertical lines.
Next, the first table area detection unit 13 converts the shape of the intersection of the protruding ruled lines (step S203). Specifically, as shown in Fig. 16, the first table area detection unit 13 calculates the length of the lines in each direction using the intersection as the origin, and determines the shape of the intersection while ignoring lines that are equal to or shorter than a threshold.

Next, the first table area detection unit 13 converts the shape of the intersection of the minute step (step S204). Specifically, as shown in Fig. 17, the first table area detection unit 13 converts the shape of the intersection while referring to a conversion table.
Next, the first table area detection unit 13 converts the shapes of the intersections on the contour line (step S205). Specifically, as shown in Fig. 18, the first table area detection unit 13 converts the shapes of the T and cross in the outer frame of the table H11 that contacts the contour line Lr by referring to a conversion table.
Then, the first table area detection unit 13 detects the table area (step S206). Specifically, the first table area detection unit 13 starts from one of the four corners of the table, circles around the outer frame, and if it returns to the start point, it detects the area as a candidate for a table.

[Third embodiment]
Next, a description will be given of a third embodiment of the task support system 1. In the third embodiment, the algorithm for detecting a table area is changed depending on the table type, which is the type of table.

FIG. 20A shows a case where the table is in contact with another component.
The left diagram in Fig. 20A shows a case where a table H12 comes into contact with a non-table object Ob4. The right diagram in Fig. 20A shows a case where a table H13 comes into contact with a table H14. In this case, the other components that come into contact with the table are object Ob4 and another table, respectively.
FIG. 20B shows a case where the outer frame of table H15 is formed by the outline Lr of the drawing and the four corners cannot be distinguished.
In this case, it can be said that the table H15 and the outline Lr of the drawing are in contact with each other. In this case, the other components that are in contact with the table are the outlines Lr.

FIG. 21 is a block diagram showing the functional configuration of a task support system 1 according to the third embodiment.
The task support system 1 shown in FIG. 21 is different from the task support system 1 in the first embodiment shown in FIG. 3 in that a second table area detection unit 15 is added, but is otherwise similar.
Therefore, in the following description of the task support system 1, the second table area detection unit 15 will be mainly described, and a description of the other functional units will be omitted.
In this embodiment, the second table area detection unit 15 determines the table type, which is the type of table, and changes the algorithm for detecting the table area.

FIG. 22 is a flowchart illustrating a process for determining the table type and detecting the table area based on the table type.
First, the second table area detection unit 15 determines the table type (step S301). Here, "table type" refers to the type of table classified according to whether or not the table is in contact with other components. Here, if the table is not in contact with other components, it is classified as a general table. On the other hand, if the table is in contact with other components, it is classified as a special table.
If the table is a general table that does not contact other components (if it is a general table in step S301), the first table area detection unit 13 performs general table area detection, which is a process for detecting normal table areas (step S302). That is, the first table area detection unit 13 detects table areas using the methods described in the first and second embodiments.
On the other hand, if the table is a special table that is in contact with other components (special table in step S301), the second table area detection unit 15 performs special table area detection, which is a process of detecting the table area of the special table (step S303).
After steps S302 and S303, the table cell detection unit 14 performs processing to detect cells in the table area (step S304), as in the cases described in the first and second embodiments.
In this way, the second table area detection unit 15 determines the table type, which is the type of table, separates the table area, which is the area of the table in the drawing, and detects the table area, which is the area of the table in the drawing.

<Explanation of surface species determination>
FIG. 23 is a diagram showing a method in which the second table area detection unit 15 determines the table type in step S301 of FIG.
In this embodiment, the type of table is determined based on the information in the title field that has been registered in advance. For example, if the name of a drawing in H16 is registered in advance as "Connection order of XX device" = special table, the special table can be determined.
In this embodiment, in the case of a special table, the second table area detection unit 15 detects the table area.

<Explanation of special tablespace detection>
FIG. 24 is a flowchart illustrating the special table area detection performed by the second table area detection unit 15 in step S303 of FIG.
First, the second table area detection unit 15 separates the table areas (step S401).
Next, the second table area detection unit 15 converts the shape of the intersections (step S402).
Then, the second table area detection unit 15 detects the table area (step S403).

25A to 25C are diagrams illustrating steps S401 to S403 in FIG. 24 in more detail.
Here, the user separates the mutually adjacent tables H13 and H14 displayed on the screen while looking at the screen, and the second table area detection unit 15 acquires the results.

FIG. 25A is a diagram showing the process of separating table areas in step S401 of FIG.
When the user clicks or drags a pointing device such as a mouse on the screen, a selection screen D1 for separation areas or shared lines is displayed. To specify a line that constitutes either table H13 or H14, the user selects "Separation Area" on the selection screen D1. To specify a line that is commonly used in tables H13 and H14, the user selects "Shared Line" on the selection screen D1. FIG. 25A shows that "Shared Line" on the selection screen D1 is selected, and shared line L2 is selected with cursor Cu1 shown on the left. FIG. 25A also shows that "Separation Area" on the selection screen D1 is selected, and lines that constitute table H14 are selected with two rectangular regions R1 and R2 with cursor Cu2 shown on the right.

FIG. 25B is a diagram showing the process of transforming the shape of the intersections in step S402 of FIG.
When the table regions are separated, a separate screen for each table region is displayed on the screen. In this state, when the user drags the mouse, a selection screen D2 appears: upper left, upper right, lower left, lower right, or none. The user then specifies the erroneously detected location using regions R3 and R4, and selects either upper left, upper right, lower left, lower right, or none to change the shape of the intersection. Figure 25(B) shows a case where the shape of the intersection where the shared line L2 extends beyond the tables H13 and H14 is changed.

FIG. 25C is a diagram showing the process of detecting a table area in step S403 of FIG.
This process is similar to the process described with reference to FIG. 9, and for each of tables H13 and H14, the table area is detected by traversing the outer frame of the table.
In this way, the second table area detection unit 15 separates the table from other elements using the lines shared by them as boundaries, converts the shapes of the intersections, and then detects the table area.

FIG. 26 is a flowchart showing the operation of the task support system 1 in the third embodiment.
In steps S507 to S509 in Fig. 26, the processing explained in Fig. 22 is performed. Other steps S501 to S506 and steps S510 to S515 in the flowchart are the same as steps S101 to S106 and steps S109 to S114 in the flowchart in Fig. 14.

The above-described aspects provide a work assistance device, work assistance method, and work assistance system that can correctly analyze table structures even when objects unrelated to table lines are present.

<Explanation of work support method>
The above-described processing performed by the task support device 2 is realized by the cooperation of software and hardware resources. That is, a processor in a computer provided in the task support device 2 loads software that realizes each of the above-described functions into memory and executes it to realize each of these functions.
Therefore, the processing performed by the work support device 2 can be considered as a work support method in which, by the processor executing software recorded in the memory, the processor converts first circuit drawing data created in a first data format configured to represent lines that make up figures that describe circuit components using geometric information into second circuit drawing data created in a second data format configured to represent figures that describe circuit components using pixels, detects lines from at least one of the first circuit drawing data and the second circuit drawing data, detects characters from at least one of the first circuit drawing data and the second circuit drawing data, detects circuit symbols excluding conductors from the image area of the second circuit drawing data, detects graphic symbols based on the detected lines and characters, detects circuit symbols and lines that contact the graphic symbols from the detected lines as conductors, detects the remaining parts of the detected lines excluding the circuit symbols, graphic symbols, and conductors as table objects that make up a table, detects a table area that is the area of the table in the drawing based on the table object, and detects cells within the table area.

The present embodiment has been described above, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear from the claims that various modifications and improvements to the above embodiment are also included within the technical scope of the present invention.

1...Work support system, 2...Work support device, 3...Work terminal, 10...Drawing analysis unit, 11...Conversion unit, 12...Drawing analysis unit, 13...First table area detection unit, 14...Table cell detection unit, 15...Second table area detection unit, 20...Overwrite data analysis unit, 30...Communication unit, 40...DB, 121...Circuit symbol detection unit, 122...Line detection unit, 123...Character detection unit, 124...Conducting line detection unit, 125...Graphical symbol detection unit, 126...Table object detection unit, H1 to H16...Table

Claims (15)

a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines constituting a figure describing a circuit component by geometric information into second circuit drawing data created in a second data format configured to represent the figure describing the circuit component by pixels;
a line detection unit that detects lines from at least one of the first circuit diagram data and the second circuit diagram data;
a character detection unit that detects characters from at least one of the first circuit diagram data and the second circuit diagram data;
a circuit symbol detection unit that detects circuit symbols excluding conductors from an image area of the second circuit drawing data;
a graphic symbol detection unit that detects a graphic symbol based on the detected line and character;
a conductor detection unit that detects, from the lines detected by the line detection unit, lines that contact the circuit symbol and the graphic symbol as conductors;
a table object detection unit that detects the remaining parts of the lines detected by the line detection unit, excluding the circuit symbol, the graphic symbol, and the conductor wire, as table objects that constitute a table;
a first table area detection unit that detects a table area in a drawing based on the table object;
a table cell detection unit that detects cells in the table region;
A work support device comprising: The work support device according to claim 1, wherein the line detection unit classifies the detected lines by line type. The work support device described in claim 2, wherein the line detection unit classifies detected lines into a group representing lines and a group representing points based on a first threshold value set for the length of the lines. The work assistance device described in claim 3, wherein the line detection unit sets a second threshold value for the spacing between detected lines in the direction of their extension, and classifies the groups representing lines into solid lines and dashed lines based on the second threshold value. The work support device described in claim 4, wherein the table cell detection unit determines that the cells adjacent to each other on either side of the dashed line are the same cell. The work assistance device described in claim 1, wherein the first table area detection unit detects the table area after converting the shape of the intersections of the lines that make up the table. The work assistance device described in claim 6, wherein the first surface area detection unit converts the shape of the intersections by excluding lines whose length is less than a predetermined threshold. The work support device described in claim 7, wherein the first table region detection unit converts the shape of the intersections by assuming that there are no lines with a length equal to or less than a predetermined threshold, and then further converts the shape of the intersections using a predetermined conversion table to account for steps between cells that make up the table. The work support device described in claim 6, wherein the first table area detection unit converts the shape of the intersection when the table contacts the outline of the drawing. The work support device described in claim 1, wherein the table cell detection unit circulates around the outer frame of the table to extract intersections between the outer frame and the ruled lines that make up the table, and detects the cells based on the extracted intersections. The work support device described in claim 10, wherein the table cell detection unit extracts the ruled lines that form the cells by repeating the process of extracting straight lines tangent to the intersections, then extracting straight lines tangent to the extracted straight lines, and then extracting straight lines tangent to the extracted straight lines until no new straight lines can be extracted. a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines constituting a figure describing a circuit component using geometric information into second circuit drawing data created in a second data format configured to represent the figure describing the circuit component using pixels;
a line detection unit that detects lines from at least one of the first circuit diagram data and the second circuit diagram data;
a character detection unit that detects characters from at least one of the first circuit diagram data and the second circuit diagram data;
a circuit symbol detection unit that detects circuit symbols excluding conductors from an image area of the second circuit drawing data;
a symbol detection unit that detects a symbol based on the detected line and character;
a conductor detection unit that detects the lines that contact the circuit symbol and the graphic symbol as conductors from the lines detected by the line detection unit;
a table object detection unit that detects the remaining parts of the lines detected by the line detection unit, excluding the circuit symbol, the graphic symbol, and the conductor wire, as table objects that constitute a table;
a second table area detection unit that identifies the table type and separates a table area that is a table area in the drawing;
a table cell detection unit that detects cells in the table region;
A work support device comprising: The work support device described in claim 12, wherein the second table area detection unit separates the table from other components using lines shared by the table and other components as boundaries, converts the shape of the intersections, and then detects the table area. The processor executes the software stored in the memory.
converting first circuit drawing data created in a first data format configured to represent lines constituting a figure describing a circuit component by geometric information into second circuit drawing data created in a second data format configured to represent the figure describing the circuit component by pixels;
Detecting lines from at least one of the first circuit drawing data and the second circuit drawing data;
Detecting characters from at least one of the first circuit drawing data and the second circuit drawing data;
Detecting circuit symbols excluding conductor lines from the image area of the second circuit drawing data;
Detecting a graphic symbol based on the detected line and character;
Detecting the circuit symbol and the wires contacting the graphic symbol as conductors from the detected wires;
detecting the remaining parts of the detected lines, excluding the circuit symbol, the graphic symbol, and the conductor, as table objects constituting a table;
Detecting a table area in the drawing based on the table object;
A work support method for detecting a cell in the table region. a work terminal on which work is performed by handwriting in table cells in the drawing;
a work support device that detects the cells and analyzes the contents handwritten in the cells;
Equipped with
The work assistance device includes:
a conversion unit that converts first circuit drawing data created in a first data format configured to represent lines constituting a figure describing a circuit component using geometric information into second circuit drawing data created in a second data format configured to represent the figure describing the circuit component using pixels;
a line detection unit that detects lines from at least one of the first circuit diagram data and the second circuit diagram data;
a character detection unit that detects characters from at least one of the first circuit diagram data and the second circuit diagram data;
a circuit symbol detection unit that detects circuit symbols excluding conductors from an image area of the second circuit drawing data;
a symbol detection unit that detects a symbol based on the detected line and character;
a conductor detection unit that detects, from the lines detected by the line detection unit, lines that contact the circuit symbol and the graphic symbol as conductors;
a table object detection unit that detects the remaining parts of the lines detected by the line detection unit, excluding the circuit symbol, the graphic symbol, and the conductor wire, as table objects that constitute a table;
a first table area detection unit that detects a table area in a drawing based on the table object;
a table cell detection unit that detects the cells in the table area;
A work support system comprising: