JP7664693B2 - Drawing recognition device and drawing recognition program - Google Patents

Description

This disclosure relates to a drawing recognition device and a drawing recognition program.

2. Description of the Related Art Drawing recognition techniques are known that recognize characters or symbols present on a drawing containing a mixture of characters and symbols and determine their correspondence.
For example, Patent Document 1 discloses a drawing recognition technology that, in drawing data containing a mixture of characters and symbols, determines the correspondence between which individually recognized characters correspond to which symbol or pipe, based on previously prepared placement knowledge of the relative positions between characters and symbols.

JP 2004-234424 A

In conventional technology such as that disclosed in Patent Document 1, there was an issue that in order to perform drawing recognition and determine the correspondence between characters, symbols, etc., placement knowledge had to be prepared in advance.

This disclosure has been made to solve the above problems, and aims to provide a drawing recognition device that is capable of determining the correspondence between characters and symbols without requiring knowledge of placement.

The drawing recognition device according to the present disclosure includes a drawing acquisition unit that acquires drawing data, a recognition unit that recognizes information about symbols, including position information of the symbols, information about attributes, including position information of the attributes, in the drawing data acquired by the drawing acquisition unit, and information about connection lines, including position information of the connection lines, in the drawing data acquired by the drawing acquisition unit, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols, and information about the symbols, attributes, and connection lines recognized by the recognition unit. and a determination unit which determines the correspondence between the symbols, attributes, and other symbols in the drawing data acquired by the drawing acquisition unit based on information regarding the symbols, attributes, and connection lines recognized by the recognition unit, and on the basis of association information regarding the degree of association between the symbols and attributes shown on the drawing data, which is represented by the number of times a symbol and an attribute related to the symbol are shown together on the drawing data, or on the degree of association between connections between symbols, which is represented by the number of times a symbol and a destination symbol to which the symbol is connected are shown connected on the drawing data .

According to this disclosure, it is possible to determine the correspondence between characters and symbols without requiring placement knowledge.

1 is a diagram illustrating an example of a configuration of a drawing recognition device according to a first embodiment; FIG. 13 is a diagram for explaining an example of an image drawing in embodiment 1. 11 is a diagram for explaining an example of drawing characteristic information in embodiment 1. FIG. 2 is a diagram illustrating a detailed configuration example of a recognition unit included in the drawing recognition device according to the first embodiment; FIG. FIG. 5A shows an example image of the symbol recognition result acquired by the symbol recognition unit, the attribute recognition result acquired by the attribute information recognition unit, and the connection line recognition result acquired by the connection line recognition unit in embodiment 1, where FIG. 5A shows an image in which the symbol recognition result, attribute recognition result, and connection line recognition result are mapped onto an image drawing acquired by the drawing acquisition unit, and FIG. 5B shows an image in which the contents of the symbol recognition result, attribute recognition result, and connection line recognition result are displayed in text. 4 is a diagram for explaining an example of an image of the contents of part information stored in a part recording unit in the first embodiment. FIG. 10 is a diagram for explaining an example of an image of the contents of relevance information stored in a relevance recording unit in the first embodiment. FIG. 11 is an explanatory diagram showing an image of a specific example of a process in which a judgment unit in embodiment 1 comprehensively judges whether symbols, attributes, and connections between symbols in inference data have been correctly recognized. FIG. 10 is a diagram for explaining an image of an example of a determination result output by a determination unit in the first embodiment. FIG. 13 is an explanatory diagram showing an image of a specific example of a process in which a comparison unit compares a post-determination image drawing output by a generation unit with an image drawing acquired by a drawing acquisition unit in embodiment 1. FIG. 4 is a flowchart for explaining an operation of the drawing recognition device according to the first embodiment; 12 is a flowchart for explaining details of the inference operation performed by the recognition unit in step ST1102 of FIG. 11. 13A and 13B are diagrams illustrating an example of a hardware configuration of the drawing recognition device according to the first embodiment. FIG. 2 is a diagram illustrating a configuration example of a learning device according to the first embodiment. 4 is a flowchart for explaining the operation of the learning device according to the first embodiment. 16A and 16B are diagrams illustrating an example of a hardware configuration of the learning device 400 according to the first embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.

FIG. 1 is a diagram showing an example of a configuration of a drawing recognition apparatus 100 according to the first embodiment.
The drawing recognition device 100 is connected to an operation input device 200 and a display device 300 .
The operation input device 200 includes, for example, a keyboard 201 and a mouse 202 .
The display device 300 includes a display 301. The display 301 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 300 may be mounted on the operation input device 200.

The operation input device 200 accepts operations for the drawing recognition device 100 by a user of the drawing recognition device 100 .
For example, the user operates the keyboard 201 or the mouse 202 provided in the operation input device 200 to input information to be input to the drawing recognition device 100. Specifically, the user operates the keyboard 201 or the mouse 202 to display, for example, a list of image drawings 1 or a list of drawing characteristic information 2 on the display 301 of the display device 300, and selects the image drawing 1 and the drawing characteristic information 2.

Here, FIG. 2 is a diagram for explaining an example of an image of image drawing 1 in the first embodiment.
The image drawing 1 is, for example, drawing data obtained by digitizing a paper drawing on which a circuit diagram as shown in Fig. 2 is printed, using a scanner or the like. The image drawing 1 may also be, for example, electronic data created on a computer from a paper drawing on which a circuit diagram as shown in Fig. 2 is printed. The format of the image drawing 1 is, for example, any general image format such as jpg or tiff format.
In the first embodiment, as an example, the image drawing 1 is assumed to be drawing data obtained by digitizing a circuit diagram.

FIG. 3 is a diagram for explaining an image of an example of drawing characteristic information 2 in the first embodiment.
The drawing characteristic information 2 is information about the characteristics of the image drawing 1, which is set for each image drawing 1. Specifically, the drawing characteristic information 2 is information that associates information capable of identifying the image drawing 1, such as a name or ID, with the purpose of the image drawing 1, and a model in which the image drawing 1 is used. In the first embodiment, the drawing characteristic information 2 is information that associates information capable of identifying a circuit diagram, such as a name or ID, with the purpose of the circuit diagram, and a model in which the circuit diagram is used.
In this embodiment, the drawing characteristic information 2 is text data, but this is merely an example. The drawing characteristic information 2 may be information that associates a drawing with a characteristic of the drawing.

For example, a plurality of image drawings 1 are stored in advance in a drawing recording unit (not shown) provided at a location that can be referenced by the drawing recognition device 100 and the operation input device 200. Also, drawing characteristic information 2 is stored in advance in a relevance recording unit 106 (details of which will be described later).
The user, for example, operates the keyboard 201 or the mouse 202 to cause the drawing recognition device 100 to display on the display 301 a list of a plurality of image drawings 1 stored in the drawing recording unit and the drawing characteristic information 2 stored in the relevance recording unit 106. Then, the user, for example, operates the keyboard 201 or the mouse 202 to select a desired image drawing 1 from the displayed list of image drawings 1, and selects the drawing characteristic information 2 corresponding to the selected image drawing 1.
The operation input device 200 receives the image drawing 1 selected by the user and the drawing characteristic information 2. The image drawing 1 and the drawing characteristic information 2 received by the operation input device 200 are input to the drawing recognition device 100.

The drawing recognition device 100 performs drawing recognition of the image drawing 1 based on the image drawing 1 and drawing characteristic information 2 input from the operation input device 200 .
In addition, in embodiment 1, drawing recognition performed by the drawing recognition device 100 refers to recognizing information on one or more symbols (hereinafter referred to as "symbol information"), information on attributes associated with the symbols (hereinafter referred to as "attribute information"), information on connecting lines between the symbols (hereinafter referred to as "connecting line information"), and the correspondence between symbols, attributes, and connecting lines on the image drawing 1.
The symbol information includes, for example, the number of symbols, position information of each symbol, the type of each symbol, and information on the accuracy of each symbol. The attribute information includes, for example, the number of attributes, position information of each attribute, the type of each attribute, and information on the accuracy of each attribute. The connection line information includes, for example, the number of connection lines, position information of each connection line, the type of each connection line, and information on the accuracy of each connection line. Note that in the first embodiment, "accuracy" refers to the probability of accuracy. For example, the accuracy of a symbol refers to the probability that it is a symbol, the accuracy of an attribute refers to the probability that it is an attribute, and the accuracy of a connection line refers to the probability that it is a connection line.

In the first embodiment, a symbol in the image drawing 1 is, for example, a part shown on the image drawing 1. Also, an attribute in the image drawing 1 is, for example, a character associated with a symbol and used to identify the symbol. Specifically, in the first embodiment, an attribute is a character indicating the tool number of a part or the name of a part. Also, a connection line in the image drawing 1 is, for example, a line connecting symbols. Note that in the first embodiment, the connection line also includes an intersection.

The drawing recognition device 100 comprises a drawing acquisition unit 101, a recognition unit 102, an inference unit storage unit 103, a judgment unit 104, a part recording unit 105, an association degree recording unit 106, a generation unit 107, a comparison unit 108, a matching data recording unit 109, an unmatched data recording unit 110, and a drawing display unit 111.
The inference unit storage unit 103 stores a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133.
The first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are models that have been trained to input the image drawing 1 and the drawing characteristic information 2, and output information relating to the result of drawing recognition of the image drawing 1. In the following first embodiment, the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are also collectively referred to simply as "machine learning models."

More specifically, in the first embodiment, the first machine learning model 131 is a machine learning model that receives an image drawing 1 and drawing characteristic information 2 as input, and outputs a result of recognizing symbol information on the image drawing 1 (hereinafter referred to as a "symbol recognition result"). That is, in the first embodiment, the first machine learning model 131 is a machine learning model for recognizing symbol information.
The second machine learning model 132 is a machine learning model that receives as input the image drawing 1 and drawing characteristic information 2, and outputs a result of recognizing attribute information associated with a symbol on the image drawing 1 (hereinafter referred to as an "attribute recognition result"). That is, in the first embodiment, the second machine learning model 132 is a machine learning model for recognizing attribute information.
The third machine learning model 133 is a machine learning model that outputs a result of recognizing connection line information between symbols on the image drawing 1 (hereinafter referred to as a "connection line recognition result"). That is, in the first embodiment, the third machine learning model 133 is a machine learning model for recognizing connection line information.

The machine learning model is generated by the learning device 400 (see FIG. 14 described later) performing machine learning learning. Details of the learning device 400 will be described later.

The drawing acquisition unit 101 acquires the image drawing 1 and drawing characteristic information 2 inputted by the user via the operation input device 200 .
The drawing acquisition unit 101 performs a predetermined process on the image drawing 1 to generate inference data.
As described above, the drawing recognition device 100 inputs the image drawing 1 into a machine learning model to obtain a drawing recognition result for the image drawing 1. Here, there is a limit to the image size that is input to the machine learning model. Specifically, for example, if the image drawing 1 is a full high definition (FHD) image size of 1920×1080 pixels, this exceeds the upper limit of the image size that can be input to the machine learning model.
Therefore, the drawing acquisition unit 101 divides the acquired image drawing 1 into data of, for example, 224×224 pixels each, and the divided data is used as inference data. The intervals at which the image drawing 1 is divided may be equal intervals of 224×224 pixels in the top, bottom, left, and right directions, and overlapping parts may be included when the image drawing 1 is divided.
In addition, for example, if the acquired image drawing 1 is within the size limit that allows it to be used as input to a machine learning model, the drawing acquisition unit 101 does not need to perform the processing described above, and can simply use the acquired image drawing 1 as inference data.
The drawing acquisition unit 101 outputs the inference data generated based on the image drawing 1 and the acquired drawing characteristic information 2 to the recognition unit 102 .

Note that, although it is assumed here that the division of image drawing 1 as described above is performed by drawing acquisition unit 101, this is merely one example. For example, the division of image drawing 1 may be performed by recognition unit 102, which will be described later. In this case, drawing acquisition unit 101 outputs the acquired image drawing 1 as is to recognition unit 102, which will be described later, and when performing inference using a machine learning model, recognition unit 102, which will be described later, divides image drawing 1 as necessary.

The recognition unit 102 acquires a symbol recognition result, an attribute recognition result, or a connection line recognition result by performing inference using the first machine learning model 131, the second machine learning model 132, or the third machine learning model 133 stored in the inference device storage unit 103, based on the inference data and the drawing characteristic information 2 output from the drawing acquisition unit 101. In this way, the recognition unit 102 recognizes symbol information, attribute information, or connection line information.
The recognition unit 102 outputs the inference result, in other words, the symbol recognition result, the attribute recognition result, or the connection line recognition result, to the determination unit 104 .

FIG. 4 is a diagram showing a detailed configuration example of the recognizing unit 102 included in the drawing recognizing apparatus 100 according to the first embodiment.
For ease of explanation, FIG. 4 shows only the drawing acquisition unit 101, the recognition unit 102, and the inference unit storage unit 103 of the configuration example of the drawing recognition device 100 shown in FIG.
As shown in FIG. 4, the recognition unit 102 includes a symbol recognition unit 1021 , an attribute information recognition unit 1022 , and a connection line recognition unit 1023 .

The symbol recognition unit 1021 acquires a symbol recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the first machine learning model 131.
Specifically, the symbol recognition unit 1021 analyzes the inference data and extracts image features from the inference data. Also, the symbol recognition unit 1021 analyzes the drawing characteristic information 2 and extracts features from the drawing characteristic information 2. The symbol recognition unit 1021 extracts features from the drawing characteristic information 2 as numerical values, such as "0" for "home appliance,""1" for "distribution board," and "2" for "car."
The symbol recognition unit 1021 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 to the first machine learning model 131 to obtain a symbol recognition result.
In embodiment 1, the symbol recognition unit 1021 obtains, based on the inference data and drawing characteristic information 2, and the first machine learning model 131, the number of symbols in the inference data, position information of each symbol, the type of each symbol, and the accuracy of each symbol, as symbol recognition results.
The symbol recognition unit 1021 outputs the obtained symbol recognition result to the determination unit 104 .

The attribute information recognition unit 1022 acquires an attribute recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the second machine learning model 132.
Specifically, the attribute information recognition unit 1022 analyzes the inference data and extracts image feature quantities from the inference data. The attribute information recognition unit 1022 also analyzes the drawing characteristic information 2 and extracts feature quantities from the drawing characteristic information 2.
The attribute information recognition unit 1022 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 to the second machine learning model 132 to obtain attribute recognition results.
In embodiment 1, the attribute information recognition unit 1022 obtains, based on the inference data and drawing characteristic information 2, and the second machine learning model 132, the number of attributes in the inference data, the position information of each attribute, the type of each attribute, and the accuracy of each attribute, as attribute recognition results.
The attribute information recognition unit 1022 outputs the acquired attribute recognition result to the determination unit 104 .

The connection line recognition unit 1023 acquires a connection line recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 and the third machine learning model 133.
Specifically, the connection line recognition unit 1023 analyzes the inference data and extracts image features from the inference data. The connection line recognition unit 1023 also analyzes the drawing characteristic information 2 and extracts features from the drawing characteristic information 2.
The connection line recognition unit 1023 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 to the third machine learning model 133 to obtain a connection line recognition result.
In embodiment 1, based on the inference data and drawing characteristic information 2, and the third machine learning model 133, the connection line recognition unit 1023 acquires, as connection line recognition results, the number of connection lines in the inference data, position information of each connection line, the type of each connection line, and the accuracy of each connection line, etc.
The connection line recognition unit 1023 outputs the acquired connection line recognition result to the determination unit 104 .

The inference performed by the recognition unit 102 based on the inference data, the drawing characteristic information 2, and the machine learning model will be specifically described with reference to FIG.
5A and 5B are diagrams showing examples of the symbol recognition result acquired by the symbol recognition unit 1021, the attribute recognition result acquired by the attribute information recognition unit 1022, and the connection line recognition result acquired by the connection line recognition unit 1023 in the first embodiment. Fig. 5A shows an image in which the symbol recognition result, the attribute recognition result, and the connection line recognition result are mapped onto the image drawing 1 acquired by the drawing acquisition unit 101, and Fig. 5B shows an image in which the contents of the symbol recognition result, the attribute recognition result, and the connection line recognition result are displayed in text form.

Note that Figure 5A and Figure 5B are linked. Specifically, for example, the symbols indicated by 511, 512, and 513 in Figure 5A correspond to the symbol No. 1 "relay (a)", the symbol No. 2 "relay (b)", and the symbol No. 3 "fuse" in Figure 5B, respectively. Also, for example, the attributes indicated by 521, 522, 523, and 524 in Figure 5A correspond to the attribute No. 1 "30X004", the attribute No. 2 "30X003", the attribute No. 3 "FL3", and the attribute No. 5 "62LCTX", respectively, in Figure 5B. Also, for example, the connection lines indicated by 531, 532, 533, and 534 in Figure 5A correspond to the connection lines No. 2, 5, 6, and 7 in Figure 5B.

The symbols, attributes, and connection lines on the image drawing 1, in other words, the inference data, are recognized in rectangular areas on the image drawing 1.
Figure 5A shows an image in which a rectangular area indicating a recognized symbol, a rectangular area indicating a recognized attribute, and a rectangular area indicating a recognized connecting line are mapped onto image drawing 1 acquired by drawing acquisition unit 101.
In Fig. 5B, for example, the symbol recognition result includes the symbol name and the position information of the recognized symbol. The position of the symbol is represented by the minimum and maximum values of the X coordinate and the minimum and maximum values of the Y coordinate of a rectangular area showing the symbol on the image drawing 1. Note that each coordinate shown in Fig. 5B is the coordinate with the upper left corner of the drawing as the origin in Fig. 5A.
5B, for example, the attribute recognition result includes the attribute, in other words, the character, and the position information of the recognized character. The position of the character is represented by the minimum and maximum values of the X coordinate and the minimum and maximum values of the Y coordinate of a rectangular area showing the character on the image drawing 1.
5B, for example, the connection line recognition result includes the type of the connection line and position information of the recognized connection line. The position of the connection line is represented by the minimum and maximum values of the X coordinate and the minimum and maximum values of the Y coordinate of a rectangular area showing the connection line on the image drawing 1.

Note that the contents of the symbol recognition results, attribute recognition results, and connection line recognition results shown in Figures 5A and 5B are merely examples. The symbol recognition results, attribute recognition results, and connection line recognition results may include information other than the information shown in Figures 5A and 5B, such as an ID.

Returning to the explanation of FIG.
The inference unit storage unit 103 stores a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133.

The judgment unit 104 judges the correspondence between the symbols, attributes, and other symbols in the inference data based on the symbol recognition results, attribute recognition results, and connection line recognition results output by the recognition unit 102, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized.
Furthermore, the determination unit 104 determines the correspondence between the symbols, attributes, and other symbols in the inference data based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and re-determines whether the symbols, attributes, and connections between the symbols have been correctly recognized. Details of the mismatch data stored in the mismatch data recording unit 110 will be described later.

The part information stored in the part recording unit 105 and the relevance information stored in the relevance recording unit 106 will be explained.

FIG. 6 is a diagram for explaining an example of an image of the contents of part information stored in part recording unit 105 in the first embodiment.
The part information is information relating to the part shown in the image drawing 1.
Specifically, the part information is, for example, part list data including the tool number and part code of the part, as shown in FIG.
In the first embodiment, the parts are shown as symbols on the image drawing 1.

FIG. 7 is a diagram for explaining an example of an image of the contents of the relevance information stored in relevance recording unit 106 in the first embodiment.
The relevance information is generated in advance by, for example, a user and recorded in the relevance recording unit 106. The user generates the relevance information in advance, for example, based on the image drawing 1 and the drawing characteristic information 2. In the first embodiment, the relevance information includes, as shown in Fig. 7, symbol-attribute information (indicated by 701 in Fig. 7), symbol-connection information (indicated by 702 in Fig. 7), usage relevance information (indicated by 703 and 704 in Fig. 7), and model relevance information (indicated by 705 and 706 in Fig. 7).

The symbol-attribute information is information that defines, for each symbol, the degree of association between the symbol and an attribute related to the symbol. In the symbol-attribute information, for each symbol, an associated attribute is defined in association with the number of times that attribute is shown together with the symbol on the image drawing 1. For example, a user generates symbol-attribute information in advance based on multiple image drawings 1 to be used in the learning phase. The number of times defined in the symbol-attribute information indicates the number of times that the attribute associated with the symbol appears on the image drawing 1 used to generate the symbol-attribute information, in other words, on the image drawing 1 to be used in the learning phase.
In the symbol-attribute information, an ID is assigned to each of the information associating the symbol, the attribute, and the occurrence frequency of the attribute associated with the symbol.

The symbol-connection information is information that defines, for each symbol, the degree of association between that symbol and other symbols connected to that symbol. In the symbol-connection information , for each symbol, a symbol that is a destination of connection (hereinafter referred to as a "destination symbol") and the number of times that the destination symbol is shown connected to the symbol on the image drawing 1 are defined in association with each other. For example, the user generates the symbol-connection information in advance based on multiple image drawings 1 to be used in the learning phase. The number of times defined in the symbol-connection information indicates the number of times that the destination symbol appears on the image drawing 1 used to generate the symbol-connection information, in other words, on the image drawing 1 to be used in the learning phase.
In the symbol-connection information, an ID is assigned to each of the pieces of information in which a symbol, a connected symbol, and the frequency of appearance of the connected symbol are associated with each other.

The usage association information includes symbol-attribute usage association information (indicated by 703 in FIG. 7) and symbol-connection usage association information (indicated by 704 in FIG. 7).
The symbol-attribute usage association information is information that defines the association between a symbol and an attribute for each purpose for which the image drawing 1 is used. More specifically, the symbol-attribute usage association information is information that classifies, for each purpose, the frequency of appearance of combinations of symbols and attributes defined in the symbol-attribute information in the image drawing 1. In the symbol-attribute usage association information, for each purpose, information indicating associated symbols and attributes is associated with the number of times the combination of the associated symbols and attributes appears.
The information indicating the associated symbol and attribute may be, for example, an ID in the symbol-attribute information. That is, the symbol-attribute use association information shown in 703 in Fig. 7 indicates that the combination of "relay (a)" and "30X004" appears 22 times in image drawing 1 used for the purpose of "home appliances."
For example, the user generates symbol-attribute use relevance information in advance based on multiple image drawings 1 to be used in the learning phase and the generated symbol-attribute information. The user can specify the use of the image drawings 1 based on the drawing characteristic information 2 as shown in FIG.

The symbol-connection usage association information is information that defines the association between a symbol and a destination symbol for each purpose for which the image drawing 1 is used. More specifically, the symbol-connection usage association information is information that classifies, for each purpose, the frequency of occurrence of a combination of a symbol and a destination symbol defined in the symbol-connection information in the image drawing 1. In the symbol-connection usage association information, for each purpose, information indicating an associated symbol and a destination symbol is associated with the number of times the combination of the associated symbol and the destination symbol occurs.
The information indicating the associated symbol and the connected symbol may be, for example, an ID in the symbol-connection information. That is, the symbol-connection use association information shown in 704 in Fig. 7 indicates that the combination of "relay (a)" and "resistor" appears five times in image drawing 1 used for the purpose of "home appliances."
For example, the user generates symbol-connection usage relevance information in advance based on a plurality of image drawings 1 used in the learning phase and the generated symbol-connection information.

The model relevance information includes symbol-attribute model relevance information (indicated by 705 in FIG. 7) and symbol-connection model relevance information (indicated by 706 in FIG. 7).
The symbol-attribute model relevance information is information that defines the relevance between symbols and attributes for each model of equipment in which the image drawing 1 is used. More specifically, the model relevance information is information that classifies, for each model, the frequency of appearance of combinations of symbols and attributes defined in the symbol-attribute information in the image drawing 1. In the symbol-attribute model relevance information, for each model, information indicating associated symbols and attributes is associated with the number of times the combination of the associated symbol and attribute appears.
The information indicating the associated symbol and attribute is, for example, an ID in the symbol-attribute information. That is, in the symbol-attribute model association information shown in 705 in Fig. 7, it means that the combination of "relay (a)" and "30X004" appears 18 times in image drawing 1 used for model "N-1001A".
For example, the user generates symbol-attribute model relevance information in advance based on multiple image drawings 1 used in the learning phase and the generated symbol-attribute information. The user can identify the model of the image drawing 1 based on the drawing characteristic information 2 as shown in FIG.

The symbol-connection model relevance information is information that defines the relevance between a symbol and a destination symbol for each model of device in which the image drawing 1 is used. More specifically, the symbol-connection model relevance information is information that classifies, for each model, the frequency of appearance of combinations of symbols and destination symbols defined in the symbol-connection information in the image drawing 1. In the symbol-connection model relevance information, for each application, information indicating associated symbols and destination symbols is associated with the number of times that the combination of the associated symbol and destination symbol appears.
The information indicating the associated symbol and the connected symbol is, for example, an ID in the symbol-connection information. That is, the symbol-connection model association information shown in 706 in Fig. 7 indicates that the combination of "relay (a)" and "resistor" appears 15 times in image drawing 1 used for model "N-1001A".
For example, the user generates symbol-connection model relevance information in advance based on a plurality of image drawings 1 used in the learning phase and the generated symbol-connection information.

In the relevance information, the greater the number of times that something is associated, the higher the relevance.
7, the relevance information includes symbol-attribute information, symbol-connection information, use relevance information, and model relevance information, but it is not essential that the relevance information includes all of the symbol-attribute information, symbol-connection information, use relevance information, and model relevance information. The relevance information may include at least one of the symbol-attribute information, symbol-connection information, use relevance information, and model relevance information.
Furthermore, the relevance information is not limited to symbol-attribute information, symbol-connection information, use relevance information, or model relevance information, but may be information about the relevance of symbols, attributes, or connection lines shown on the image drawing 1. For example, the relevance information may include country/region relevance information that classifies the frequency of appearance of combinations of symbols and attributes, or combinations of symbols and connection destination symbols in the image drawing 1, by country or region where the image drawing 1 is mainly used.

Returning to the explanation of the determination unit 104, the determination by the determination unit 104 as to whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized will be specifically described with reference to FIG.
FIG. 8 is an explanatory diagram showing an image of a specific example of the process in which the judgment unit 104 in embodiment 1 comprehensively judges whether the symbols, attributes, and connections between symbols in the inference data have been correctly recognized.
It should be noted that, for example, the symbol recognition result, attribute recognition result, and connection line recognition result output from the recognition unit 102 have contents as shown in Fig. 5, and the contents of the recognition result shown in Fig. 8 are the same as the contents of the recognition result shown in Fig. 5. Also, for example, it should be noted that the parts information stored in the parts recording unit 105 has contents as shown in Fig. 6, and the contents of the parts information shown in Fig. 8 are the same as the contents of the parts information shown in Fig. 6.
In the following, as an example, the judgment process by the judgment unit 104 will be explained using an example in which the judgment unit 104 judges whether the connection relationship between the symbols “relay (a)” and “relay (b)” has been correctly recognized based on the symbol recognition result, attribute recognition result, and connection line recognition result output from the recognition unit 102.

The determination unit 104 first refers to the part recording unit 105, compares the part information with the attribute recognition result, and determines whether or not an attribute included in the attribute recognition result matches the tool number defined in the part information. If an attribute included in the attribute recognition result does not match the tool number defined in the part information, the determination unit 104 discards the attribute from the attribute recognition result.
Here, for convenience, it is assumed that "30X004,""30X003,""FL3,""FL4," and "6LCTX57" included in the attribute recognition results all match the tool numbers defined in the part information.

Next, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, and infers the association between the symbol, the attribute, and the connection line based on the coordinates indicating the position of the symbol, the coordinates indicating the position of the attribute, and the coordinates indicating the position of the connection line from each recognition result, and tentatively determines the correspondence between the symbol, the attribute, and the connection line.
Specifically, for example, the determining unit 104 calculates the distance between the symbol, the attribute, and the connection line, and infers the association based on the calculated distance.
For example, the determination unit 104 calculates the distance between "relay (a)" and "30X004" based on the coordinates indicating the position of "relay (a)" No. 1 in the symbol recognition result and the coordinates indicating the position of "30X004" No. 1 in the attribute recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (a)" and "30X004" are associated with each other. Note that in the first embodiment, the determination unit 104 calculates the distance between the symbol and the attribute, or the distance between the symbol and the connection line, for example, by calculating the center position of a rectangle from the position information in the recognition result and obtaining a difference vector of the center position. For example, the determination unit 104 may determine whether the calculated distance is close or not by comparing the calculated distance with a preset threshold value (hereinafter referred to as "distance determination threshold value"). For example, the determination unit 104 determines that the distance is close if the calculated distance is equal to or less than the distance determination threshold value.
The determining unit 104 may determine, for example, from the position information, whether the rectangles overlap, and determine that they are likely to overlap.

Also, for example, the determination unit 104 calculates the distance between "relay (a)" and the connection line No. 2 based on the coordinates indicating the position of "relay (a)" No. 1 in the symbol recognition result and the coordinates indicating the position of the connection line No. 2 in the connection line recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (a)" and the connection line No. 2 are associated with each other.
Also, for example, the determination unit 104 determines the distance between the No. 2 connection line and the No. 5 connection line, and the distance between the No. 5 connection line and the No. 6 connection line, based on the coordinates indicating the position of the No. 2 connection line, the coordinates indicating the position of the No. 5 connection line, and the coordinates indicating the position of the No. 6 connection line in the connection line recognition result. Since the calculated distances are close to each other, the determination unit 104 infers that the No. 2 connection line, the No. 5 connection line, and the No. 6 connection line are associated with each other.

Furthermore, for example, the determination unit 104 calculates the distance between "relay (b)" and the connection line No. 7 based on the coordinates indicating the position of "relay (b)" No. 2 in the symbol recognition result and the coordinates indicating the position of the connection line No. 7 in the connection line recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (b)" and the connection line No. 7 are associated with each other.
Also, for example, the determination unit 104 calculates the distance between "relay (b)" and "30X003" based on the coordinates indicating the position of "relay (b)" No. 2 in the symbol recognition result and the coordinates indicating the position of "30X003" No. 2 in the attribute recognition result. Since the calculated distance is close, the determination unit 104 infers that "relay (b)" and "30X003" are associated with each other.

From the results of the above estimation, the determination unit 104 provisionally determines that "relay (a)" is "30X004", "relay (b)" is "30X003", and "relay (a)" and "relay (b)" are connected by connection lines No. 2, 5, 6, and 7.

When the correspondence relationships between the symbols, attributes, and connection lines are provisionally determined, the determination unit 104 refers to the relevance information stored in the relevance recording unit 106 and determines whether the correspondence relationships between the symbols, attributes, and connection lines have been correctly determined based on the provisionally determined correspondence relationships. Specifically, the determination unit 104 determines whether the provisionally determined correspondence relationships are correct by comparing the relevance information stored in the relevance recording unit 106 with a preset threshold value.
For example, here, as described above, the judgment unit 104 provisionally judges that the relay (a) and the relay (b) are connected. Therefore, the judgment unit 104, for example, refers to the symbol-connection information of the relevance information, and compares the number of times that the relay (a) and the relay (b) are associated with each other with a preset threshold (hereinafter referred to as the "threshold for symbol connection judgment"). For example, if the number of times defined in the symbol-connection information exceeds the threshold for symbol connection judgment, the judgment unit 104 judges that the provisional judgment that the relay (a) and the relay (b) are connected is correct. Here, for example, if the threshold for symbol connection judgment is "10", the number of times that the relay (a) and the relay (b) are associated with each other in the symbol-connection information is "25" (see 702 in FIG. 7), so the judgment unit 104 judges that the relay (a) and the relay (b) are connected. In other words, the judgment unit 104 judges that the relay (a) and the relay (b) are recognized in a correct correspondence relationship.

The determination unit 104 may delete information on the provisional determination that relay (a) and relay (b) are connected from each recognition result output from the recognition unit 102 when the number of times defined in the relevance information does not exceed a preset threshold, for example, when the number of times associated with relay (a) and relay (b) does not exceed the symbol connection determination threshold. Here, the determination unit 104 may delete, for example, the recognition results of "relay (a)", "relay (b)", "30X004", "30X003", and connection lines No. 2, 5, 6, and 7. Note that this is merely an example, and the determination unit 104 may not delete the provisionally determined information from the recognition result even when the number of times defined in the relevance information does not exceed a preset threshold.

For example, when the degree of association between a symbol and a destination symbol defined in the symbol-connection usage association information (see 704 in FIG. 7) is high but no association based on the symbol-connection information (see 702 in FIG. 7) is found, the determination unit 104 may associate the symbol with the destination symbol that is considered to have a high degree of association based on the symbol-connection usage association information.
As a specific example, suppose that the symbol recognition results show that symbols A and B are recognized, but not symbol C. Symbols A and C are symbols that are preferentially used in image drawing 1 used for the purpose of "home appliances," and symbol B is a symbol that is preferentially used in image drawing 1 used for the purpose of "switchboard."
In addition, in the association degree information, it is assumed that the symbol-connection information defines that the association degree between the A symbol and the C symbol is high, and the association degree between the A symbol and the B symbol is low.
In this case, the determination unit 104 provisionally determines that the A symbol and the B symbol are connected based on the recognition result output from the recognition unit 102, but as a result of referring to the relevance information, the determination unit 104 re-determines that the A symbol and the B symbol are not highly related. On the other hand, the determination unit 104 may determine that the A symbol and the C symbol are highly related, for example, in the application of "home appliances", based on the relevance information. In this case, the determination unit 104 may replace the C symbol with the B symbol in the recognition result, for example.

In addition, the determination unit 104 may associate the symbols, attributes, and connecting lines based on the relevance information when the determination unit 104 is unable to provisionally determine the correspondence relationship from the recognition result, for example. As a specific example, the determination unit 104 calculates the distance between "Fuse" and "FL3" based on the coordinates indicating the position of "Fuse" No. 3 in the symbol recognition result and the coordinates indicating the position of "FL3" No. 3 in the attribute recognition result. In addition, the determination unit 104 calculates the distance between "Fuse" and "62LCTX" based on the coordinates indicating the position of "Fuse" No. 3 in the symbol recognition result and the coordinates indicating the position of "62LCTX" No. 5 in the attribute recognition result. As a result, the distance between "Fuse" and "FL3" and the distance between "Fuse" and "62LCTX" are the same, and it is not possible to determine which of "FL3" and "62LCTX" is associated with "Fuse". In this case, for example, the determination unit 104 may perform the association based on the relevance information, depending on which combination has a higher relevance, "Fuse" and "62LCTX" or "FL3" and "62LCTX", in other words, which combination appears more frequently.

In this way, the judgment unit 104 judges the correspondence between symbols, attributes, and other symbols in the inference data, taking into account, for example, the association between symbols and attributes, the association between symbols and connected symbols, the purpose of image drawing 1, or the model in which image drawing 1 is used, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized.
Furthermore, for symbols or attributes that could not be associated based on the recognition result, the determining unit 104 can complement the association based on the association degree information.

The determination unit 104 outputs the determination result as to whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized to the generation unit 107.

FIG. 9 is a diagram for explaining an image of an example of a determination result output by determining unit 104 in the first embodiment.
The judgment result is information that is set for each symbol, for example as shown in Fig. 9, in which the symbol, the position of the symbol, the attribute associated with the symbol, the position of the attribute, the symbol's destination symbol, and the position of the destination symbol are associated with each other (hereinafter referred to as "associated judgment result information"). If there are multiple destination symbols, all of the multiple destination symbols are associated with the associated judgment result information. In addition, in the associated judgment result information, the position of the symbol, the position of the attribute, and the position of the destination symbol are represented by, for example, the coordinates on the image drawing 1 of the center of the rectangle in which the symbol, the attribute, and the destination symbol are recognized, respectively.

When the determination unit 104 receives a notification from the control unit (not shown) that the determination end condition has not been satisfied, the determination unit 104 re-determines whether the symbols, attributes, and connections between symbols in the inference data have been correctly recognized, based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106. The re-determination performed by the determination unit 104 will be described later.

The generating unit 107 generates generated data equivalent to the image drawing 1 (hereinafter referred to as "determined image drawing data") from the determination result output by the determining unit 104. More specifically, the generating unit 107 restores the image drawing 1 based on each piece of related determination result information set in the determination result output by the determining unit 104, and generates the post-determined image drawing data. In the post-determined image drawing data, the symbols, attributes, and connection destination symbols associated in each piece of related determination result information are shown at the positions associated in each piece of related determination result information.
The generating unit 107 outputs the generated post-determination image drawing data to the comparing unit 108. The generating unit 107 outputs the determination result output by the determining unit 104 to the comparing unit 108 together with the post-determination image drawing data.

The comparison unit 108 compares the post-judgment image drawing output by the generation unit 107 with the image drawing 1 acquired by the drawing acquisition unit 101. The comparison unit 108 may acquire the image drawing 1 from the drawing acquisition unit 101 via the recognition unit 102, the judgment unit 104, and the generation unit 107.

The comparison between the post-determination image drawing and image drawing 1 by the comparison unit 108 will be specifically described with reference to FIG.
The comparison unit 108 compares the post-determination image drawing with the image drawing 1 for each symbol in the post-determination image drawing and each attribute associated with the symbol. Note that the symbol and the attribute are associated with each other by the related determination result information in the determination result output by the determination unit 104.
Specifically, the comparison unit 108 compares, for example, the post-determination image drawing with the image drawing 1, and determines whether or not the symbols on the post-determination image drawing and the attributes associated with the symbols are also present at the same positions on the image drawing 1. Note that, at this time, the comparison unit 108 mainly compares the symbols and attributes, and excludes the connection lines.

The comparison unit 108 may determine the attribute associated with the symbol from the determination result output by the determination unit 104. As a specific example, if the determination result output by the determination unit 104 is as shown in Fig. 9, the comparison unit 108 determines whether or not "relay (a)" at the position (x, y) = (30, 18) associated with the characters "30X004" at the position (x, y) = (31, 10) exists on the image drawing 1.
In addition, the comparison unit 108 does not necessarily require that the symbol on the judged image drawing and the attribute associated with the symbol are in the same position on the image drawing 1. For example, the comparison unit 108 extracts a range (first range) set in advance around the first reference point, with respect to the judged image drawing, as a reference point (first reference point). The first range is usually a rectangular area in which the symbol is recognized by the recognition unit 102. In addition, the comparison unit 108 extracts a range (second range) set in advance around the second reference point, with respect to the judged image drawing, as a reference point (second reference point). The second range is usually a rectangular area in which the attribute is recognized by the recognition unit 102. In addition, the comparison unit 108 extracts a range (third range) set in advance around the third reference point, with respect to the image drawing 1, as a reference point (third reference point). The third range is the same size as the first range. The comparison unit 108 also sets a position on the image drawing 1 that corresponds to the position of the attribute associated with the symbol on the determined image drawing as a reference point (fourth reference point), and extracts a preset range (fourth range) centered on the fourth reference point. The fourth range is the same size as the second range.
Then, the comparison unit 108 compares, for example, the extracted post-judgment image drawing of the first range with the extracted image drawing 1 of the third range by template matching, and calculates the similarity (first similarity) between the post-judgment image drawing and the image drawing 1. Also, the comparison unit 108 compares, for example, the extracted post-judgment image drawing of the second range with the extracted image drawing 1 of the fourth range by template matching, and calculates the similarity (second similarity) between the post-judgment image drawing and the image drawing 1. The comparison unit 108 may compare the feature amounts of the extracted post-judgment image drawing and the image drawing 1, and calculate the similarity between the post-judgment image drawing and the image drawing 1.
The comparison unit 108 determines whether or not the symbol on the post-determination image drawing and the attribute associated with the symbol are also present in the same position on the image drawing 1, depending on whether the calculated first similarity and second similarity are equal to or greater than a predetermined threshold value (hereinafter referred to as the "similarity determination threshold value").

For example, when the calculated first similarity and second similarity are both equal to or greater than the similarity judgment threshold, the comparison unit 108 judges that the symbol on the post-judgment image drawing and the attribute associated with the symbol are also present at the same position on the image drawing 1. In other words, the comparison unit 108 judges that the symbol on the post-judgment image drawing and the attribute associated with the symbol match the symbol on the image drawing 1 and the attribute associated with the symbol. In this case, the comparison unit 108 records the related judgment result information regarding the symbol judged to match as match data in the match data recording unit 109.

On the other hand, when either or both of the calculated first similarity or second similarity is not equal to or greater than the similarity judgment threshold, the comparison unit 108 judges that the symbol on the judged image drawing and the attribute associated with the symbol do not exist at the same position on the image drawing 1. In other words, the comparison unit 108 judges that the symbol on the judged image drawing and the attribute associated with the symbol do not match the symbol on the image drawing 1 and the attribute associated with the symbol. In this case, the comparison unit 108 records the symbol recognition result for the symbol judged to be mismatched and the attribute recognition result for the attribute associated with the symbol judged to be mismatched in the mismatch data recording unit 110. For example, the comparison unit 108 may acquire the recognition result (see FIG. 5 ) from the recognition unit 102 via the judgment unit 104 and the generation unit 107, and extract the symbol recognition result for the symbol judged to be mismatched and the attribute recognition result for the attribute associated with the symbol judged to be mismatched from the acquired recognition result.
In addition, when extracting mismatched data, the comparison unit 108 also determines that the connection lines connected to the symbols determined to be mismatched are also mismatched, and records the connection line recognition results for those connection lines in the mismatched data recording unit 110.

The comparison unit 108 performs this comparison process on all related judgment result information included in the judgment result output from the judgment unit 104.

Here, Figure 10 is an explanatory diagram showing an image of the process in which the comparison unit 108 compares the post-judgment image drawing output by the generation unit 107 with the image drawing 1 acquired by the drawing acquisition unit 101 in embodiment 1.
For the sake of convenience, FIG. 10 is not related to the determination result in FIG. 9, but is merely a diagram for explaining an image of a comparison between the post-determination image diagram and image diagram 1.
For ease of explanation, Fig. 10 shows areas including matched data and mismatched data extracted from image drawing 1 and the post-determination image drawing, enlarged. In Fig. 10, the area extracted from image drawing 1 and enlarged is indicated by 1001, and the area extracted from the post-determination image drawing and enlarged is indicated by 1002.
For example, "coil" associated with "88TBL1C" shown in the area indicated by 1002 does not exist in the area indicated by 1001 at a position corresponding to the area indicated by 1002. Therefore, the comparison unit 108 records the symbol recognition result for "coil" and the attribute recognition result for "88TBL1C" in the mismatch data recording unit 110 as mismatch data.

Drawing recognition device 100 repeats re-determination by determination unit 104, generation of a post-determination image drawing by generation unit 107, and comparison processing by comparison unit 108 until a preset condition (hereinafter referred to as a "determination end condition") is satisfied. The "determination end condition" is, for example, "determination processing has been performed a preset number of times" or "there is no mismatching data and it has been determined that all data is matching." The "determination end condition" can be appropriately set by the user.
Specifically, for example, when the comparison unit 108 records match data or mismatch data, the control unit of the drawing recognition device 100 determines whether or not the judgment termination condition is satisfied. If it is determined that the judgment termination condition is not satisfied, the control unit notifies the judgment unit 104 that the judgment termination condition is not satisfied. When the judgment unit 104 receives the notification from the comparison unit 108 that the judgment termination condition is not satisfied, the judgment unit 104 re-determines whether or not the symbols, attributes, and connections between symbols in the inference data have been correctly recognized, based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106.
The mismatched data recorded in the mismatched data recording unit 110 is obtained by removing, from the recognition results, the symbol recognition results for the symbols determined to be matching data by the comparing unit 108, the attribute recognition results for the attributes determined to be matching data by the comparing unit 108, and the connection line recognition results for the connection lines determined to be matching data by the comparing unit 108. Therefore, when the determining unit 104 performs re-determination, the range of symbols, attributes, or connecting lines to be associated is narrowed, and the determining unit 104 can perform a more accurate determination.

When it is determined that the judgment end condition is satisfied, the control unit notifies the judgment unit 104 and the drawing display unit 111 that the judgment end condition is satisfied. When the judgment unit 104 receives notification from the control unit that the judgment end condition is satisfied, it ends the judgment process. In other words, the comparison unit 108 does not perform comparison processing thereafter. Note that here, it is assumed that the control unit of the drawing recognition device 100 determines whether or not the judgment end condition is satisfied and notifies based on the judgment result, but this is merely one example. For example, the judgment unit 104 or the comparison unit 108 may determine whether or not the judgment end condition is satisfied and notify based on the judgment result.

The matched data recording unit 109 records the matched data output by the comparison unit 108 .
The mismatched data recording section 110 records the mismatched data output by the comparison section 108 .

The drawing display unit 111 displays the matched data recorded in the matched data recording unit 109 on the display 301 of the display device 300 .
Specifically, the drawing display unit 111 converts the matched data into CAD data using, for example, a CAD application, and displays the CAD data on the display 301 of the display device 300 .
The drawing display unit 111 displays the matching data when the above-mentioned determination end condition is satisfied.
For example, as described above, when the determination end condition is satisfied, the control unit notifies the drawing display unit 111 that the determination end condition is satisfied. When the drawing display unit 111 is notified by the comparison unit 108 that the determination end condition is satisfied, the drawing display unit 111 refers to the matched data recording unit 109 and displays the matched data.

The operation of drawing recognition apparatus 100 according to the first embodiment will be described.
FIG. 11 is a flowchart for explaining the operation of drawing recognition apparatus 100 according to the first embodiment.

The drawing acquisition unit 101 acquires the image drawing 1 and the drawing characteristic information 2 inputted by the user via the operation input device 200 (step ST1101). The drawing acquisition unit 101 performs a predetermined process on the image drawing 1 to generate inference data.
The drawing acquisition unit 101 outputs the inference data generated based on the image drawing 1 and the acquired drawing characteristic information 2 to the recognition unit 102 .

The recognition unit 102 acquires a symbol recognition result, an attribute recognition result, or a connection line recognition result by performing inference using the first machine learning model 131, the second machine learning model 132, or the third machine learning model 133 stored in the inference device storage unit 103 based on the inference data and the drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101. As a result, the recognition unit 102 recognizes the symbol information, the attribute information, or the connection line information (step ST1102).
The recognition unit 102 outputs the inference result, in other words, the symbol recognition result, the attribute recognition result, or the connection line recognition result, to the determination unit 104 .

FIG. 12 is a flowchart for explaining details of the inference operation performed by the recognition unit 102 in step ST1102 in FIG.
The symbol recognition unit 1021 acquires a symbol recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG.
Specifically, the symbol recognition unit 1021 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 into the first machine learning model 131 to obtain the symbol recognition result (step ST1201).
The symbol recognition unit 1021 outputs the obtained symbol recognition result to the determination unit 104 .

The attribute information recognition unit 1022 acquires an attribute recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG. 11 and the second machine learning model 132.
Specifically, the attribute information recognition unit 1022 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 into the second machine learning model 132 to obtain the attribute recognition results (step ST1202).
The attribute information recognition unit 1022 outputs the acquired attribute recognition result to the determination unit 104 .

The connection line recognition unit 1023 acquires a connection line recognition result based on the inference data and drawing characteristic information 2 output from the drawing acquisition unit 101 in step ST1101 of FIG. 11 and the third machine learning model 133.
Specifically, the connection line recognition unit 1023 inputs the image features extracted from the inference data and the features extracted from the drawing characteristic information 2 into the third machine learning model 133 to obtain the connection line recognition results (step ST1203).
The connection line recognition unit 1023 outputs the acquired connection line recognition result to the determination unit 104 .

Returning to the explanation of the flowchart in FIG.
The judgment unit 104 judges the correspondence between the symbols, attributes, and other symbols in the inference data based on the symbol recognition result, attribute recognition result, or connection line recognition result output by the recognition unit 102, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between symbols have been correctly recognized.

The determination unit 104 first refers to the part recording unit 105, compares the part information with the attribute recognition result, and determines whether or not the attribute included in the attribute recognition result matches the fixture number defined in the part information (step ST1103). If the attribute included in the attribute recognition result does not match the fixture number defined in the part information, the determination unit 104 discards the attribute from the attribute recognition result.

Next, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, and infers the association between the symbol, the attribute, and the connection line based on the coordinates indicating the position of the symbol, the coordinates indicating the position of the attribute, and the coordinates indicating the position of the connection line from each recognition result, and tentatively determines the correspondence between the symbol, the attribute, and the connection line.
Then, the judgment unit 104 refers to the relevance information stored in the relevance recording unit 106 and judges, based on the provisionally determined correspondence, whether or not the correspondence between the symbol, the attribute, and the connection line has been correctly determined (step ST1104).
The determining unit 104 outputs to the generating unit 107 a determination result as to whether or not the correspondence between the symbols, attributes, and connection lines in the inference data has been correctly determined.

The generating section 107 generates post-determination image drawing data from the determination result output by the determining section 104 in step ST1104 (step ST1105).
The generating unit 107 outputs the generated post-determination image drawing data to the comparing unit 108 .

The comparison section 108 compares the post-determination image drawing output by the generation section 107 in step ST1105 with the image drawing 1 acquired by the drawing acquisition section 101 in step ST1101 (step ST1106).
Specifically, the comparison unit 108 compares, for example, the post-determination image drawing with the image drawing 1, and determines whether or not the symbols on the post-determination image drawing and the attributes associated with the symbols are also present at the same positions on the image drawing 1. Note that, at this time, the comparison unit 108 mainly compares the symbols and attributes, and excludes the connection lines from the comparison.

If the comparison unit 108 determines, as a result of the comparison judgment, that the symbol on the judged image drawing and the attribute associated with the symbol match the symbol on the image drawing 1 and the attribute associated with the symbol (if "YES" in step ST1106), it records the related judgment result information regarding the symbol that has been judged to match as match data in the match data recording unit 109 (step ST1107).

If the comparison unit 108 determines, as a result of the comparison, that the symbol on the post-determination image drawing and the attribute associated with that symbol are inconsistent with the symbol on the image drawing 1 and the attribute associated with that symbol (if "NO" in step ST1106), it records the symbol recognition result for the symbol determined to be inconsistent and the attribute recognition result for the attribute associated with the symbol determined to be inconsistent as inconsistent data in the inconsistent data recording unit 110 (step ST1108).

The control unit of drawing recognition apparatus 100 judges whether or not a judgment end condition is satisfied (step ST1109).
When it is determined that the judgment end condition is not satisfied (in the case of "NO" in step ST1109), the control unit notifies the judgment unit 104 that the judgment end condition is not satisfied. Then, the operation of the drawing recognition device 100 returns to step ST1103, and the judgment unit 104 re-determines whether or not the symbols, attributes, and connections between symbols in the inference data are correctly recognized based on the mismatch data stored in the mismatch data recording unit 110, the part information stored in the part recording unit 105, and the relevance information stored in the relevance recording unit 106.

When it is determined that the judgment end condition is satisfied ("YES" in step ST1109), the control unit notifies the judgment unit 104 and the drawing display unit 111 that the judgment end condition is satisfied. When the judgment unit 104 receives the notification from the control unit that the judgment end condition is satisfied, the judgment process is terminated. That is, the comparison unit 108 does not perform the comparison process thereafter.
When the control unit notifies the drawing display unit 111 in step ST1109 that the determination end condition has been satisfied, the drawing display unit 111 displays the matched data recorded in the matched data recording unit 109 on the display 301 of the display device 300 (step ST1110).

In this way, in the drawing recognition device 100, the recognition unit 102 can use the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 to recognize symbol information regarding one or more symbols contained in the image drawing 1, attribute information regarding attributes associated with the symbols, and connection line information regarding the connection lines between the symbols.
Furthermore, in the drawing recognition device 100, the judgment unit 104 compares the recognition result (symbol recognition result, attribute recognition result, or connection line recognition result) output from the recognition unit 102 with part information and relevance information to determine the correspondence between the symbols, attributes, and other symbols in the image drawing 1, and makes a comprehensive judgment as to whether the symbols, attributes, and connections between the symbols have been correctly recognized, thereby improving accuracy.
Furthermore, in the drawing recognition device 100, the comparison unit 108 compares the symbols and attributes associated with the symbols between the post-judgment image drawing data generated by the generation unit 107 and the image drawing 1, thereby detecting errors in the recognition of the symbols or attributes and enabling the selection of a recognition result with high accuracy.

In the above-described first embodiment, in drawing recognition apparatus 100, determination unit 104 compares the recognition result output from recognition unit 102 with both the part information stored in part recording unit 105 and the relevance information recorded in relevance recording unit 106, but this is merely an example. Determination unit 104 may compare only with either the part information or the relevance information, or may determine the recognition result without comparing with either the part information or the relevance information.
When no matching is performed with either the part information or the relevance information, the determination unit 104 determines the correspondence between the symbol, the attribute, and other symbols in the image drawing 1 based on the information on the symbol, the information on the attribute, and the information on the connection line recognized by the recognition unit 102. Specifically, for example, the determination unit 104 refers to the symbol recognition result, the attribute recognition result, or the connection line recognition result, calculates the distance between the symbol, the attribute, and the connection line from each recognition result, and determines the correspondence between the symbol, the attribute, and the connection line according to the calculated distance.
However, by comparing the recognition result with part information or relevance information, the determination unit 104 can determine the correspondence between symbols, attributes, and connecting lines with higher accuracy than when the recognition result is determined without comparing with either the part information or the relevance information.

Furthermore, an interface may be provided that allows a user to select the required recognition result or judgment result by referring to the symbol recognition result, attribute recognition result, or connection line recognition result output by the recognition unit 102 (see FIG. 5 ) or the judgment result output by the judgment unit 104 (see FIG. 9 ).
Specifically, for example, a display control unit (not shown) of the drawing recognition device 100 causes each recognition result output by the recognition unit 102 or the judgment result output by the judgment unit 104 to be displayed on the display 301 of the display device 300. The user understands the contents of each recognition result or judgment result by checking the display 301. For example, if there is a clearly erroneous symbol, attribute, connection line, etc., the user operates the keyboard 201 or mouse 202 of the operation input device 200 to input an instruction to delete the clearly erroneous symbol, attribute, connection line, etc. The control unit of the drawing recognition device 100 accepts the instruction input by the user and deletes information about the symbol, attribute, connection line, etc. from the recognition result or judgment result based on the accepted instruction.
With this configuration, the drawing recognition device 100 can select information related to symbols, attributes, connection lines, or the like that is required by the user.

In the above embodiment 1, the inference unit storage unit 103 is provided in the drawing recognition device 100, but this is merely an example. The inference unit storage unit 103 may be provided in a location outside the drawing recognition device 100 that can be referenced by the drawing recognition device 100. Specifically, for example, a single or multiple network storage devices (not shown) arranged on a communication network may have the function of the inference unit storage unit 103 and store the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133. The drawing recognition device 100 accesses the network storage device. As a result, the external network storage device can accumulate the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133, and a database can be constructed outside the drawing recognition device 100.

In addition, in the above embodiment 1, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 are provided in the drawing recognition device 100, but this is merely one example. The part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 may be provided in a location outside the drawing recognition device 100 that can be referenced by the drawing recognition device 100.

In the above embodiment 1, the drawing recognition device 100 is provided with the drawing acquisition unit 101, the recognition unit 102, the inference device storage unit 103, the judgment unit 104, the part recording unit 105, the relevance recording unit 106, the generation unit 107, the comparison unit 108, the matched data recording unit 109, the mismatched data recording unit 110, and the drawing display unit 111, but this is merely an example. The drawing recognition device 100 is required to have at least the drawing acquisition unit 101, the recognition unit 102, and the judgment unit 104. In this case, for example, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be provided in a device (a drawing generation device, not shown) external to the drawing recognition device 100, and the drawing generation system may be configured by the drawing recognition device 100 and the drawing generation device. The inference unit storage unit 103, the part recording unit 105, the matching data recording unit 109, and the mismatching data recording unit 110 are provided, for example, in a server that can be referenced by the drawing recognition device 100 and the drawing generation device.

The drawing recognition apparatus 100 according to the first embodiment is suitable for use in, for example, restoring CAD data from a paper drawing, or converting CAD data for a different CAD system.
Specifically, for example, a user inputs a printout created in a specific CAD system into the drawing recognition device 100 as an image drawing 1. The drawing recognition device 100 recognizes symbols, attributes, or connection lines present on the input image drawing 1, and records the matching data. By inputting the recorded matching data into the specific CAD system, the user can restore the printout to CAD data, and can reuse the CAD data for design revision or improvement, etc.
Furthermore, the user can convert the matched data into CAD data for use in another CAD system by, for example, separately providing a conversion program for converting the data into a data format for a different CAD system.
In this manner, the drawing recognition apparatus 100 according to the first embodiment can improve the accuracy of restoring CAD data in a CAD system by being used to restore CAD data in the CAD system, for example.

13A and 13B are diagrams illustrating an example of a hardware configuration of the drawing recognition apparatus 100 according to the first embodiment.
In the first embodiment, the functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 are realized by a processing circuit 1301. That is, the drawing recognition device 100 includes the processing circuit 1301 for recognizing symbols and the like on the input image drawing 1.
The processing circuit 1301 may be dedicated hardware as shown in FIG. 13A, or may be a CPU (Central Processing Unit) 1304 that executes a program stored in a memory 1305 as shown in FIG. 13B.

When the processing circuit 1301 is dedicated hardware, the processing circuit 1301 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

When the processing circuit 1301 is a CPU 1304, the functions of the drawing acquisition unit 101, the recognition unit 102, the judgment unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1305. The processing circuit 1301 executes the functions of the drawing acquisition unit 101, the recognition unit 102, the judgment unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 by reading and executing the program stored in the memory 1305. That is, the drawing recognition device 100 includes a memory 1305 for storing a program that, when executed by the processing circuit 1301, results in the execution of steps ST1101 to ST1110 in FIG. 11 and steps ST1201 to ST1203 in FIG. 12. In addition, it can be said that the program stored in memory 1305 causes the computer to execute the procedures or methods of the drawing acquisition unit 101, the recognition unit 102, the judgment unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111. Here, memory 1305 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

The functions of the drawing acquisition unit 101, the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the drawing acquisition unit 101 may be realized by a processing circuit 1301 as dedicated hardware, and the functions of the recognition unit 102, the determination unit 104, the generation unit 107, the comparison unit 108, and the drawing display unit 111 may be realized by the processing circuit 1301 reading out and executing a program stored in a memory 1305.
Also, for example, an HDD is used for the inference unit storage unit 103, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110. Note that this is just one example, and the inference unit storage unit 103, the part recording unit 105, the relevance recording unit 106, the matching data recording unit 109, and the mismatching data recording unit 110 may be configured with the memory 1305, an SSD (Solid State Drive), a DVD, or the like.
The drawing recognition apparatus 100 also includes an input interface device 1302 and an output interface device 1303 that perform wired or wireless communication with devices such as the operation input device 200 or the display device 300 .

In the first embodiment, the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133 are generated by the learning device 400 (see FIG. 14 described later) performing learning in machine learning. The learning device 400 will be described below.

FIG. 14 is a diagram illustrating an example of a configuration of a learning device 400 according to the first embodiment.
The learning device 400 is connected to the drawing recognition device 100 via a network.
The learning device 400 generates a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on the input image drawing 1 and drawing characteristic information 2. The image drawing 1 is input together with a teacher label. The learning device 400 trains a machine learning model composed of a neural network by so-called supervised learning. The teacher label is information indicating a symbol, an attribute, and a connection line on the image drawing 1. Regarding the teacher label, for example, the teacher label regarding a symbol is text data in which coordinates indicating the position of the symbol on the image drawing 1 are associated with information that can identify the symbol. The position of the symbol is represented, for example, by the coordinates on the image drawing 1 of the four corners of the rectangle when the symbol is surrounded by the rectangle. The information that can identify the symbol may be, for example, the name of the symbol. The teacher label regarding the attribute and the teacher label regarding the connection line are also text data in which coordinates indicating the position of the attribute or connection line on the image drawing 1 are associated with information that can identify the attribute or connection line, similar to the teacher label regarding the symbol. The training data is generated in advance for each image drawing 1, for example, by a user.

The learning device 400 stores the generated first machine learning model 131, second machine learning model 132, and third machine learning model 133 in the inference device storage unit 103 of the drawing recognition device 100. The learning device 400 also stores the input drawing characteristic information 2 in the relevance recording unit 106. Note that the relevance recording unit 106 stores relevance information (see FIG. 7) that has been generated in advance by the user. The user generates relevance information in advance based on the image drawing 1 and the drawing characteristic information 2. The learning device 400 stores the drawing characteristic information 2 together with the relevance information.
In addition, in order to simplify the explanation, FIG. 14 illustrates only the inference device storage unit 103 and the relevance recording unit 106 as components of the drawing recognition device 100, but the configuration example of the drawing recognition device 100 is the same as the configuration example of the drawing recognition device 100 described using FIG. 1.

In addition, as shown in FIG. 14, the learning device 400 is a separate device from the drawing recognition device 100 and is connected to the drawing recognition device 100, but this is merely an example. The learning device 400 may be mounted on the drawing recognition device 100.

The learning device 400 includes a learning data acquisition unit 401, a symbol model generation unit 402, an attribute information model generation unit 403, and a connection line model generation unit 404.

The learning data acquisition unit 401 performs data acquisition processing for learning.
Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 to which teacher labels have been assigned, and drawing characteristic information 2 corresponding to the image drawings 1. The teacher labels are, for example, text data as described above. Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 and text data for each image drawing 1. The plurality of image drawings 1 to which teacher labels have been assigned and the drawing characteristic information 2 are input by a user. For example, the learning device 400 is connected to the operation input device 200 (see FIG. 1), and the user inputs the plurality of image drawings 1 and the drawing characteristic information 2 from the operation input device 200.

Then, the learning data acquisition unit 401 generates learning data based on the acquired image drawing 1.
Specifically, the learning data acquisition unit 401 first divides the acquired image drawing 1 into data of, for example, 224×224 pixels each. The image drawing 1 may be divided at equal intervals of 224×224 pixels in the vertical and horizontal directions, and overlapping portions may be included when the image drawing 1 is divided. Note that the learning data acquisition unit 401 does not need to perform the above-described processing if, for example, the acquired image drawing 1 is within the size limit that can be used as an input for a machine learning model.
Furthermore, the learning data acquisition unit 401 analyzes the acquired drawing characteristic information 2 and extracts feature amounts from the drawing characteristic information 2. The learning data acquisition unit 401 extracts feature amounts from the drawing characteristic information 2 as numerical values, such as "0" for "home appliance", "1" for "distribution board", and "2" for "car".
The learning data acquisition unit 401 generates, as learning data, data including the divided image drawing 1 to which teacher labels have been added and feature amounts extracted from the drawing characteristic information 2.
The learning data acquisition unit 401 outputs the generated learning data to the symbol model generation unit 402 , the attribute information model generation unit 403 , and the connection line model generation unit 404 .

In this embodiment, the learning data acquisition unit 401 performs an analysis process on the drawing characteristic information 2, extracts features, and includes the features in the learning data, but this is merely an example. For example, the learning data acquisition unit 401 may use the divided image drawing 1 to which teacher labels are assigned and the drawing characteristic information 2 as learning data. In this case, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below each perform the above-mentioned analysis process on the drawing characteristic information 2 included in the learning data when generating a machine learning model.
Furthermore, the division of the image drawing 1 as described above may be performed by the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below. In this case, the learning data acquisition unit 401 includes the acquired image drawing 1 as it is in the learning data, and the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 described below each divide the image drawing 1 when generating a machine learning model.

Furthermore, the learning data acquisition unit 401 stores the acquired drawing characteristic information 2 in the relevance recording unit 106. As described above, the relevance recording unit 106 stores relevance information in advance.
In addition to the relevance information and the drawing characteristic information 2, the relevance recording unit 106 may store various information related to symbols, attributes, or connection lines.

The symbol model generation unit 402 learns to judge symbols based on the learning data output by the learning data acquisition unit 401, and generates a first machine learning model 131 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of symbols on the image drawing 1, the position information of each symbol, the type of each symbol, and the accuracy of each symbol as a symbol recognition result.
In the first embodiment, the symbol model generating unit 402 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as SSD (Single Shot Detector) or YOLO (You only Look Once) may be used.
The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit storage unit 103 .

The attribute information model generation unit 403 learns to judge attributes based on the learning data output by the learning data acquisition unit 401, and generates a second machine learning model 132 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of attributes on the image drawing 1, the position information of each attribute, the type of each attribute, and the accuracy of each attribute as attribute recognition results.
In the first embodiment, the attribute information model generating unit 403 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as VGSL (Variable-size Graph Specification Language) may be used.
The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit storage unit 103.

The connection line model generation unit 404 learns to determine connection lines based on the learning data output by the learning data acquisition unit 401, and generates a third machine learning model 133 that uses the image drawing 1 and drawing characteristic information 2 as input and outputs information regarding the number of connection lines on the image drawing 1, the position information of each connection line, the type of each connection line, and the accuracy of each connection line, etc. as a connection line recognition result.
In the first embodiment, the connection line model generating unit 404 performs learning related to object detection of a neural network using the learning data output by the learning data acquiring unit 401. That is, learning is performed by extracting image features, classifying them, and correcting coordinates. As a learning method, a well-known learning method such as SSD (Single Shot Detector) or YOLO (You only Look Once) may be used.
The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit storage unit 103.

The operation of learning device 400 according to the first embodiment will be described.
FIG. 15 is a flowchart for explaining the operation of learning device 400 according to the first embodiment.

The learning data acquiring section 401 performs a process of acquiring data for learning (step ST1501).
Specifically, the learning data acquisition unit 401 acquires a plurality of image drawings 1 to which teacher labels have been added, and the drawing characteristic information 2 corresponding to the image drawings 1 .
Then, the learning data acquisition unit 401 generates learning data based on the acquired image drawing 1. The learning data acquisition unit 401 outputs the generated learning data to the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404.
Furthermore, the learning data acquisition unit 401 stores the acquired drawing characteristic information 2 in the relevance recording unit 106 .

The symbol model generating section 402 learns symbol determination based on the learning data output by the learning data acquiring section 401 in step ST1501, and generates the first machine learning model 131 (step ST1502).
The symbol model generation unit 402 stores the generated first machine learning model 131 in the inference unit storage unit 103 .

The attribute information model generation unit 403 learns attribute judgment based on the learning data output by the learning data acquisition unit 401 in step ST1501, and generates the second machine learning model 132 (step ST1503).
The attribute information model generation unit 403 stores the generated second machine learning model 132 in the inference unit storage unit 103.

The connection line model generation unit 404 learns how to determine connection lines based on the learning data output by the learning data acquisition unit 401 in step ST1501, and generates the third machine learning model 133 (step ST1504).
The connection line model generation unit 404 stores the generated third machine learning model 133 in the inference unit storage unit 103.

In this way, the learning device 400 can generate a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on multiple image drawings 1 and drawing characteristic information 2 through so-called supervised learning.

The learning device 400 can perform weighting based on the drawing characteristic information 2 by generating a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on the image drawing 1 and the drawing characteristic information 2.
For example, if the feature extracted from the drawing characteristic information 2 indicates "home appliances", the learning device 400 will preferentially recognize symbols related to "home appliances" when training the machine learning model.
In this way, by using the drawing characteristic information 2 in addition to the image drawing 1 as input to the machine learning model, the learning device 400 can weight the symbols, attributes, or connecting lines on the image drawing 1 based on the drawing characteristic information 2, and can generate a machine learning model that outputs recognition results that take into account the purpose of the image drawing 1.

In the above first embodiment, the learning device 400 generates a machine learning model based on a plurality of image drawings 1 and drawing characteristic information 2, but the learning device 400 does not require the drawing characteristic information 2 in learning. In this case, the learning data acquisition unit 401 of the learning device 400 does not need to acquire the drawing characteristic information 2. In addition, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 of the learning device 400 generate the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133, respectively, based on the image drawing 1.
However, by generating a machine learning model using drawing characteristic information 2, the learning device 400 can generate a machine learning model with higher accuracy than when a machine learning model is generated without using drawing characteristic information 2.
In addition, if the learning device 400 does not use drawing characteristic information 2 in learning, the drawing recognition device 100 does not require drawing characteristic information 2 when performing inference based on the first machine learning model 131, the second machine learning model 132, and the third machine learning model 133.

16A and 16B are diagrams illustrating an example of a hardware configuration of the learning device 400 according to the first embodiment.
In the first embodiment, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 are realized by a processing circuit 1601. That is, the learning device 400 includes the processing circuit 1601 for generating a first machine learning model 131, a second machine learning model 132, and a third machine learning model 133 based on an image drawing 1 and drawing characteristic information 2.
The processing circuit 1601 may be dedicated hardware as shown in FIG. 16A, or may be a CPU (Central Processing Unit) 1604 that executes a program stored in a memory 1605 as shown in FIG. 16B.

When the processing circuit 1601 is dedicated hardware, the processing circuit 1601 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

When the processing circuit 1601 is a CPU 1604, the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1605. The processing circuit 1601 executes the functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 by reading and executing the program stored in the memory 1605. That is, the learning device 400 includes a memory 1605 for storing a program that, when executed by the processing circuit 1601, results in the execution of steps ST1501 to ST1504 in FIG. 15 described above. In addition, it can be said that the program stored in the memory 1605 causes the computer to execute the procedures or methods of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404. Here, memory 1605 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

The functions of the learning data acquisition unit 401, the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the learning data acquisition unit 401 can be realized by a processing circuit 1601 as dedicated hardware, and the functions of the symbol model generation unit 402, the attribute information model generation unit 403, and the connection line model generation unit 404 can be realized by the processing circuit 1601 reading out and executing a program stored in a memory 1605.
The learning device 400 also includes an input interface device 1602 and an output interface device 1603 that perform wired or wireless communication with devices such as the drawing recognition device 100 .

As described above, the drawing recognition device 100 of the first embodiment is configured to include a drawing acquisition unit 101 that acquires drawing data (image drawing 1), a recognition unit 102 that recognizes information about symbols in the drawing data acquired by the drawing acquisition unit 101, information about attributes in the drawing data acquired by the drawing acquisition unit 101, and information about connecting lines in the drawing data acquired by the drawing acquisition unit 101 based on the drawing data acquired by the drawing acquisition unit 101 and a machine learning model that receives drawing data as input and outputs recognition results about symbols, attributes, and connecting lines connecting symbols, and a determination unit 104 that determines the correspondence between symbols, attributes, and other symbols in the drawing data acquired by the drawing acquisition unit 101 based on the information about symbols, information about attributes, and information about connecting lines recognized by the recognition unit 102. Therefore, the drawing recognition device 100 can determine the correspondence between characters and symbols without requiring placement knowledge.

Note that this disclosure allows for modification of any of the components of the embodiments, or omission of any of the components of the embodiments.

1 Image drawing, 2 Drawing characteristic information, 100 Drawing recognition device, 101 Drawing acquisition unit, 102 Recognition unit, 1021 Symbol recognition unit, 1022 Attribute information recognition unit, 1023 Connection line recognition unit, 103 Inference device storage unit, 131 First machine learning model, 132 Second machine learning model, 133 Third machine learning model, 104 Judgment unit, 105 Part recording unit, 106 Association degree recording unit, 107 Generation unit, 108 Comparison unit, 109 Matching data recording unit, 110 Mismatching data recording unit, 111 Drawing display unit, 200 Operation input device, 201 Keyboard, 202 Mouse, 300 Display device, 301 Display, 400 Learning device, 401 Learning data acquisition unit, 402 Symbol model generation unit, 403 Attribute information model generation unit, 404 Connection line model generation unit, 1301, 1601 Processing circuit, 1302, 1602 input interface device, 1303, 1603 output interface device, 1304, 1604 CPU, 1305, 1605 memory.

Claims (23)

A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit ,
The determination unit is
Based on information on the symbol, the attribute, and the connection line recognized by the recognition unit, and on the degree of association between the symbol and the attribute shown on the drawing data, which is represented by the number of times the symbol and the attribute related to the symbol are shown together on the drawing data, or on the degree of association between the symbols, which is represented by the number of times the symbol and the connection destination symbol to which the symbol is connected are shown connected on the drawing data, the corresponding relationship between the symbol, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit is determined.
A drawing recognition device characterized by the above . The recognition unit is
The drawing recognition device of claim 1, characterized in that it recognizes information regarding the symbols, information regarding the attributes, and information regarding the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that takes the drawing data as input and outputs a symbol recognition result regarding the symbols on the drawing data, a second machine learning model that takes the drawing data as input and outputs an attribute recognition result regarding the attributes on the drawing data, and a third machine learning model that takes the drawing data as input and outputs a connection line recognition result regarding the connection lines connecting the symbols on the drawing data. The recognition unit is
The drawing recognition device according to claim 1, characterized in that it recognizes information about the symbols, information about the attributes, and information about the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to the characteristics of the drawing data, and the machine learning model. The recognition unit is
The drawing recognition device according to claim 2, characterized in that it recognizes information about the symbols, information about the attributes, and information about the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to the characteristics of the drawing data , and the machine learning model. The relevance information is
2. The drawing recognition device according to claim 1, further comprising at least one of: symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each of the symbols; symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each of the symbols; usage relevance information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used; and model relevance information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used. 6. The drawing recognition device according to claim 1, further comprising: a generation unit that generates a post-determination image drawing data based on a result of the determination by the determination unit of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data. 7. The drawing recognition device according to claim 6, further comprising a comparison unit which calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the determined image drawing data as matching data when the similarity is high, and records the determined image drawing data as non-matching data when the similarity is low. The determination unit is
8. The drawing recognition device according to claim 7, further comprising: a step of re-determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on the mismatch data recorded by the comparison unit. A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit ,
The determination unit is
determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on information related to the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, or on a degree of association between the connections between the symbols;
The relevance information is
The data includes at least one of symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each symbol, symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each symbol, usage association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used, and model association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used.
A drawing recognition device characterized by the above . The recognition unit is
The drawing recognition device of claim 9, characterized in that it recognizes information regarding the symbol, information regarding the attribute, and information regarding the connection line connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that takes the drawing data as input and outputs a symbol recognition result regarding the symbol on the drawing data, a second machine learning model that takes the drawing data as input and outputs an attribute recognition result regarding the attribute on the drawing data, and a third machine learning model that takes the drawing data as input and outputs a connection line recognition result regarding the connection line connecting the symbols on the drawing data . The recognition unit is
The drawing recognition device according to claim 9, characterized in that it recognizes information about the symbols, information about the attributes, and information about the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information related to the characteristics of the drawing data , and the machine learning model. 12. The drawing recognition device according to claim 9 , further comprising: a generation unit configured to generate determined image drawing data based on a result of the determination made by the determination unit regarding the correspondence relationship between the symbol , the attribute, and another symbol in the drawing data. 13. The drawing recognition device according to claim 12, further comprising a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the determined image drawing data as matching data when the similarity is high, and records the determined image drawing data as non-matching data when the similarity is low . The determination unit is
14. The drawing recognition device according to claim 13 , further comprising: a step of re-determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on the mismatch data recorded by the comparison unit. A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit ;
a generating unit that generates determined image drawing data based on a result of the determination by the determining unit of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data;
a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low.
A drawing recognition device equipped with the above. The recognition unit is
The drawing recognition device of claim 15, characterized in that it recognizes information regarding the symbol, information regarding the attribute, and information regarding the connection line connecting the symbols in the drawing data acquired by the drawing acquisition unit based on the drawing data acquired by the drawing acquisition unit, a first machine learning model that takes the drawing data as input and outputs a symbol recognition result regarding the symbol on the drawing data, a second machine learning model that takes the drawing data as input and outputs an attribute recognition result regarding the attribute on the drawing data, and a third machine learning model that takes the drawing data as input and outputs a connection line recognition result regarding the connection line connecting the symbols on the drawing data. The recognition unit is
The drawing recognition device according to claim 15, characterized in that, based on the drawing data acquired by the drawing acquisition unit, drawing characteristic information relating to characteristics of the drawing data, and the machine learning model, the device recognizes information regarding the symbols, information regarding the attributes, and information regarding the connection lines connecting the symbols in the drawing data acquired by the drawing acquisition unit. The determination unit is
18. The drawing recognition device according to claim 15, further comprising: a determining unit that determines a correspondence between the symbol, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit based on information regarding the symbol, the attribute, and the connection line recognized by the recognition unit, and based on a degree of association between the symbol and the attribute shown on the drawing data, or on a degree of association between the connections between the symbols. The determination unit is
16. The drawing recognition device according to claim 15 , further comprising: a step of re-determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on the mismatch data recorded by the comparison unit. Computer,
A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a program for causing the program to function as a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on information about the symbol, information about the attribute, and information about the connection line recognized by the recognition unit, the program comprising:
The determination unit is
Based on information on the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, which is represented by the number of times the symbol and the attribute related to the symbol are shown together on the drawing data, or on a degree of association between the symbols, which is represented by the number of times the symbol and a destination symbol to which the symbol is connected are shown connected on the drawing data, the corresponding relationship between the symbol, the attribute, and other symbols in the drawing data acquired by the drawing acquisition unit is determined.
A drawing recognition program comprising : Computer,
A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a program for causing the program to function as a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on information about the symbol, information about the attribute, and information about the connection line recognized by the recognition unit, the program comprising:
The determination unit is
determining a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit based on information related to the symbol, the attribute, and the connection line recognized by the recognition unit, and on a degree of association between the symbol and the attribute shown on the drawing data, or on a degree of association between the connections between the symbols;
The relevance information is
The data includes at least one of symbol-attribute information in which a degree of association between the symbol and the attribute is defined for each symbol, symbol-connection information in which a degree of association between the symbol and other symbols connected to the symbol is defined for each symbol, usage association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each usage in which the drawing data acquired by the drawing acquisition unit is used, and model association information in which a degree of association between the symbol and the attribute and a degree of association between the symbol and other symbols are defined for each model of equipment in which the drawing data acquired by the drawing acquisition unit is used.
A drawing recognition program comprising: Computer,
A drawing acquisition unit that acquires drawing data;
a recognition unit that recognizes, based on the drawing data acquired by the drawing acquisition unit and a machine learning model that receives the drawing data as an input and outputs recognition results related to symbols, attributes, and connection lines connecting the symbols on the drawing data, information related to the symbols in the drawing data acquired by the drawing acquisition unit, including position information of the symbols, information related to the attributes in the drawing data acquired by the drawing acquisition unit, including position information of the attributes, and information related to the connection lines in the drawing data acquired by the drawing acquisition unit, including position information of the connection lines;
a determination unit that determines a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit, based on the information about the symbol, the information about the attribute, and the information about the connection line recognized by the recognition unit;
a generating unit that generates determined image drawing data based on a result of the determination by the determining unit of the correspondence relationship between the symbol, the attribute, and another symbol in the drawing data;
a comparison unit that calculates a similarity between the determined image drawing data generated by the generation unit and the drawing data acquired by the drawing acquisition unit, and records the similarity as matched data when the similarity is high, and records the similarity as non-matched data when the similarity is low.
A drawing recognition program to function as a. The determination unit is
Based on the mismatch data recorded by the comparison unit, a correspondence relationship between the symbol, the attribute, and another symbol in the drawing data acquired by the drawing acquisition unit is re-determined.
23. The drawing recognition program according to claim 22.

Images