JP7830257B2 - Secondary visual inspection apparatus, visual inspection system, and secondary visual inspection method - Google Patents

Description

This invention relates to a technique for accurately determining the state of an object by performing a secondary determination on an object that has been determined to be defective by a primary determination, which determines the state of the object in each of multiple inspection areas.

A visual inspection device is known that determines the condition of solder based on an image of the solder used to join components to a circuit board. In such a visual inspection device, to determine the condition of the solder in detail, judgments may be performed on multiple inspection areas, each with different characteristics. For example, three inspection areas at different distances from the component's electrodes may be set. The inspection area closest to the electrode (first closest) determines whether there is a solder shape defect (unsoldered) where the solder is indented before reaching the electrode. The inspection area second closest to the electrode determines whether there is a solder shortage defect (insufficient solder). The inspection area third closest to the electrode determines whether there is a metal foil exposure defect (red eye) where the metal foil (copper foil) that should be covered by the solder is exposed.

However, such visual inspection devices may sometimes incorrectly identify a product as defective even if it is actually in good condition. Therefore, a secondary assessment is performed by the user to complement the visual inspection device's initial assessment. Specifically, if the visual inspection device's initial assessment identifies any of the multiple inspection areas as defective, an image of the solder that was the subject of the initial assessment is displayed on the screen. This allows the user to perform a final assessment based on visual inspection of the display (secondary assessment).

Japanese Patent Publication No. 2010-8159

Incidentally, it is conceivable that the secondary judgment, which was previously performed by the user, could be automated to reduce manpower. For example, Patent Document 1 proposes a neural network that determines the state of an object based on abnormalities in pixels contained in the object's image. However, there were the following problems when using a machine to perform the secondary judgment that complements the primary judgment by the visual inspection device.

In other words, the secondary assessment performed by the user is intended to confirm the status of the inspection areas that were determined to be defective in the primary assessment. However, if the image of the object contains multiple inspection areas, the machine may calculate the abnormalities contained in the image, potentially leading to over-judgment where inspection areas not determined to be defective in the primary assessment are incorrectly identified as defective.

This invention was made in view of the above-mentioned problems, and aims to suppress the occurrence of over-judgments when a machine performs a secondary judgment to determine the state of an object that has been determined to be defective by a primary judgment, which determines the state of the object in each of multiple inspection areas.

The secondary visual inspection apparatus according to the present invention comprises: an information acquisition unit that acquires defective area information indicating a defective area among multiple inspection areas in a primary determination that determines the state of the object in each of the multiple different inspection areas; an abnormality data calculation unit that calculates defective area abnormality data indicating abnormalities contained in the image of the object in the defective area indicated by the defective area information; and a secondary determination execution unit that performs a secondary determination that determines the state of the object based on the defective area abnormality data.

The visual inspection system according to the present invention comprises a primary visual inspection device that performs a primary determination to determine the state of an object in each of a plurality of different inspection areas and outputs defective area information indicating the defective areas determined to be defective in the primary determination, and the above-mentioned secondary visual inspection device, and the information acquisition unit acquires the defective area information output from the primary visual inspection device.

The secondary visual inspection method according to the present invention comprises the following steps: a primary determination step in which the state of an object in each of a plurality of different inspection areas is determined; a primary determination step in which defective area information is obtained indicating a defective area among the plurality of inspection areas; a primary determination step in which defective area anomaly data is calculated indicating anomalies contained in the image of the object in the defective area indicated by the defective area information; and a secondary determination step in which the state of the object is determined based on the defective area anomaly data.

In the present invention (secondary visual inspection apparatus, visual inspection system, and secondary visual inspection method) configured as described above, defective area information is acquired, indicating the defective areas among multiple inspection areas that were determined to be defective in the primary inspection. Then, defective area anomaly data is calculated, indicating abnormalities contained in the image of the object within the defective areas indicated by the defective area information. Based on this defective area anomaly data, the state of the object is determined (secondary inspection). In other words, inspection areas that were not determined to be defective in the primary inspection are excluded from the criteria for determining the state of the object in the secondary inspection. As a result, it is possible to suppress the occurrence of over-judgments, where inspection areas that were not determined to be defective in the primary inspection are mistakenly judged as defective.

Furthermore, the abnormality data calculation unit may be configured such that the secondary visual inspection device includes an image acquisition unit that acquires an object image, which is an image of the object in an imaging area containing multiple inspection areas, captured in the primary judgment; multiple machine learning models, each corresponding to one of the multiple inspection areas; and a data calculation unit that calculates defective area abnormality data using the corresponding model among the multiple machine learning models that corresponds to the defective area. Each of the multiple machine learning models outputs inspection area abnormality data indicating abnormalities contained in the image captured in the corresponding inspection area. The data calculation unit extracts images contained in the defective area from the object image as defective area images and calculates the inspection area abnormality data output by the corresponding model for the defective area images as defective area abnormality data. In such a configuration, inspection areas that were not judged as defective in the primary judgment are excluded from the criteria for determining the state of the object in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas that were not judged as defective in the primary judgment are judged as defective.

Furthermore, the abnormality data calculation unit may be configured as follows: an image acquisition unit that acquires an object image, which is an image of the object in an imaging area containing multiple inspection areas, captured in the primary judgment; a machine learning model that outputs imaging area abnormality data indicating abnormalities contained in the image captured in the imaging area; and a data calculation unit that calculates defective area abnormality data by extracting data corresponding to defective areas from the imaging area abnormality data output by the machine learning model for the object image. In such a configuration, inspection areas that were not judged as defective in the primary judgment are excluded from the criteria for determining the state of the object in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas that were not judged as defective in the primary judgment are judged as defective. Moreover, instead of providing multiple machine learning models corresponding to multiple inspection areas, it is sufficient to provide one machine learning model for the imaging area containing multiple inspection areas. Therefore, the user does not need to manage multiple machine learning models, and the user's management burden can be reduced.

Furthermore, the secondary visual inspection apparatus may be configured to further include a user interface and a display control unit that displays visualized abnormal data, including at least defective area abnormal data, on the user interface. In such a configuration, the user can visually confirm the defective area abnormal data included in the visualized abnormal data.

Furthermore, the secondary visual inspection device may be configured to include a mode selection unit that selectively executes two modes: an automatic judgment mode in which the quality of the object is determined by a secondary judgment performed by a secondary judgment execution unit, and a manual judgment mode in which the quality of the object is determined based on the user's operations on the user interface while displaying abnormality data on the user interface. In such a configuration, selecting the automatic judgment mode reduces the user's burden in the secondary judgment, while selecting the manual judgment mode allows the user to perform a secondary judgment visually.

Furthermore, the mode selection unit may be configured to execute one mode selected by the user's operation on the user interface, from among the automatic and manual judgment modes. In such a configuration, the system can execute the mode requested by the user, from among the automatic and manual judgment modes.

As described above, according to the present invention, when a secondary determination is performed by a machine to further determine the state of an object that has been determined to be defective by a primary determination, which determines the state of the object in each of the multiple inspection areas, it becomes possible to suppress the occurrence of over-determination.

A block diagram showing an example of the visual inspection system according to the present invention. A flowchart showing an example of a primary judgment performed by the primary visual inspection device in Figure 1. This diagram schematically illustrates the actions performed in the initial determination shown in Figure 2. A block diagram showing a first example of a secondary visual inspection apparatus according to the present invention. A flowchart showing the secondary determination performed by the first example of the secondary visual inspection apparatus in Figure 4. This diagram schematically illustrates the actions performed in the secondary determination shown in Figure 5. A block diagram showing a second example of a secondary visual inspection apparatus according to the present invention. A flowchart showing the secondary determination performed by the second example of the secondary visual inspection apparatus in Figure 7.

Figure 1 is a block diagram illustrating an example of a visual inspection system according to the present invention. The visual inspection system 1 in Figure 1 comprises a primary visual inspection device 2 and a secondary visual inspection device 4, and inspects the condition of solder S used to join components E, such as capacitors, resistors, or integrated circuits, to a substrate B. The primary visual inspection device 2 has a light irradiation unit 21 and an imaging camera 22. It irradiates a predetermined imaging area Ac, including the solder S being inspected, with light from the light irradiation unit 21, while simultaneously imaging the imaging area Ac with the imaging camera 22, thereby capturing a solder image Is representing the solder S. Furthermore, the primary visual inspection device 2 includes a communication unit 28 for communication with the secondary visual inspection device 4, and a controller 29 for controlling the light irradiation unit 21, the imaging camera 22, and the communication unit 28. The controller 29 acquires a primary judgment result R, which is the result of determining the quality of the solder S based on the solder image Is. The communication unit 28 transmits the solder image Is and the primary judgment result R to the secondary visual inspection device 4.

As for the specific configuration of the primary visual inspection device 2, for example, the device configuration disclosed in Japanese Patent Publication No. 2010-071844 can be adopted. In the device described in that publication, light with different wavelengths (infrared, red, green, and blue) is irradiated onto the solder at different angles, and a camera facing the substrate captures the light reflected from the solder, thereby acquiring an image. The quality of the solder is then determined based on this image. Note that the specific configuration of the primary visual inspection device 2 is not limited to this example; for example, the device configuration described in WO2018/163278 can also be adopted.

Figure 2 is a flowchart showing an example of the primary judgment performed by the primary visual inspection apparatus in Figure 1, and Figure 3 is a schematic diagram showing the operations performed in the primary judgment in Figure 2. In Figure 3, only component E is shown, and solder S is not shown.

In the initial assessment shown in Figure 2, multiple distinct inspection areas A(N) (N=1, 2, 3) are set, as shown in Figure 3. Inspection area A(1) is set to determine the presence or absence of solder shape defects (unsoldered) where the solder S is indented in front of the electrode of component E; inspection area A(2) is set to determine the presence or absence of solder deficiency defects (slight soldering) where the amount of solder S is insufficient; and inspection area A(3) is set to determine the presence or absence of metal foil exposure defects (red eye) where the metal foil (copper foil) that should be covered by the solder S is exposed.

As shown in Figure 2, in step S101, the controller 29 illuminates a predetermined imaging area Ac containing the solder S with light from the light irradiation unit 21, and simultaneously captures the imaging area Ac with the imaging camera 22, thereby capturing a solder image Is. This solder image Is is image data indicating the brightness of each of the multiple pixels included in the imaging area Ac. In step S102, the controller 29 resets the count value N, which identifies the inspection area A(N), to zero, and in step S103, the controller 29 increments the count value N by 1.

The controller 29 then determines the quality of the condition in inspection area A(N) based on the solder image Is that includes the area A(N) (step S104). The controller 29 also saves the judgment result (good/bad) for the inspection area A(N) (step S105). In step S106, it is checked whether the count value N has reached the maximum count value Nx (=3). The maximum count value Nx corresponds to the number of inspection areas A(N) (=3). Steps S104 and S105 are repeated while incrementing the count value N until the count value Nx is reached. As a result, judgment results are obtained for each of the multiple inspection areas A(N) (N=1, 2, 3).

In step S107, the controller 29 checks whether there is a defect in each of the multiple inspection areas A(N). If there is a defect (if "YES" is found in step S107), the controller 29 outputs the primary judgment result R, which indicates the defective area An among the multiple inspection areas A(N), and the solder image Is captured in step S101, to the secondary visual inspection device 4 via the communication unit 28.

Figure 4 is a block diagram showing a first example of a secondary visual inspection apparatus according to the present invention. The secondary visual inspection apparatus 4 is a computer equipped with a calculation unit 41, a storage unit 42, a communication unit 43, and a UI (User Interface) 44. The calculation unit 41 is a CPU (Central
The processor is a Processing Unit (PCU), and the memory unit 42 is a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). The communication unit 43 communicates with the communication unit 28 of the primary visual inspection device 2, and receives, for example, the primary judgment result R and solder image Is output from the communication unit 28 and stores them in the memory unit 42. The UI 44 has input devices such as a mouse and keyboard, and output devices such as a display. Note that the input and output devices of the UI 44 do not need to be configured separately, and they may be integrated using a touch panel display.

The calculation unit 41 executes a predetermined secondary judgment program to configure the information acquisition unit 411, image acquisition unit 412, data calculation unit 413, automatic judgment execution unit 414, manual judgment control unit 415, and mode selection unit 416. The storage unit 42 stores multiple machine learning models M(N) corresponding to multiple inspection areas A(N) (N=1, 2, 3). Each machine learning model M(N) has learned the relationship between the inspection area image Ia(N) (Figure 3), which is an image within the inspection area A(N) corresponding to the machine learning model M(N) from the solder image Is, and the abnormality level of the inspection area image Ia(N). When an inspection area image Ia(N) is input, the abnormality level of the inspection area image Ia(N) is output. In other words, machine learning model M(1) outputs the abnormality level of inspection area A(1) and can be used to determine the presence or absence of solder shape defects (unsoldered areas). Machine learning model M(2) outputs the degree of abnormality of inspection area A(2) and can be used to determine the presence or absence of solder deficiency (small solder). Machine learning model M(3) outputs the degree of abnormality of inspection area A(3) and can be used to determine the presence or absence of metal foil exposure defects (red eye).

Furthermore, the training method for the machine learning model that outputs the image anomaly score can employ well-known techniques. For example, the feature vectors of the intermediate layer of an existing machine learning model obtained when a good product image is input can be saved as a reference. Then, the anomaly score can be calculated by the distance between the feature vector obtained when an image captured by the primary visual inspection device 2 is input to this machine learning model and the reference feature vector. This anomaly score can be calculated for each of the multiple pixels that make up the image.

Figure 5 is a flowchart showing the secondary judgment performed by the first example of the secondary visual inspection apparatus in Figure 4, and Figure 6 is a schematic diagram showing the operations performed in the secondary judgment in Figure 5. Note that in Figure 6, only component E is shown; solder S is not.

As shown in Figure 5, the information acquisition unit 411 acquires the primary judgment result R received from the primary visual inspection device 2 by the communication unit 43 (step S201), and the image acquisition unit 412 acquires the solder image Is received from the primary visual inspection device 2 by the communication unit 43 (step S202). As illustrated in the column for step S202 in Figure 6, this solder image Is shows the solder S captured in the imaging area Ac which includes multiple inspection areas A(N) set in the primary judgment. Note that the execution order of steps S201 and S202 is not limited to this example.

In step S203, the data calculation unit 413 extracts a defective area image In from the solder image Is, which is the image of the defective area An indicated by the primary judgment result R. In the example shown in the step S203 column of Figure 6, among the multiple inspection areas A(1), A(2), and A(3), inspection area A(2) corresponds to the defective area An, and the defective area image In of the defective area An is extracted from the solder image Is. Furthermore, the data calculation unit 413 selects the machine learning model M(2) that corresponds to the defective area An (i.e., inspection area A(2)) from among the machine learning models M(1), M(2), and M(3) that correspond to the multiple inspection areas A(1), A(2), and A(3) respectively (step S204). Then, the data calculation unit 413 inputs the defective area image In to the machine learning model M(2) selected in step S204 and calculates the degree of abnormality of the defective area image In.

In step S206, the mode selection unit 416 confirms whether to execute an automatic determination mode, in which the automatic determination execution unit 414 automatically performs the secondary determination, or a manual determination mode, in which the manual determination control unit 415 performs the secondary determination based on the user's input operation to the UI 44. Specifically, the user can set which mode to execute (automatic determination mode or manual determination mode) to the mode selection unit 416 by operating the UI 44.

If the automatic judgment mode is set (if "YES" is selected in step S206), the automatic judgment execution unit 414 determines the quality of the solder S based on the abnormality level of the defective area image In calculated in step S205 (step S207). Specifically, if the abnormality level of each pixel constituting the defective area image In is below the threshold (if "YES" is selected in step S207), the automatic judgment execution unit 414 obtains a good product judgment as the result of the secondary judgment, indicating that the solder S shown in the solder image Is is in good condition (step S208). On the other hand, if there is a pixel among the pixels constituting the defective area image In whose abnormality level is greater than the threshold (if "NO" is selected in step S207), the automatic judgment execution unit 414 obtains a defective product judgment as the result of the secondary judgment, indicating that the solder S shown in the solder image Is is in poor condition (step S209). The results of the secondary judgment obtained in steps S208 and S209 are displayed, for example, on the UI 44 display.

If manual judgment mode is set (if "NO" is selected in step S206), the manual judgment control unit 415 displays an abnormality map (heatmap) that visualizes the degree of abnormality of the defective area image In on the UI 44's display (step S210). Then, the manual judgment control unit 415 determines whether the user's judgment entered into the UI 44 is good or not (step S211). If the user's judgment entered into the UI 44 is good (if "YES" is selected in step S211), the manual judgment control unit 415 obtains a good product judgment as a result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is good (step S208). On the other hand, if the user's judgment entered into the UI 44 is bad (if "NO" is selected in step S211), the manual judgment control unit 415 obtains a bad product judgment as a result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is bad (step S209). The results of the secondary determination obtained in steps S208 and S209 are displayed, for example, on the UI44 display.

In the embodiment described above, a primary judgment result R (defective area information) is obtained, indicating the defective area An (inspection area A(2)) that was determined to be defective in the primary judgment among the multiple inspection areas A(N) (step S201). Then, the degree of abnormality (defective area abnormality data), i.e., the degree of abnormality of the defective area image In, which indicates the abnormality contained in the solder image Is of the solder S (object) in the defective area An indicated by the primary judgment result R, is calculated (step S205). In steps S206 to S209, the state of the solder S is determined based on this degree of abnormality (secondary judgment). In other words, inspection areas A(1) and A(3), which were not determined to be defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and A(3), which were not determined to be defective in the primary judgment, are judged to be defective.

In particular, the solder image Is (object image), which is an image of the solder S in the imaging area Ac that includes multiple inspection areas A(N) captured in the primary judgment, is acquired by the information acquisition unit 411 (step S202). In response to this, multiple machine learning models M(N), each corresponding to one of the multiple inspection areas A(n), are stored in the storage unit 42. Then, using the machine learning model M(2) (corresponding model) corresponding to the defective area image In (inspection area A(2)) from among the multiple machine learning models M(N), the abnormality level of the defective area image In (defective area abnormality data) is calculated by the data calculation unit 413 (steps S203 to S205). In other words, each of the multiple machine learning models M(N) is configured to output the abnormality level (inspection area abnormality data) contained in the inspection area image Ia(N) captured in the corresponding inspection area A(N). Therefore, the data calculation unit 413 extracts the image included in the defective area An (inspection area A(2)) from the solder image Is as the defective area image In (step S203), and calculates the abnormality level (inspection area abnormality data) output for the defective area image In by the machine learning model M(2) (corresponding model) corresponding to the defective area An as the abnormality level (defective area abnormality data) of the defective area image In (steps S204, S205). In this configuration, inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, in which inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are judged as defective.

Furthermore, the system includes a UI 44 and a manual judgment control unit 415 (display control unit) that displays an abnormality map (visualized abnormality data) containing at least the abnormality level (abnormality data of the abnormal region) in the defective region image In on the UI 44. In this configuration, the user can visually confirm the abnormality level in the defective region image In included in the abnormality map.

Furthermore, the system is equipped with a mode selection unit 416 that selectively executes two modes: an automatic judgment mode (steps S207, S208, S209) in which the quality of the solder S is determined by a secondary judgment performed by an automatic judgment execution unit 414 (secondary judgment execution unit), and a manual judgment mode (steps S210, S211, S208, S209) in which the quality of the solder S is determined based on the user's operations on the UI 44 while displaying an abnormality map on the UI 44. In this configuration, selecting the automatic judgment mode reduces the user's burden during the secondary judgment, while selecting the manual judgment mode allows the user to perform a secondary judgment visually.

Furthermore, the mode selection unit 416 executes one mode selected by the user's operation on the UI 44 from among the automatic determination mode and manual determination mode (step S206). In this configuration, the system can execute a mode according to the user's request from among the automatic determination mode and manual determination mode.

Figure 7 is a block diagram showing a second example of the secondary visual inspection apparatus according to the present invention, and Figure 8 is a flowchart showing the secondary determination performed by the second example of the secondary visual inspection apparatus in Figure 7. The following explanation will focus on the differences from the first example, and commonalities with the first example will be denoted by corresponding reference numerals and their explanations will be omitted as appropriate. However, it goes without saying that by incorporating configurations common to the first example, the same effects as the first example will be achieved.

In the memory unit 42 of the primary visual inspection device 2 shown in Figure 7, a machine learning model Mc is stored. This machine learning model Mc is provided in correspondence with the imaging area Ac which includes multiple inspection areas A(N). In other words, the machine learning model Mc has learned the relationship between the solder image Is captured in the imaging area Ac and the abnormality level of the solder image Is, and when a solder image Is is input, it outputs the abnormality level of the solder image Is.

As shown in Figure 8, the information acquisition unit 411 acquires the primary judgment result R received from the primary visual inspection device 2 by the communication unit 43 (step S301), and the image acquisition unit 412 acquires the solder image Is received from the primary visual inspection device 2 by the communication unit 43 (step S302). Note that the execution order of steps S301 and S302 is not limited to this example.

In step S303, the data calculation unit 413 inputs the solder image Is to the machine learning model Mc to calculate the degree of abnormality of the solder image Is. Furthermore, the data calculation unit 413 extracts the degree of abnormality in the defective region An indicated by the primary judgment result R from the degree of abnormality calculated in step S303 (step S304).

In step S306, the mode selection unit 416 checks whether automatic judgment mode or manual judgment mode is set. If automatic judgment mode is set (if "YES" is selected in step S306), the automatic judgment execution unit 414 determines whether the solder S is good or bad based on the degree of abnormality in the defective region An extracted in step S304 (step S307). In other words, if the degree of abnormality of each pixel included in the defective region An is below the threshold (if "YES" is selected in step S307), the automatic judgment execution unit 414 obtains a good product judgment as the result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is good (step S308). On the other hand, if there is a pixel among the pixels included in the defective region An whose degree of abnormality is greater than the threshold (if "NO" is selected in step S307), the automatic judgment execution unit 414 obtains a bad product judgment as the result of the secondary judgment, indicating that the state of the solder S shown in the solder image Is is bad (step S309). The results of the secondary determination obtained in steps S308 and S309 are displayed, for example, on the UI44 display.

If manual judgment mode is set (i.e., "NO" in step S306), steps S310, S311, S308, and S309 are executed in the same manner as steps S210, S211, S208, and S209 in the first example.

In the embodiment described above, a primary judgment result R (defective area information) is obtained, indicating the defective area An (inspection area A(2)) that was determined to be defective in the primary judgment among the multiple inspection areas A(N) (step S301). Then, the degree of image abnormality (defective area abnormality data) in the defective area An indicated by the primary judgment result R is calculated from the solder image Is (steps S303, S304). Then, in steps S307 to S309, the state of the solder S is determined based on the degree of image abnormality (defective area abnormality data) in the defective area An (secondary judgment). In other words, inspection areas A(1) and S(3), which were not determined to be defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and S(3), which were not determined to be defective in the primary judgment, are judged to be defective.

In particular, the image acquisition unit 412 acquires a solder image Is (object image), which is an image of the solder S in imaging area Ac, which includes multiple inspection areas A(N), captured in the primary judgment. A machine learning model Mc, which outputs the degree of abnormality (imaging area abnormality data) of the image captured in imaging area Ac, is stored in the storage unit 42. The data calculation unit 413 then calculates the degree of abnormality (defective area abnormality data) in defective area An by extracting the degree of abnormality (data) corresponding to defective area An from the degree of abnormality (imaging area abnormality data) output by the machine learning model Mc for solder image Is (steps S303, S304). In this configuration, inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are excluded from the criteria for determining the state of the solder S in the secondary judgment. As a result, it is possible to suppress the occurrence of over-judgment, where inspection areas A(1) and A(3), which were not judged as defective in the primary judgment, are judged as defective. Furthermore, instead of creating multiple machine learning models M(1), M(2), and M(3) corresponding to multiple examination areas A(1), A(2), and A(3), it is sufficient to create one machine learning model Mc for the imaging area Ac which includes multiple examination areas A(1), A(2), and A(3). Therefore, users do not need to manage multiple machine learning models M(1), M(2), and M(3), which reduces the management burden on users.

As described above, in this embodiment, the appearance inspection system 1 corresponds to an example of the "appearance inspection system" of the present invention, the primary appearance inspection device 2 corresponds to an example of the "primary appearance inspection device" of the present invention, the secondary appearance inspection device 4 corresponds to an example of the "secondary appearance inspection device" of the present invention, the information acquisition unit 411 corresponds to an example of the "information acquisition unit" of the present invention, the image acquisition unit 412, the machine learning model M(N), Mc, and the data calculation unit 413 cooperate to function as the "abnormal data calculation unit" of the present invention, the automatic judgment execution unit 414 corresponds to an example of the "secondary judgment execution unit" of the present invention, the inspection area A(N) corresponds to an example of the "inspection area" of the present invention, the defective area An corresponds to an example of the "defective area" of the present invention, the degree of abnormality in the defective area An of the solder image Is corresponds to an example of the "defective area abnormal data" of the present invention, the primary judgment result R corresponds to an example of the "defective area information" of the present invention, and the solder S corresponds to an example of the "object" of the present invention.

Furthermore, the present invention is not limited to the embodiments described above, and various modifications can be made to those described without departing from the spirit of the invention. For example, the specific configuration of the multiple inspection areas A(N) in the primary determination is not limited to the examples above. Therefore, two inspection areas A(N) may be set from inspection areas A(1), A(2), and A(3), or different inspection areas may be set from inspection areas A(1), A(2), and A(3).

Furthermore, the subjects of the primary and secondary assessments are not limited to the solder S described above.

Furthermore, it is not necessarily required that the primary visual inspection device 2 and the secondary visual inspection device 4 be configured as separate units; they may be configured as an integrated unit.

Furthermore, the manual judgment mode described above is not mandatory, and it is not necessary to include a manual judgment mode in the secondary judgment.

1…Visual inspection system 2…Primary visual inspection device 4…Secondary visual inspection device 411…Information acquisition unit 412…Image acquisition unit (abnormal data calculation unit)
413...Data calculation unit (abnormal data calculation unit)
414...Automatic judgment execution unit (secondary judgment execution unit)
A(N)...Inspection area An...Defective area Is...Solder image M(N)... Machine learning model (anomaly data calculation unit)
Mc... Machine learning model (anomalous data calculation unit)
R…Primary judgment result (defective area information)
S... Solder (object)

Claims (8)

In a primary determination that determines the state of an object in each of several different inspection areas, an information acquisition unit acquires defective area information indicating the defective area that was determined to be defective among the multiple inspection areas.
An abnormality data calculation unit calculates abnormality data for a defective region that indicates an abnormality contained in the image of the object in the defective region indicated by the defective region information,
The system includes a secondary determination execution unit that performs a secondary determination to determine the state of the object based on the aforementioned defective area abnormality data,
The aforementioned abnormal data calculation unit,
An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination;
Multiple machine learning models are provided, each corresponding to one of the aforementioned multiple inspection areas, and each outputs inspection area anomaly data indicating anomalies contained in an image that includes only the corresponding inspection area.
The system includes a data calculation unit that calculates abnormal data for the defective region using a corresponding model among the plurality of machine learning models that corresponds to the defective region,
The data calculation unit extracts images from the object image that are included in the defective area as defective area images, thereby removing the inspection area that was not determined to be defective in the primary determination from the criteria for determining the state of the object, and using the inspection area abnormality data output by the corresponding model for the defective area image as the defective area abnormality data .
The secondary determination execution unit is a secondary visual inspection device that determines the state of the object based on the inspection area abnormality data output by the corresponding model as the defective area abnormality data with respect to the defective area image extracted by the data calculation unit . In a primary determination that determines the state of an object in each of several different inspection areas, an information acquisition unit acquires defective area information indicating the defective area that was determined to be defective among the multiple inspection areas.
An abnormality data calculation unit calculates abnormality data for a defective region that indicates an abnormality contained in the image of the object in the defective region indicated by the defective region information,
The system includes a secondary determination execution unit that performs a secondary determination to determine the state of the object based on the aforementioned defective area abnormality data,
The aforementioned abnormal data calculation unit,
An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination;
A machine learning model that outputs imaging region anomaly data, which is an anomaly score obtained for each of the multiple pixels that make up the image captured in the aforementioned imaging region,
The machine learning model outputs abnormal imaging region data for the object image, and the data calculation unit extracts data corresponding to the defective region from this data , thereby removing the inspection region that was not determined to be defective in the initial judgment from the criteria for determining the state of the object, and using this data as the abnormal defective region data.
The secondary determination execution unit is a secondary visual inspection device that determines the state of the object based on the defective area abnormality data, which is the data corresponding to the defective area extracted by the data calculation unit . User interface and
The secondary visual inspection apparatus according to claim 1 or 2, further comprising a display control unit that displays visualized abnormal data, including at least the abnormal data of the defective area, on the user interface. The secondary visual inspection apparatus according to claim 3, further comprising a mode selection unit that selectively executes an automatic determination mode in which the quality of the object is determined by the secondary determination performed by the secondary determination execution unit, and a manual determination mode in which the quality of the object is determined based on operations performed by the user on the user interface while displaying the abnormality data on the user interface. The secondary visual inspection apparatus according to claim 4, wherein the mode selection unit executes one mode selected by user operation on the user interface from among the automatic determination mode and the manual determination mode. A primary visual inspection device that performs a primary determination to determine the state of an object in each of several different inspection areas, and outputs defective area information indicating the defective areas among the multiple inspection areas that were determined to be defective in the primary determination,
The apparatus comprises a secondary visual inspection apparatus according to claim 1 or 2,
The information acquisition unit is an appearance inspection system that acquires the defective area information output from the primary appearance inspection device. In a primary determination of the state of an object in each of several different inspection areas, the process includes acquiring defective area information indicating the defective area that was determined to be defective among the multiple inspection areas,
A step of calculating abnormality data data for a defective area that indicates an abnormality contained in the image of the object in the defective area indicated by the defective area information, using an abnormality data calculation unit.
The process includes a step in which a secondary determination is performed by a secondary determination execution unit to determine the state of the object based on the aforementioned defective area abnormality data,
The aforementioned abnormal data calculation unit,
An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination;
Multiple machine learning models are provided, each corresponding to one of the aforementioned multiple inspection areas, and each outputs inspection area anomaly data indicating anomalies contained in an image that includes only the corresponding inspection area.
The system includes a data calculation unit that calculates abnormal data for the defective region using a corresponding model among the plurality of machine learning models that corresponds to the defective region,
The data calculation unit extracts images from the object image that are included in the defective area as defective area images, thereby removing the inspection area that was not determined to be defective in the primary determination from the criteria for determining the state of the object, and using the inspection area abnormality data output by the corresponding model for the defective area image as the defective area abnormality data .
The secondary determination execution unit is a secondary visual inspection method that determines the state of the object based on the inspection area abnormality data output by the corresponding model as the defective area abnormality data with respect to the defective area image extracted by the data calculation unit . In a primary determination of the state of an object in each of several different inspection areas, the process includes acquiring defective area information indicating the defective area that was determined to be defective among the multiple inspection areas,
A step of calculating abnormality data data for a defective area that indicates an abnormality contained in the image of the object in the defective area indicated by the defective area information, using an abnormality data calculation unit.
The process includes a step in which a secondary determination is performed by a secondary determination execution unit to determine the state of the object based on the aforementioned defective area abnormality data,
The aforementioned abnormal data calculation unit,
An image acquisition unit acquires an object image, which is an image of the object in an imaging area including the plurality of inspection areas, captured in the primary determination;
A machine learning model that outputs imaging region anomaly data, which is an anomaly score obtained for each of the multiple pixels that make up the image captured in the aforementioned imaging region,
The machine learning model outputs abnormal imaging region data for the object image, and the data calculation unit extracts data corresponding to the defective region from this data , thereby removing the inspection region that was not determined to be defective in the initial judgment from the criteria for determining the state of the object, and using this data as the abnormal defective region data.
The secondary determination execution unit determines the state of the object based on the defective area abnormality data, which is the data corresponding to the defective area extracted by the data calculation unit, as a secondary visual inspection method.