JP7728536B2 - Method for detecting voids and cracks in joints connecting electronic components - Google Patents

Description

The present invention relates to a visualization device, visualization method, and computer program for non-destructively visualizing spaces, voids, cracks, etc. that occur inside or at the joints of objects such as electronic components, semiconductor elements, thin-film devices, thick-film devices, and electronic equipment using transmission X-ray images and ultrasonic microscope images.

The terminals of the electronic components and the electrodes of the substrate on which the electronic components are mounted are joined with solder 5. Voids 36 occur in the joints of the solder 5 due to preheating and reflow conditions. Furthermore, the electronic components and substrate repeatedly expand and contract due to thermal stress, and cracks 39 occur in the joints due to differences in their expansion rates.
In order to improve the yield of electronic circuit boards, it is important to detect voids 36 and cracks 39 in joints and other areas and to carry out quality inspections.
Conventionally, methods of visually inspecting transmitted X-ray images or ultrasonic microscope images have been used as non-destructive methods of inspecting joints such as solder 5 .

Visual inspection can detect slight changes in the image, but there is a large degree of variability in pass/fail judgments between inspectors, and time constraints limit the areas that can be inspected. It is difficult to accurately detect and measure voids 36 and cracks 39 with visual inspection, and the process takes a long time.

Patent Publication No. 2012-68209

In order to achieve high-quality, rapid non-destructive inspection, attempts have been made to automate and visualize the detection of voids 36 and cracks 39. However, image processing techniques that involve binarizing transmitted X-ray images or ultrasonic microscope images are commonly used for automation and visualization. However, this requires an appropriate setting of the binarization threshold, making it impossible to automatically detect voids 36 with different shadings and a wide variety of cracks 39 all at once.

In particular, it has been difficult to robustly and automatically detect voids 36 and cracks 39 in transmission X-ray images due to challenges such as reflections from vias, plating or vias, inconsistent lighting, and noise.

In other words, with conventional image processing methods, in order to detect voids 36 and cracks 39, it was necessary for a human to define image features such as brightness values one by one, making it impossible to detect a wide variety of voids 36 and cracks 39.

To address these challenges, the present invention takes into account the need for high-quality, rapid inspection of the interiors and joints of objects and employs a machine learning approach, including deep learning.
AI can learn image features that humans cannot define, and can detect a wide variety of voids 36 and cracks 39 with high accuracy, significantly reducing the inspection time required compared to conventional methods.

The present invention is a method for visualizing the inside of an object, characterized by comprising a first operation of classifying at least one of a transmission X-ray image acquired by an X-ray device and an ultrasound image acquired by an ultrasound device into a plurality of image patterns using a first machine learning model, and a second operation of detecting a predetermined region from the image patterns using a second machine learning model.

In the first operation, the performance of a newly trained machine learning model is periodically evaluated using a training program for image pattern classification, and if degradation is detected, the machine learning model is reconstructed and retrained.

It also has a training program for area detection, and the performance of newly trained machine learning models is periodically evaluated, and if degradation is detected, the model is reconstructed and retrained.

The method is characterized in that at least one of the following processes is performed on at least one of the transmitted X-ray image acquired by the X-ray device and the ultrasound image acquired by the ultrasound device: noise reduction, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing, to generate an image pattern for the second machine learning model.

The present invention is a computer program for visualizing the interior of an object, characterized by comprising a first process for classifying at least one of a transmission X-ray image acquired by an X-ray device and an ultrasound image acquired by an ultrasound device into a plurality of image patterns using a first machine learning model, and a second process for visualizing the interior of the object by detecting a predetermined region from the image patterns using a second machine learning model.

In the first process, the performance of the newly trained machine learning model is periodically evaluated using a pattern classification training program, and if degradation is detected, the model is reconstructed and retrained to ensure pattern classification performance.

It also has a training program for area detection, and the performance of newly trained machine learning models is regularly evaluated. If degradation is detected, the model is automatically reconstructed and retrained, thereby ensuring area detection performance.

The method is characterized in that at least one of the following processes is performed on at least one of the transmitted X-ray image acquired by the X-ray device and the ultrasound image acquired by the ultrasound device: noise reduction, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing, to generate an image pattern for the second machine learning model.

The present invention is a method for visualizing the inside of an object, which performs a first process by predicting and dividing the object into multiple regions during region detection, and then performs a second process that integrates the results of the first process. The first process detects overlapping regions and calculates the average of the respective processing results.

The present invention is a computer program for visualizing the interior of an object, which performs a first process of dividing the object into multiple regions using a prediction program for region detection, performs a second process of integrating the results of the first process, and calculates the average of the results of each process for overlapping regions detected in the first process.
The present invention builds an automatic void and crack detection system for joints such as solder 5 implemented on a cloud service, and provides a void rate and crack rate measurement service.

The automatic detection device 1 comprises a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16. Images captured by the ultrasound microscope 2 and X-ray CT scanner 3 are uploaded to the auxiliary memory unit 16 by a captured image upload program 103 and saved in a captured image (1) DB. Image processing involves binarizing transmitted X-ray images, and a machine learning approach including deep learning is used. Through use by users, transmitted X-ray images and ultrasound microscope images are collected, and automatic detection performance is improved through learning.

According to the present invention, conventional image analysis-based void and crack inspections of solder joints (5) were unable to simultaneously detect voids with different shadings or a wide variety of cracks.

By adopting a machine learning approach, including deep learning, this invention enables highly accurate automatic detection of voids 36 and cracks 39 inside materials, in joints, etc., thereby significantly reducing conventional inspection and detection times.

FIG. 1 is an explanatory diagram of voids and cracks in the interior of an object and in joints using the visualization device of the present invention. 1 is an explanatory diagram of a method for visualizing voids and cracks inside an object and joints according to the present invention. FIG. 1 is an explanatory diagram of a method for pattern classification of captured images in the method for visualizing the inside of an object and a joint of the present invention. FIG. 1A to 1C are explanatory diagrams illustrating a method for visualizing the inside of an object and a joint according to the present invention. FIG. 1 is a schematic diagram showing a transmission X-ray image of a void occurring in a solder portion. 1 is an explanatory diagram illustrating a method of machine learning for detecting voids and cracks occurring inside an object and at a joint according to the present invention. FIG. 1 is an explanatory diagram illustrating a method of machine learning for detecting voids and cracks occurring inside an object and at a joint according to the present invention. FIG. FIG. 1 is an explanatory diagram illustrating a method of dividing an image of a void or crack occurring inside an object or at a joint into patch images according to the present invention. FIG. 10 is an explanatory diagram illustrating a method for updating machine learning data in order to reconstruct and retrain a machine learning model that automatically detects voids and cracks that occur inside objects and at joints according to the present invention. 1A and 1B are explanatory diagrams of a method for visualizing the inside of an object and a joint of the present invention and a computer program therefor. 1A and 1B are explanatory diagrams of a method for visualizing the inside of an object and a joint of the present invention and a computer program therefor. 1A and 1B are explanatory diagrams of a method for visualizing the inside of an object and a joint thereof according to the present invention and a computer program therefor. 1 is an explanatory diagram of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor.

The present invention will now be described with reference to drawings showing the implementation thereof.

In the embodiments described in this specification, the solder joints of electronic components 4 are used as examples to facilitate understanding, but the present invention is not limited to this. The technical concept of the present invention can be applied to many objects and various types of structures, such as buildings, electrical equipment, and internal human diagnostics.

In the drawings used to explain the embodiments of the invention, elements having the same functions are given the same reference numerals, and descriptions thereof may be omitted. Furthermore, embodiments of the present invention may be combined with other embodiments. Furthermore, drawings may be enlarged, reduced, or omitted for purposes such as facilitating understanding and making the illustrations easier to understand.

FIG. 1 is a block diagram showing the configuration of an automatic void/crack detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and a joint between an electronic component 4 and solder 5, etc.
In the examples of this specification, the pattern 40 is described as being acquired by the X-ray CT device 3, but it may be replaced by an X-ray fluoroscopic observation device or an ultrasonic microscope 2.

The X-ray microscope (X-ray CT) 3 is a microscope used to observe the interior of an object non-destructively. Unlike an X-ray fluoroscopy device, it converts X-rays that pass through a sample into light and magnifies the light using an optical lens. X-rays have the ability to pass through matter. As X-rays pass through a sample, some of them are absorbed. The rate of absorption increases with the density (high atomic number) and thickness of the material, resulting in a decrease in the intensity of the transmitted X-rays.

If voids 36 or cracks 39 occur within the object, the X-ray transmittance of the voids 36 or cracks 39 increases, and the voids 36 and cracks 39 are displayed in the pattern 40.

The X-ray CT device 3 uses a computer to process X-ray transmission information from 360° (DEG.) of the sample being observed, and constructs 3D data of the sample. By narrowing the rotation pitch and obtaining more information, highly accurate 3D data can be obtained.

X-ray images are essentially two-dimensional images. Many voids 36 and cracks 39 occur, and the resulting voids 36 and cracks 39 are distributed three-dimensionally. Therefore, the acquired X-ray image shows a pattern 40 in which voids 36 and cracks 39 overlap.

The ultrasonic microscope 2 is a microscope used to non-destructively observe defects such as delaminations, voids, cracks, and foreign objects inside an object. When ultrasonic waves propagate through different materials, some of the waves are reflected and some are transmitted. If the object being observed has a delamination (void), for example, a strong reflected wave is detected. By scanning the object in a plane and marking the positions where this reflected wave is detected, it is possible to obtain a two-dimensional distribution of the delaminations (voids). Therefore, the ultrasonic microscope image produced from this will resemble a pattern 40, with overlapping voids 36 and cracks 39, just like an X-ray image.

The automatic void/crack detection (visualization) device 1 of the present invention has the function of automatically or by predetermined operations observing, inspecting, judging, determining, etc. the location and occurrence state of voids and cracks, and automatically visualizing, detecting, etc. voids and cracks. Hereinafter, the automatic void/crack detection (visualization) device will be referred to as the automatic detection device or visualization device.

The X-ray CT device 3 of the present invention utilizes the differences in the "ease of penetration" and "ease of absorption" of X-rays when they pass through an object, and performs reconstruction processing to obtain the internal structure of the object.

The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in the "ease of transmission" and "ease of reflection" that occur when ultrasonic waves propagate within the object.
The automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

The automatic detection device 1 can be configured, for example, as a desktop computer, a laptop personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources as a service via a computer network such as the Internet.

The control unit 11 can be configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 11 may also be configured to include a GPU (Graphics Processing Unit).

The main memory unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 11 to execute arithmetic processing.
The communication unit 13 has a function of communicating with the automatic detection device 1 via the network 17, and can send and receive required information.
The operation unit 14 is configured with, for example, a hardware keyboard, a mouse, a touch panel, and the like.

The display panel 15 can be configured as a liquid crystal panel or an organic EL (Electro Luminescence) display panel, etc. The control unit 11 controls the display panel 15 to display the required information.

The auxiliary storage unit 16 is a large-capacity memory, hard disk, etc., and stores programs necessary for the control unit 11 to execute processing, as well as a user authentication program 101, a case management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, and an analysis report editing and viewing program 115.

The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, and analysis report editing and viewing program 115 stored in the auxiliary storage unit 16 may be provided by a recording medium 18 on which each program is readably recorded.

The recording medium 18 is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a micro SD card, or a CompactFlash (registered trademark).

Each program recorded on the recording medium 18 is provided by communication via the communication unit 13. Alternatively, the program may be read from the recording medium 18 using a reading device (not shown) and installed in the auxiliary storage unit 16.
Next, how images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are stored in the auxiliary storage unit 16 will be described.

When the ultrasonic microscope 2 or X-ray CT scanner 3 is connected to the Internet, user authentication is automatically performed by the user authentication program 101 based on the setting information of the ultrasonic microscope 2 or X-ray CT scanner 3, and case registration is automatically performed by the case management program 102.

Images captured by the ultrasound microscope 2 and X-ray CT scanner 3 are uploaded to the auxiliary memory unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

On the other hand, if the ultrasonic microscope 2 or X-ray CT scanner 3 is not connected to the Internet, the images captured by the device are extracted and copied to a computer or other device connected to the Internet.

On the computer, a login operation is performed using the user authentication program 101, and a case is registered using the case management program 102. Next, an image is uploaded to the auxiliary storage unit 16 using the captured image upload program 103, where it can be saved in the captured image (1) DB 104.

In the auxiliary storage unit 16, uploaded image data is stored in the captured image (1) DB104, the results of automatically detected voids and cracks are stored in the analysis result (1) DB105, analysis reports containing the void and crack rates automatically measured from the automatic detection results are stored in the analysis report (1) DB106, and data sets created using the captured image (1) DB104 and analysis result (1) DB105 are stored in the machine learning data (1) DB107, each for each user, group, and case. Only users who have been authenticated by the user authentication program 101 can access each data.

The auxiliary storage unit 16 also stores an automatic detection program 111. This automatic detection program 111 uses a machine learning model (pattern classification) DB 108 and a machine learning model (area detection) DB 109 to automatically detect voids and cracks in images stored in the captured image (1) DB 104. Furthermore, the images stored in the captured image (1) DB 104 can be processed in batches of a certain amount. Details of this automatic detection program 111 will be described later.

The results automatically detected by the automatic detection program 111 are saved in the analysis result (1) DB 105. The saved analysis result data is read by the analysis result editing and viewing program 112, allowing users to edit and view the analysis results. This editing work allows users authorized by the user authentication program 101 to collaborate on the editing.

The automatic analysis report creation program 113 reads the images stored in the captured image (1) DB 104 and the analysis results stored in the analysis result (1) DB 105, automatically generates numerical data such as void rate and crack rate, and text data such as the cause of defects, according to a format specified by the user, compiles this into an analysis report, and saves it in the analysis report (1) DB 106.

When a newly created or updated analysis report is available, the analysis report notification program 114 notifies the user of the contents via email, message, chat service, etc. The user can then load the analysis report editing and viewing program 115 to edit and view the analysis report. This editing process allows collaborative editing by users authorized by the user authentication program.

The auxiliary storage unit 16 stores a machine learning program 110. This machine learning program 110 uses the captured image (1) DB 104 and the analysis result (1) DB 105 to create a machine learning data (1) DB, performs machine learning using this data set, and creates a machine learning model (pattern classification) DB 108 and a machine learning model (area detection) DB 109.

The datasets used for machine learning may be individual image databases and analysis result databases stored for each user, group, or project, or they may be shared and used.

The created machine learning model (pattern classification) DB108 and machine learning model (area detection) DB109 are used by the aforementioned automatic detection program 111. Therefore, the machine learning program 110 is executed at a different time from when the automatic detection program 111 is executed (for example, during a time period when the user is not using the system).

Figure 2 illustrates the details of the automatic detection program 111. First, a group of captured images 21 newly uploaded from an ultrasonic microscope 2 or an X-ray CT scanner 3 is read into the pattern classification prediction program 201.

The pattern classification prediction program 201 classifies image patterns. Examples of patterns to be classified include cases where the mounted electronic components are different, such as chip capacitors, resistors, and coils. Also, examples include cases where the pin shapes of IC packages are different, such as leaded and leadless types. Also, examples include cases where the types of boards are different, such as surface mount and through-hole. Also, examples include cases where the materials of the electronic components are different, such as copper, brass, and iron. Also, examples include cases where the imaging methods are different, such as optical microscopes and electron microscopes.
The group of captured images 21 is classified into, for example, pattern A, pattern B, and pattern C by the pattern classification prediction program 201 .

The image group of pattern A is loaded into the area detection and prediction program for pattern A 202. The image group of pattern B is loaded into the area detection and prediction program for pattern B 203. The image group of pattern C is loaded into the area detection and prediction program for pattern C 204, after which each program automatically detects the void 36 area and the crack 39 area. These analysis result images are saved as the analysis result (mask image) group 22.

The present invention focuses on the fact that there is an optimal area detection method for each pattern, and prepares an area detection program for each pattern. This is because the accuracy of area detection can be improved compared to when an area detection program is not prepared for each pattern. Therefore, a machine learning model must be prepared for each pattern.
FIG. 3 illustrates the process flow of the training program and the prediction program in the pattern classification described above.

The purpose of a training program for pattern classification is to create a machine learning model (pattern classification) DB108. For example, if a group of captured images contains two types of images, pattern A and pattern B, a machine learning model is created that can classify a given captured image as either pattern A or pattern B using a captured image (AB) DB301 and a pattern (AB) DB302 that can reference whether each captured image is pattern A or pattern B.

The data augmentation preprocessing program 303 reads the captured image (AB) DB 301 and pattern (AB) DB 302 and creates a dataset for machine learning. The created dataset is stored in the machine learning data (AB) DB 304.

Next, the machine learning (training) program 305 reads the machine learning data (AB) DB 304, performs training, and stores the trained machine learning model in the machine learning model (pattern classification) DB 108.
When training is performed, an existing machine learning model stored in the machine learning model (pattern classification) DB 108 may be used.

When training, machine learning models can sometimes be trained by assigning specific weights to machine learning data held by a specific user, group, or project, or to machine learning data that has been recently uploaded, thereby differentiating the importance of the data.

If a newly trained machine learning model is determined to have higher accuracy than an existing machine learning model during performance evaluation, it may automatically replace the existing machine learning model in the machine learning model (pattern classification) DB108.

If a newly trained machine learning model is determined to have lower accuracy than existing machine learning models during performance evaluation, the trained machine learning model may be automatically discarded.

As described above, a feature of the present invention is that the pattern classification training program periodically evaluates the performance of newly trained machine learning models, and if degradation is detected, the model is automatically reconstructed and retrained.

3 , in the prediction program for pattern classification, a preprocessing program 307 reads a captured image group 306 and performs image preprocessing. The preprocessed images are read into a machine learning (prediction) program 308, and a machine learning model trained by a training program is read from the machine learning model (pattern classification) DB 108 to perform pattern classification. The patterns classified for the captured image group are saved in a pattern classification result group 309.
FIG. 4 illustrates the process flow of the training program and prediction program in the above-mentioned region detection.

The purpose of the training program for area detection is to create a machine learning model (area detection) DB109. Using the captured image (A) DB401 and mask image (A) DB402, a machine learning model is created that can detect which areas of the captured image correspond to voids 36 and cracks 39.

The data augmentation preprocessing program 403 reads the captured image (A) DB 401 and the mask image (A) DB 402 and creates a dataset for machine learning.

The created dataset is stored in the machine learning data (A) DB 404. Thereafter, the machine learning (training) program 405 reads the machine learning data (A) DB 404, performs training, and stores the trained machine learning model in the machine learning model (area detection) DB 109.
When training is performed, an existing machine learning model stored in the machine learning model (area detection) DB 109 may be used.

When training, machine learning models can sometimes be trained by assigning specific weights to machine learning data held by a specific user, group, or project, or to machine learning data that has been recently uploaded, thereby differentiating the importance of the data.

If a newly trained machine learning model is determined to have higher accuracy than an existing machine learning model during performance evaluation, it may automatically replace the existing machine learning model in the machine learning model (pattern classification) DB108.

If a newly trained machine learning model is determined to have lower accuracy than existing machine learning models during performance evaluation, the trained machine learning model may be automatically discarded.

As described above, a feature of the present invention is that the performance of newly trained machine learning models is periodically evaluated using a training program for area detection, and if degradation is detected, the model is automatically reconstructed and retrained.

Specific training examples will be explained using Figures 5, 6, and 7. Note that the acquired image pattern 40 will be explained as a pattern 40 of voids 36, but it goes without saying that this is not limited to this, and for example, voids 36 may be replaced with cracks 39, etc. Furthermore, even in a pattern in which voids 36 and cracks 39 are mixed, voids 36 and cracks 39 can be distinguished and detected by training and learning.

Figure 5 shows a schematic diagram (pattern 40) of the solder part 5 and the occurrence of voids 36 captured in a transmission X-ray image. Normally, voids 36 are not circular as shown in Figure 5, but consist of multiple connected circles, with shading and reduced contrast.

In the following examples, the pattern 40 will be described as a pattern 40 acquired by a transmission X-ray device, but it goes without saying that the pattern 40 can also be acquired by an X-ray CT device.
Several thousand transmission X-ray images and images in which the void region 36 is masked (manually inspected data) are prepared.

These are randomly cut out as 256 x 256 (pixel) patch images (division 37), and then subjected to one or more processes, such as increasing noise, adjusting contrast, adjusting brightness, inverting brightness, smoothing, scaling, rotating, shifting horizontally or vertically, or partially masking (filling in black), or a combination of multiple processes, to augment the data and generate a dataset of hundreds of thousands of images.

As an example, in Figure 5, the solder portion 38 is shown as four patch images (sections 37a, 37b, 37c, and 37d). It is preferable that each section 37 has the same shape and area.

Each patch image (section 37) contains a void 36. As shown in Figure 6(a), section 37a is inverted to black and white and used as training data. As shown in Figure 6(b), section 37a is enlarged and used as training data. As shown in Figure 6(c), section 37a is reduced and used as training data.

The section 37a shown in FIG. 7(a) is rotated a predetermined angle (10° in the figure) to the left as shown in FIG. 6(b) to obtain training data. The section 37a is rotated a predetermined angle (10° in the figure) to the right as shown in FIG. 6(c) to obtain training data. The section 37a may also be inverted by line symmetry.

A feature of this invention is that even if there is only a small amount of data available for machine learning, it can self-generate a large dataset by expanding the amount of data by inverting, rotating, shrinking, and enlarging the divisions as described above.

Next, 70% of the patch images (category 37) are used as training data, and the remaining 30% are used as test data. Using an encoder-decoder model for segmentation, deep learning of the U-Net is performed using medical segmentation (especially cell segmentation, etc.). In other words, AI (artificial intelligence) functions are used. It is preferable to use 60% or more of the patch images (category 37) as training data. It is preferable to use 40% or less as test data.

The loss function during training is the Dice coefficient between the mask image as training data and the mask image as network output, and is set to Dice coefficient x (-1). The Dice coefficient uses the average number of elements in the two sets instead of the number of elements in the union of the two sets. During training, the machine learning model that minimizes the loss on the aforementioned test data is saved.

On the other hand, in the prediction program for area detection, the preprocessing program 407 reads the captured image group 406 and performs image preprocessing. The preprocessed images are read into the machine learning (prediction) program 408, which reads the machine learning model trained by the training program from the machine learning model (area detection) DB 109 and performs area prediction. The predicted area for the captured image is saved in the mask image group 409.

A specific prediction example will be explained using Figure 8. The captured transmission X-ray image is divided into 256 x 256 (pixel) patch images at equal intervals. Alternatively, the patch image can be divided into multiple parts. In Figure 8, the image is divided vertically and horizontally starting from section 37e in the upper left corner of the image. However, the patch image can be divided by overlapping sections 37f, 37g, and 37h at equal intervals to match the width and height of the captured transmission X-ray image. Next, each patch image is input into a trained machine learning model.

From the trained machine learning model, a probability map (values between 0 and 1) indicating the presence of a void at each pixel location is obtained. To obtain a binary map (mask image), the obtained probability map is binarized using a set threshold of 0.5 (1 indicates a void, 0 indicates the background).

The resulting 256 x 256 (pixel) binary maps are combined (as divided into sections 37e, 37f, 37g, and 37h in Figure 8) to generate an overall binary map (mask image) corresponding to the original transmitted X-ray image.

Note that 256 is the 8th power of 2. It is preferable that the pixels be 2 to the nth power (n is an integer equal to or greater than 2). Finally, as post-processing, a binary map (mask image) is optimized using a conditional random field (CRF) of the transmitted X-ray image.
As described above, the feature of the present invention is that processing is performed by dividing into a plurality of small regions (sections) 37, and the respective processing results are integrated to obtain a single processing result.

For example, even if a void 36 requires that the object be contained within the boundaries of a section 37, it may be possible to clearly detect it in another section 37. For overlapping detected areas, the processing results for each batch image are averaged, and pixels exceeding a threshold of 0.5 are considered voids, enabling automatic, highly robust detection of voids.

For example, even if voids 36 cannot be detected in an image of section 37, they may be detectable if the image is rotated 90 degrees. By averaging the processing results for images of section 37 rotated multiple times and determining pixels that exceed a threshold of 0.5 as voids 36, voids 36 can be detected automatically with high robustness.

A method for updating machine learning data to rebuild and retrain a machine learning model will be described with reference to FIG. 9. The results of the automatic detection of voids 36 described above are not always accurate. The user can check and edit the results of the automatic detection of voids 36 using the analysis result editing and viewing program 112 of this system.
To improve the automatic detection performance, it is necessary to incorporate the user's confirmations and edits into the machine learning data of this system and update it.

Figure 9(a) shows a case where a group of captured images 901 is input into an area detection and prediction program 902, and a group of analysis results (mask images) 903 is output, but the user has not confirmed these analysis results. In this case, the confirmation status and editing results (mask images) DB 904 is registered as "unconfirmed," and the data is not incorporated into or updated for machine learning.

Figure 9(b) shows the case where, after a captured image group 901 is input into an area detection and prediction program 902 and an analysis result (mask image) group 903 is output, the user views the analysis results for a specific period of time or presses a confirmation button or an analysis result output button. In this case, the confirmation status and editing results (mask image) DB 904 is registered as "confirmed," and the data is incorporated into and updated for machine learning.

FIG. 9C illustrates a case where a captured image group 901 is input into an area detection and prediction program 902, an analysis result (mask image) group 903 is output, and then the user edits the analysis result. In this case, the confirmation status and editing result (mask image) DB 904 is registered as "edited," the analysis result is determined to be incorrect, and the analysis result is weighted and incorporated into the machine learning data for update. Furthermore, the greater the difference between the analysis result (mask image) group 903 and the results edited by the user, the more weighted the data is incorporated into the machine learning data for update. One method of weighting is to expand the number of data items by inverting black and white, rotating, shrinking, or enlarging the captured image group 901 and editing result (mask image) DB 904 in FIG. 9C as described above, and then incorporate the expanded data into the machine learning data.
As described above, the captured image group 901 is classified and registered as unconfirmed, confirmed, edited, etc., and is incorporated into machine learning data for updating.
By using the above method of updating machine learning data, the machine learning model is automatically reconstructed and retrained, improving automatic detection performance.

The present invention uses a machine learning approach, including deep learning. By using this approach, voids 36 and cracks 39 can be detected well and accurately.

The above examples of the present invention have described the detection, visualization device, and visualization method of voids 36 inside a material. Patterns 40 of cracks 39 can also be obtained using an X-ray CT scanner 3 or the like. Although images of voids 36 and cracks 39 are different, by training them as cracks 39, it becomes possible to detect cracks 39. Therefore, it goes without saying that matters described as voids 36 in the examples of this specification can be replaced with cracks 39.

Furthermore, the pattern 40 can be obtained not only as a transmission X-ray image using an X-ray CT scanner 3 or the like, but also by using, for example, an ultrasonic microscope 2 to obtain a pattern 40 corresponding to the transmission X-ray image. Therefore, it goes without saying that in the device of the present invention, an X-ray device such as an X-ray CT scanner can be replaced with an ultrasonic device 2 such as an ultrasonic microscope.

Furthermore, by combining an X-ray device such as an X-ray CT device 3 with an ultrasonic microscope 2 to obtain the pattern 40, the detection accuracy of voids 36, cracks 39, etc. is improved.

The above embodiments are directed to a method and apparatus for detecting voids 36, cracks 39, etc. in a transmitted X-ray image, but the present invention is not limited to this. It goes without saying that the present invention can be used to non-destructively analyze a wide variety of objects, such as air bubbles in adhesive resin, spaces in concrete blocks, metal inclusions in iron parts, and fat particles in the internal organs of living organisms.
10, 11, 12 and 13 are explanatory diagrams of a business model using the automatic detection method and computer program of the present invention.

Figure 10 is an explanatory diagram of the business model in the first embodiment. Company A provides or sells to Customer A, who owns images captured with an ultrasonic microscope 2, X-ray CT scanner 3, or the like, a program (hereinafter referred to as the "automatic void/crack detection program") that includes the user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing/viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, and analysis report editing/viewing program 115 shown in Figure 1.

Customer A will be able to install and use the Boyd Crack Automatic Detection Program on desktop computers, laptops, tablets, and smartphones. Company A will receive license fees for the Boyd Crack Automatic Detection Program from Customer A.

Figure 11 is an explanatory diagram of a business model in the second embodiment. Company A provides or sells a group of images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc. owned by Company A, and an automatic void/crack detection program to Customer A, who does not own images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc.

Customer A can install and use the void/crack automatic detection program on desktop computers, laptops, tablets, and smartphones. Company A receives image capture fees and license fees for the void/crack automatic detection program from Customer A.

Figure 12 is an explanatory diagram of a business model in the third embodiment. Company A runs an automatic void/crack detection program on a cloud server for Customer A, who owns images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc.

Customer A can use the void/crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives a usage fee for the void/crack automatic detection program from Customer A.

Figure 13 is an explanatory diagram of a business model in the fourth embodiment. For Customer A, who does not own images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc., Company A prepares a group of images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc. owned by Company A, and runs an automatic void/crack detection program on a cloud server.

Customer A can use the void/crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives image capture fees and usage fees for the void/crack automatic detection program from Customer A.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims. It goes without saying that the matters or contents described in this specification and drawings can be mutually combined.

1. Void and crack automatic detection device 2. Acoustic microscope 3. X-ray CT device 4. Electronic components (measurement sample)
5 Solder 11 Control unit 12 Main memory unit 13 Communication unit 14 Operation unit 15 Display panel 16 Auxiliary memory unit 17 Network 18 Recording medium 21 Photographed image group 22 Analysis result (mask image) group 31 Pattern classification training program 32 Pattern classification prediction program 36 Void 37 Classification (processing unit)
38 Solder part 39 Crack 40 Pattern 41 Area detection training program 42 Area detection prediction program 101 User authentication program 102 Case management program 103 Photographed image upload program 104 Photographed image (1) DB
105 Analysis results (1) DB
106 Analysis Report (1) DB
107 Machine Learning Data (1) DB
108 Machine learning model (pattern classification) DB
109 Machine learning model (area detection) DB
110 Machine learning program 111 Automatic detection program 112 Analysis result editing and viewing program 113 Analysis report automatic creation program 114 Analysis report notification program 115 Analysis report editing and viewing program 201 Pattern classification prediction program 202 Pattern A area detection and prediction program 203 Pattern B area detection and prediction program 204 Pattern C area detection and prediction program 301 Photographed image (A-B) DB
302 Pattern (AB) DB
303 Data augmentation preprocessing program 304 Machine learning data (AB) DB
305 Machine learning (training) program 306 Photographed image group 307 Preprocessing program 308 Machine learning (prediction) program 309 Pattern classification result group 401 Photographed image (A) DB
402 Mask Image (A) DB
403 Data augmentation preprocessing program 404 Machine learning data (A) DB
405 Machine learning (training) program 406 Photographed image group 407 Preprocessing program 408 Machine learning (prediction) program 409 Analysis result (mask image) group 901 Photographed image group 902 Area detection prediction program 903 Analysis result (mask image) group 904 Confirmation status and editing result (mask image) DB

Claims (6)

A method for detecting voids and cracks in a joint connecting electronic components, comprising:
a first process of classifying at least one of the X-ray image of the joint and the ultrasound image of the joint into image patterns based on at least one of the type of the electronic component, the shape of the connection portion of the electronic component, the type of board on which the electronic component is mounted, and an imaging method;
A method for detecting voids and cracks in joints that connect electronic components, characterized by having a second process of detecting void and crack areas in the joints in the classified image patterns and saving the void and crack areas as analysis result images. A method for detecting voids and cracks in a joint connecting electronic components, comprising:
a first process of classifying at least one of the captured images of the X-ray image of the joint and the ultrasound image of the joint into image patterns based on at least one of the type of the electronic component, the shape of a connection portion of the electronic component, the type of board on which the electronic component is mounted, and the type of imaging method;
a second process of detecting void/crack regions of the joint in the classified image pattern and saving the void/crack regions as an analysis result image;
A method for detecting voids and cracks in joints connecting electronic components, characterized in that it includes a third process of confirming or editing the void/crack area in the analysis result image and updating the analysis result image. A method for detecting voids and cracks in a joint connecting electronic components, comprising:
a first process of classifying at least one of the captured images of the X-ray image of the joint and the ultrasound image of the joint into image patterns based on at least one of the type of the electronic component, the shape of a connection portion of the electronic component, the type of board on which the electronic component is mounted, and the type of imaging method;
a second process of detecting a void/crack region of the joint in the classified image pattern and saving the void/crack region as an analysis result image;
performing at least one or more processes of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing on the captured image to generate a first data set, and performing the first processing based on the first data set;
A method for detecting voids and cracks in joints connecting electronic components, characterized in that the captured image is subjected to at least one of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing to generate a second data set, and a second processing is performed based on the second data set. performing at least one or more processes of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing on the captured image to generate a first data set, and performing the first processing based on the first data set;
performing at least one or more processes of noise processing, contrast processing, brightness processing, smoothing processing, enlargement processing, reduction processing, rotation processing, translation processing, and mask processing on the captured image to generate a second data set, and performing a second process based on the second data set;
a performance evaluation of image pattern classification of the first process is performed based on the first data set;
A method for detecting voids and cracks in joints connecting electronic components according to claim 1 or claim 2, characterized in that a performance evaluation of the second process for detecting voids and crack areas is carried out based on the second data set. A method for detecting voids and cracks in joints connecting electronic components according to claim 1, 2, or 3, characterized in that images of multiple regions are generated from the captured image, and void and crack regions in the images of the multiple regions are detected and combined to create an image, which is then saved as the analysis result image. A method for detecting voids and cracks in joints connecting electronic components, as described in claim 1, 2, or 3, characterized in that the captured image is a combination of an X-ray image of the joint and an ultrasound image of the joint.