JP7754486B2 - Visualization device for voids and cracks in joints and method for visualizing voids and cracks in joints - Google Patents

Description

The present invention relates to a visualization device and visualization method that non-destructively visualizes spaces, voids, and cracks that occur inside or at joints of objects such as electronic components, semiconductor elements, thin film devices, thick film devices, and electronic equipment using transmission X-ray images and ultrasonic microscope images, as well as a computer program, a method for operating the computer program, a display method, etc.
The present invention also relates to a method for selecting and applying a machine learning model, a method for creating a predictive model, and a method for using a predictive model.

The terminals of the electronic components are soldered to the electrodes of the substrate on which the electronic components are mounted. Voids 36 occur at the joints of the solder 5 due to preheating and reflow conditions. Furthermore, the electronic components and substrate repeatedly expand and contract due to thermal stress, and the difference in their expansion rates causes cracks 39 to occur at the joints.
In order to improve the yield of electronic circuit boards, it is important to non-destructively detect voids 36 and cracks 39 in joints and inspect the quality.
Conventionally, methods of visually inspecting transmitted X-ray images or ultrasonic microscope images have been used as non-destructive methods of inspecting joints such as solder 5 .

Visual inspection can detect slight changes in the image, but there is a large degree of variability in pass/fail judgments between inspectors, and time constraints limit the areas that can be inspected. It is difficult to accurately detect and measure voids 36 and cracks 39 with visual inspection, and the process takes a long time.

Patent Publication No. 2012-68209

In conventional image inspection methods using machine learning models, in order to robustly detect voids 36 and cracks 39 with various characteristics, it was necessary to prepare multiple machine learning models tailored to the object being photographed, rather than using a single machine learning model.
Therefore, when an image of an unknown subject is input, it is difficult to automatically determine which machine learning model should be applied to make a prediction.

In response to these challenges, the present invention takes an approach that allows users to select the most appropriate machine learning model from multiple machine learning models using a dedicated display method. By selecting the machine learning model that produces the best prediction results, users can detect a wide variety of voids 36 and cracks 39 with high accuracy, significantly reducing the inspection time required compared to conventional methods.

The present invention relates to a device for visualizing the interior of an object, a method for visualizing the interior of an object, and a computer program for the device for visualizing the interior of an object, which are characterized by selecting the most appropriate machine learning model from among multiple machine learning models used to visualize the interior of an object.

The present invention is characterized by the fact that it uses a pattern classification program 133 and an image scale adjustment program 134 to classify and adjust images of unknown subjects, thereby narrowing down the optimal machine learning model from among multiple machine learning models.

The present invention is characterized by the fact that, in the process of selecting the most appropriate model from the group of candidate machine learning models limited as described above, the model selection program 504 of the present invention has a display method that allows users who do not have special knowledge of machine learning models to easily select the most appropriate machine learning model.

The present invention is characterized by taking an approach that uses a user editing status confirmation program 135 to indirectly obtain the user's evaluation of the prediction results, thereby retaining which machine learning model should be applied when the same type of image is input, thereby improving the convenience of image inspection for the user.
The automatic detection device 1 also includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

Images captured by the ultrasound microscope 2 and X-ray CT scanner 3 are uploaded to the auxiliary memory unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

The present invention narrows down the optimal machine learning model from among multiple machine learning models used to visualize the interior of an object by performing a first operation of classifying the type of image data using a machine learning model from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device, and a second operation of adjusting the position, angle, and scale of the object in the image data.

The present invention also provides a display method that allows a user with no special knowledge of machine learning models to easily select the optimal machine learning model from a limited group of candidate models through a first operation of classifying the type of image data from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device, and a second operation of adjusting the position, angle, and scale of the object in the image data.

Furthermore, the present invention improves the convenience of image inspections for users by performing a first operation of classifying the type of image data using a machine learning model from at least one of a transmission X-ray image acquired by an X-ray device, a cross-sectional image acquired by an X-ray CT device, and an ultrasound image acquired by an ultrasound device; a second operation of adjusting the position, angle, and scale of the object in the image data; and a third operation of visualizing the interior of the object in the image data using a machine learning model selected by the user. After this, the user's evaluation of the visualization results is indirectly obtained, and a decision on whether the selected machine learning model should be applied next time is retained.

By using multiple machine learning models rather than a single one and then selecting the most appropriate one, it becomes possible to automatically detect a wide variety of voids 36 and cracks 39 in electronic components with high accuracy, significantly reducing the inspection or detection time required.

1 is an explanatory diagram of a visualization device for the inside and joints of an object according to the present invention. 10A and 10B are explanatory diagrams of visualized observation images of solder joints and the like. 1 is an explanatory diagram of a method for detecting void and crack regions from a captured image in a method for visualizing the inside of an object and a joint of the present invention. 1 is an explanatory diagram of a method for detecting void and crack regions from a captured image in a method for visualizing the inside of an object and a joint of the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. FIG. 10 is an explanatory diagram illustrating a method for selecting a machine learning model (area detection) when automatically detecting voids and cracks that occur inside an object or at a joint in accordance with the present invention. 1A to 1C are explanatory diagrams illustrating a method for visualizing the inside of an object and a joint according to the present invention. 1A and 1B are explanatory diagrams of a method for visualizing the inside of an object and a joint of the present invention and a computer program therefor. 1A and 1B are explanatory diagrams of a method for visualizing the inside of an object and a joint of the present invention and a computer program therefor. 1 is an explanatory diagram of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor. 1 is an explanatory diagram of a method for visualizing the inside of an object and a joint of the object according to the present invention and a computer program therefor.

The present invention will be described below with reference to the drawings showing the embodiments. In the embodiments described in the specification, the solder joints of electronic components 4 will be used as examples to facilitate understanding, but the present invention is not limited to this.

The technical concept of the present invention can be applied to many objects and a wide variety of structures and shapes, such as buildings, electrical equipment, electronic devices, mechanical components, mechanical equipment, electronic components, electronic component mounting boards, mechanical components, mechanical jigs, and internal human body diagnosis.
In the drawings for explaining the embodiments of the invention, elements having the same functions are given the same reference numerals, and descriptions thereof may be omitted.
In addition, the embodiments of the present invention can be combined with each other, and some of the embodiments can be combined with other embodiments.
In addition, for the purpose of facilitating understanding and facilitating illustration, the drawings may be enlarged, reduced, or omitted.

Figure 1 is a block diagram showing the configuration of an automatic void/crack detection (visualization) device 1, an ultrasonic microscope 2, an X-ray CT (Computed Tomography) device 3, and a joint between an electronic component 4 and solder 5, etc.

In the examples of this specification, the pattern is described as being acquired using an X-ray CT scanner 3, but this may be replaced with an X-ray fluoroscopic observation device, an ultrasonic microscope 2, or other non-destructive observation device.

Furthermore, the device for acquiring the pattern is not limited to an ultrasonic microscope 2 or an X-ray CT scanner 3, but may be, for example, an infrared thermometer that measures a pattern consisting of heat distribution, or a gamma ray measurement device that measures or observes the internal state of an object using gamma rays.

The X-ray microscope (X-ray CT, etc.) 3 is a microscope that non-destructively observes the inside of an object. Unlike an X-ray fluoroscopy observation device, it converts X-rays that pass through the sample into light, etc., and magnifies them using optical lenses, etc.

X-rays have the property of penetrating matter. As they pass through a sample, some of the X-rays are absorbed. The absorption rate increases as the material's density (atomic number) increases and its thickness increases, resulting in a decrease in the intensity of the penetrating X-rays (see, for example, Figure 17 and its accompanying explanation).

If voids 36 or cracks 39 occur within the object, the X-ray transmittance of the voids 36 or cracks 39 increases, and the voids 36 or cracks 39 will appear in the acquired pattern.

The X-ray CT device 3 uses a computer to process X-ray transmission information from 360° (DEG.) of the sample being observed, and constructs 3D data of the sample. By narrowing the rotation pitch and obtaining more information, highly accurate 3D data can be obtained.

X-ray images are basically two-dimensional images. Many voids 36 and cracks 39 occur, and these voids 36 and cracks 39 are distributed three-dimensionally. Therefore, the acquired X-ray image shows a pattern in which the voids 36 and cracks 39 overlap.

The ultrasonic microscope 2 is a microscope that non-destructively observes defects such as delamination, voids, cracks, and foreign matter inside an object. When ultrasonic waves propagate through different materials, some of the waves are reflected and some are transmitted. If the object being observed has a delamination (void), for example, a strong reflected wave will be detected. By scanning the object in a plane and marking the positions where this reflected wave is detected, a two-dimensional distribution of the delamination (void) can be obtained.
The ultrasonic microscope image, like the X-ray image, is an image of a pattern in which voids 36 and cracks 39 are superimposed.

The automatic void/crack detection (visualization) device 1 of the present invention observes, inspects, judges, and determines the location and occurrence state of voids and cracks automatically or through predetermined operations, and also has the function of automatically visualizing and detecting voids and cracks. Hereinafter, the automatic void/crack detection (visualization) device will be referred to as the automatic detection device.

The X-ray CT device 3 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in the "ease of penetration" and "ease of absorption" of X-rays when they pass through the object.

The ultrasonic microscope 2 of the present invention can obtain the internal structure of an object by performing reconstruction processing using the differences in the "ease of transmission" and "ease of reflection" that occur when ultrasonic waves propagate within the object.
The automatic detection device 1 includes a control unit 11 that controls the entire device, a main memory unit 12, a communication unit 13, an operation unit 14, a display panel 15, and an auxiliary memory unit 16.

The automatic detection device 1 can be configured, for example, as a desktop computer, a laptop personal computer, a tablet, a smartphone, or a computer in a cloud environment that provides computer resources as a service via a computer network such as the Internet.

The control unit 11 can be configured with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 11 may also be configured to include a GPU (Graphics Processing Unit).

The main memory unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data required for the control unit 11 to execute arithmetic processing.
The communication unit 13 has a function of communicating with the automatic detection device 1 via the network 17, and can send and receive required information.
The operation unit 14 is configured with, for example, a hardware keyboard, a mouse, a touch panel, and the like.

The display panel 15 can be configured as a liquid crystal panel or an organic EL (Electro Luminescence) display panel, etc. The control unit 11 controls the display panel 15 to display the required information.

The auxiliary storage unit 16 is a large-capacity memory, hard disk, etc., and stores programs necessary for the control unit 11 to execute processing, as well as a user authentication program 101, a case management program 102, a captured image upload program 103, a machine learning program 110, an automatic detection program 111, an analysis result editing and viewing program 112, an automatic analysis report creation program 113, an analysis report notification program 114, an analysis report editing and viewing program 115, a pattern classification program 133, an image scale adjustment program 134, and a user editing status confirmation program 135.

Figure 17 is an explanatory diagram of the method of the present invention for visualizing the inside of an object and joints. This is an explanatory diagram for determining or adjusting the acquisition range when the object to be measured or observed is an electronic component 4.

The distance between the X-ray focus 162 and the X-ray detector is referred to as the distance 161 from the X-ray focus to the X-ray detector. The distance between the X-ray focus 162 and the electronic component 4 is referred to as the distance 160 from the X-ray focus to the sample.

X-rays are emitted from the X-ray focal point 162. The size of the enlarged projection image 164 projected onto the X-ray detector 163 changes by adjusting or setting the distance 160 from the X-ray focal point (electronic component 4) to the sample, which is the X-ray focal point 162. Also, the distance 161 from the X-ray focal point to the X-ray detector is adjusted or set.

When data is acquired using an X-ray CT scanner such as an X-ray CT scanner, magnification is the ratio of the size of the enlarged projection image 164 on the detector input surface of the X-ray CT scanner to the size of the sample, and can be calculated by dividing the distance 161 from the X-ray focus 162 to the X-ray detector 163 by the distance 160 from the X-ray focus to the sample. As the distance 160 from the X-ray focus 163 to the sample increases, the magnification decreases and the acquisition range becomes wider.

By adjusting or changing the size of the enlarged projection image 164 on the X-ray detector, it is possible to obtain many captured images, machine learning data, etc. Furthermore, by rotating, tilting, and moving the electronic component 4 as a sample left and right, it is possible to obtain many captured images, machine learning data, etc.

The user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing and viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, analysis report editing and viewing program 115, pattern classification program 133, image scale adjustment program 134, and user editing status confirmation program 135 stored in the auxiliary memory unit 16 may be provided by a recording medium 18 on which each program is readably recorded.

The recording medium 18 is a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a micro SD card, or a CompactFlash (registered trademark).
Alternatively, the program may be read from the recording medium 18 using a reading device (not shown) and installed in the auxiliary storage unit 16 .
Next, how images captured by the ultrasonic microscope 2 and the X-ray CT scanner 3 are stored in the auxiliary storage unit 16 will be described.

When the ultrasonic microscope 2 or X-ray CT scanner 3 is connected to the Internet, user authentication is automatically performed by the user authentication program 101 based on the setting information of the ultrasonic microscope 2 or X-ray CT scanner 3, and case registration is automatically performed by the case management program 102.

Therefore, images captured by the ultrasound microscope 2 and X-ray CT scanner 3 are uploaded to the auxiliary storage unit 16 by the captured image upload program 103 and stored in the captured image (1) DB.

On the other hand, if the ultrasonic microscope 2 or X-ray CT scanner 3 is not connected to the Internet, the images captured by the device are extracted and copied to a computer or other device connected to the Internet.

On the computer, a login operation is performed using the user authentication program 101, and a case is registered using the case management program 102. Next, images are uploaded to the auxiliary storage unit 16 using the captured image upload program 103, and can then be stored in the captured image (1) DB 104.

In the auxiliary storage unit 16, uploaded image data is stored in the captured image (1) DB 104, and the results of automatically detected voids and cracks are stored in the analysis results (1) DB 105, each for each user, group, and case.

Analysis reports containing the void and crack rates automatically measured from the automatic detection results are saved in the Analysis Report (1) DB 106 for each user, group, and case.

Data sets created using the captured image (1) DB 104 and the analysis results (1) DB 105 are stored in the machine learning data (1) DB 107 for each user, group, and case.
Only users who have been authenticated by the user authentication program 101 can access each piece of data.

The auxiliary storage unit 16 stores an automatic detection program 111. This automatic detection program 111 uses the machine learning model (area detection) DB 109 to automatically detect cracks and automatically measure the crack rate for images stored in the captured image (1) DB 104.

In addition, images stored in the captured image (1) DB 104 can be processed in batches of a certain amount. Details of this automatic detection program 111 will be described later.

The results of automatic detection and measurement by the automatic detection program 111 are stored in the analysis result (1) DB 105. The stored analysis result data is read by the analysis result editing and viewing program 112, allowing users to edit and view the analysis results. This editing work allows users authorized by the user authentication program 101 to edit together.

The automatic analysis report creation program 113 reads the images stored in the captured image (1) DB 104 and the analysis results stored in the analysis result (1) DB 105, automatically generates numerical data such as void rate and crack rate, and text data such as the cause of defects, in accordance with the format specified by the user, compiles this into an analysis report, and saves it in the analysis report (1) DB 106.

When a newly created or updated analysis report is available, the analysis report notification program 114 notifies the user of its contents via email, message, chat service, etc. The user can then load the analysis report editing and viewing program 115 to edit and view the analysis report. This editing process allows users authorized by the user authentication program to collaborate on the editing.

The auxiliary storage unit 16 stores a machine learning program 110. This machine learning program 110 uses the captured image (1) DB 104 and the analysis result (1) DB 105 to create a machine learning data (1) DB, performs machine learning using this data set, and creates a machine learning model (pattern classification) DB 108 and a machine learning model (area detection) DB 109.

The datasets used for machine learning may be individual image databases and analysis result databases stored for each user, group, or project, or they may be shared and used.

The created machine learning model (pattern classification) DB 108 and machine learning model (area detection) DB 109 are used by the pattern classification program 133 and the automatic detection program 111. Therefore, the machine learning program 110 is executed at a different time (for example, during a time period when the user is not using the system) from the time when the pattern classification program 133 and the automatic detection program 111 are executed.

The pattern classification program 133 is a program that uses the machine learning model (pattern classification) DB 108 to classify electronic components by the patterns shown on the vertical axis of Figure 2, particularly by the shape, form, structure, etc. of the electronic components.

The image scale adjustment program 134 is a program that adjusts for differences in scale, particularly enlargement/reduction, rotation, and position, as shown on the horizontal axis of Figure 2. Adjustment methods include using the scale value 501 (e.g., the number of micrometers equivalent to one pixel) written in the image, the BGA ball diameter 502, or the component length 503, and specific adjustment methods are defined according to the patterns classified by the pattern classification program 133. This image scale adjustment program 134 reduces variation in the images input to the subsequent automatic detection program 111, contributing to improved accuracy of the automatic detection results.

The user editing status confirmation program 135 is a program that checks the user's editing status in the analysis result editing and viewing program 112, etc., such as the amount of editing that has been done on the output results of the automatic detection program 111, the viewing time, etc., and determines whether to retain the selection results of the model selection program 504, which will be described later.

Figure 3 is an explanatory diagram showing a flowchart for automatically detecting voids 36 and cracks 39 in a group of captured images using a machine learning model (pattern classification) DB 108 and a machine learning model (area detection) DB 109 created by a machine learning program 110.

First, a group of captured images is input and uploaded using at least one of the ultrasonic microscope 2 and the X-ray CT scanner 3. An example of uploading is saving the images in the case management program 102. The upload destination is not limited to a cloud server, and could be, for example, a smartphone with the functionality of the automatic detection device 1.

Next, the pattern classification program 133 uses the machine learning model (pattern classification) DB 108 to classify the captured images into patterns based on the shape, form, structure, etc. of the electronic components.

The captured images classified into each pattern are then adjusted by the image scale adjustment program 134 using a specific adjustment method depending on each pattern, such as enlarging/reducing, rotating, and positioning.

Next, for the adjusted captured images, an appropriate machine learning model (area detection) is selected from the machine learning model (area detection) DB 109, and the automatic detection program 111 automatically detects the areas of voids 36 and cracks 39 in the images. Details of the automatic detection program 111 will be described later.

The automatically detected results can be edited and viewed by the user using the analysis result editing and viewing program 112, and the status of the editing and viewing is monitored by a user editing status checking program, which will be described in detail later.
FIG. 4 is an explanatory diagram showing details of the automatic detection program 111 and the user editing status confirmation program 134. As shown in FIG.

First, the captured image output by the image scale adjustment program 134 is checked to see if a machine learning model (area detection) exists for the case to which the captured image belongs. As mentioned above, each captured image is saved for each user, group, and case, and some of these have a machine learning model (area detection) and some do not.

This is because, as mentioned above, the machine learning program 110 that creates the machine learning model (area detection) is executed at a different time than when the pattern classification program 133 is executed (for example, during a time period when the user is not using the system).

If the answer to the question "Does the case to which this image belongs have a machine learning model (area detection)?" in Figure 4 is "Yes," that is, if the case to which the captured image belongs has a machine learning model (area detection) (for the sake of explanation, let's call this case case (1)), the area detection prediction program 505 uses the machine learning model (area detection) (1) to automatically detect voids 36 and cracks 39 in the image, and saves the results in the analysis result (mask image) group 22.

On the other hand, if the answer to the question "Does the case to which this image belongs have a machine learning model (area detection)?" in Figure 4 is "No," that is, if the case to which the captured image belongs does not have a machine learning model (area detection) (for the sake of explanation, let's call this case (3)), the model selection program 504 selects a machine learning model (area detection) for automatically detecting voids 36 and cracks 39 in the image from the machine learning model (area detection) DB 109. The selection method will be described later.

As in the previous case, the area detection and prediction program 505 uses the machine learning model (area detection) selected by the user to automatically detect voids 36 and cracks 39 in the image, and saves the results in the analysis results (mask image) group 22.

The analysis result editing and viewing program 112 then reads the results, allowing the user to edit and view them. At that time, the user editing status confirmation program 134 checks the extent to which the results have been edited and viewed.

If the answer to the question "Have the analysis results been edited to a certain extent?" in Figure 4 is "No," that is, if the user editing status confirmation program 134 determines that the analysis results have not been edited much or that the viewing time has been short, it determines that the user's selection of the machine learning model (area detection) by the aforementioned model selection program 504 was appropriate, and the machine learning model copy program 506 copies a reference to the machine learning model (area detection) selected by the user for its own case (case (3)).

This copy allows users to use the same machine learning model they previously selected when working with images from their own case (case (3)) in the future, without having to select a model. This improves the convenience of image inspection for users.

On the other hand, if the answer to the question "Have the analysis results been edited to a certain extent?" in Figure 4 is "Yes," that is, if the user editing status confirmation program 134 determines that the analysis results have been edited frequently or that the viewing time has been long, it determines that the user's selection of the machine learning model (area detection) by the model selection program 504 described above was inappropriate, and does not perform the above-mentioned copying.
Therefore, in the future, when handling captured images for one's own case (case (3)), the appropriate machine learning model (area detection) will be selected again.

Figure 5 is an explanatory diagram of the operation screen of the model selection program 133. Figure 5 etc. shows an example in which a region detection result area 137 is provided, and four region detection result areas 137 (region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137d).

The area detection result area 137 may be one area in the display area 40, or may be four or more areas in the display area 40. One or more area detection result areas 137 are generated in the display area 40 in accordance with the operation state, the usage situation, or the processing state.
The display area 40 is displayed on the display panel 15. It is also displayed online or offline on a display device used by the user.

In the following explanation, voids 36 will be used as an example of the pattern to be displayed in the region detection result area 137, but it goes without saying that this is not limited to voids 36 and other evaluation patterns such as cracks 39 may also be used.

The present invention is characterized by the fact that, in the process of selecting the most appropriate model from the group of candidate machine learning models limited as described above, the model selection program 504 of the present invention has a display method that allows users who do not have special knowledge of machine learning models to easily select the most appropriate machine learning model.

The present invention is characterized by its approach of indirectly obtaining the user's evaluation of the prediction results using the user editing status confirmation program 135, thereby determining which machine learning model should be applied when the same type of image is input, thereby improving the convenience of image inspection for the user.

In the display area 40, the results of area detection using multiple machine learning models (area detection) DB109 are displayed in the area detection result area 137, divided into four sections. To make it easier to understand the operations performed on the input device (finger touch input, mouse input, input pad input, etc.), these are illustrated as input device operations 140.

The model selection program 133 displays the detection results of candidate machine learning models (area detection) in the display area 40. The display area 40 is divided into multiple areas, and the detection results are displayed in each divided area.

By operating the input device 140, the user can switch to the captured image display 138b by pressing the left corner of the area detection result area 137b or swiping rightward (in the direction of the arrow) on the area detection result area 137b. The animation state during switching is shown as switching animation 139.

Figure 6 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 6, while the display is transitioning from the area detection result area 137b to the captured image display 138b, the difference between the two is emphasized by changing the display using switching animation 139, such as front/back display or page turning display.

Figure 7 is an explanatory diagram of the operation screen of the model selection program 133. To switch from the captured image display 138b to the area detection result area 137b, you can press the left corner of the captured image display 138b or swipe left on the captured image display 138b to switch to the captured image display 138b.

Figure 8 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 8, it is also possible to add a switching animation effect, as in Figure 6. As shown in Figure 8, while the display transitions from the captured image display 138b to the area detection result area 137b, the difference between the two can be highlighted by changing the display using switching animation 139, such as front/back display or page turning display.

Figure 9 is an explanatory diagram of the operation screen of the model selection program 133. To display the selected region detection result area 137b across the entire display area 40, press near the center of the region detection result area 137b.

For example, press and hold near the center of the region detection result area 137b for one second or longer. By pressing near the center of the region detection result area 137b, the region detection result area 137b will be displayed across the entire display area 40.

Figure 10 is an explanatory diagram of the operation screen of the model selection program 133. In the embodiment of Figure 9, it is also effective to add a switching animation effect, as in Figure 6. By pressing near the center of the region detection result area 137b, the region detection result area 137b expands as shown by the arrow. The region detection result area 137b expands to fill the entire display area 40, and the expansion of the region detection result area 137b is visually recognized.

Figure 11 is an explanatory diagram of the operation screen of the model selection program 133. The size of each area in the divided area detection result area 137 (area detection result area 137a, area detection result area 137b, area detection result area 137c, area detection result area 137b) can also be changed by dragging the dividing line.

As shown in FIG. 12, press the intersection of region detection result area 137a, region detection result area 137b, region detection result area 137c, and region detection result area 137b, and move the operation 140 on the input device as shown in FIG. 12.

Figure 12 is an explanatory diagram of the operation screen of the model selection program 133. As shown in Figure 12, the display area of the area detection result area 137a, area detection result area 137c, etc. can be changed by moving the position of the operation 140 on the input device.

Figure 13 is an explanatory diagram of the operation screen of the model selection program 133. Similar to the operation in Figure 9, the area detection result area 137, which is now displayed as a single image, can be switched to a captured image display 138 by pressing the left corner of the area detection result area 137 or swiping right on the area detection result area 137. Selection of the machine learning model (area detection) can be completed by pressing the adopt button (OK) 141.

In addition, the adoption button may be configured to display multiple selection buttons such as "Back" and "Update" instead of just "OK," allowing any one or more buttons to be clicked or selected.

Figure 14 is an explanatory diagram of the operation screen of the model selection program 133. As with Figure 6, it is also possible to add a switching animation effect. As shown in Figure 14, while the display is transitioning from the area detection result area 137 to the captured image display 138, the difference between the two can be highlighted by changing the display using switching animation 139, such as front/back display or page turning display.

15 is an explanatory diagram of the operation screen of the model selection program 133. It is possible to switch to the photographed image display 138 by pressing the left corner of the photographed image display 138 or swiping left on the photographed image display 138. Also, as in FIG. 13 , it is possible to complete the selection of the machine learning model (area detection) by pressing the adopt button 141.
16 is an explanatory diagram of the operation screen of the model selection program 133. As in FIG. 6, it is also possible to add a switching animation effect.

The above examples of the present invention have described the detection, visualization device, and visualization method of voids 36 inside a material. Patterns of cracks 39 can also be acquired using an X-ray CT scanner 3 or similar. Although images of voids 36 and cracks 39 are different, by training them as cracks 39, it becomes possible to detect cracks 39. Therefore, it goes without saying that matters described as voids 36 in the examples of this specification can be replaced with cracks 39.

Furthermore, the pattern can be obtained not only by a transmission X-ray image using an X-ray CT scanner 3 or the like, but also by using, for example, an ultrasonic microscope 2. A gamma ray device can also be used. Therefore, it goes without saying that in the device of the present invention, an X-ray device such as an X-ray CT scanner can be replaced with an ultrasonic device 2 such as an ultrasonic microscope.
Furthermore, by combining an X-ray device such as the X-ray CT device 3 with the ultrasonic microscope 2 to obtain a pattern, the accuracy of detecting voids 36, cracks 39, etc. can be improved.

The above embodiments are directed to a method and apparatus for detecting voids 36, cracks 39, etc. in a transmitted X-ray image, but the present invention is not limited to this. It goes without saying that the present invention can be used to non-destructively analyze a wide variety of objects, such as air bubbles in adhesive resin, spaces in concrete blocks, other metals mixed in iron parts, and fat particles in the internal organs of living organisms.
18, 19, 20 and 21 are explanatory diagrams of business models that use the automatic detection method and computer program of the present invention.

Figure 18 is an explanatory diagram of the business model in the first embodiment. Company A provides or sells to Customer A, who owns images captured with an ultrasonic microscope 2, X-ray CT scanner 3, or the like, a program (hereinafter referred to as the "automatic void/crack detection program") that includes the user authentication program 101, case management program 102, captured image upload program 103, machine learning program 110, automatic detection program 111, analysis result editing/viewing program 112, automatic analysis report creation program 113, analysis report notification program 114, and analysis report editing/viewing program 115 shown in Figure 1.

Customer A will be able to install and use the Boyd Crack Automatic Detection Program on desktop computers, laptops, tablets, and smartphones. Company A will receive license fees for the Boyd Crack Automatic Detection Program from Customer A.

Figure 19 is an explanatory diagram of a business model in the second embodiment. Company A provides or sells a group of images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc. owned by Company A, and an automatic void/crack detection program to Customer A, who does not own images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc.

Customer A will be able to install and use the void/crack automatic detection program on desktop computers, laptops, tablets, and smartphones. Company A will receive image capture fees and license fees for the void/crack automatic detection program from Customer A.

Figure 20 is an explanatory diagram of a business model in the third embodiment. Company A runs an automatic void/crack detection program on a cloud server for Customer A, who owns images taken with an ultrasonic microscope 2, an X-ray CT scanner 3, etc.

Customer A can use the void/crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives a usage fee for the void/crack automatic detection program from Customer A.

Figure 21 is an explanatory diagram of a business model in the fourth embodiment. For Customer A, who does not own images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc., Company A provides a group of images taken with an ultrasonic microscope 2, X-ray CT scanner 3, etc. owned by Company A, and runs an automatic void/crack detection program on a cloud server.

Customer A can use the void/crack automatic detection program using a web browser on a desktop computer, laptop computer, tablet, or smartphone. Company A receives image capture fees and usage fees for the void/crack automatic detection program from Customer A.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims. It goes without saying that the matters or contents described in this specification and drawings can be mutually combined.

1. Void and crack automatic detection device 2. Acoustic microscope 3. X-ray CT device 4. Electronic components (measurement sample)
5 Solder 11 Control unit 12 Main memory unit 13 Communication unit 14 Operation unit 15 Display panel 16 Auxiliary memory unit 17 Network 18 Recording medium 21 Photographed image group 22 Analysis result (mask image) group 31 Pattern classification training program 32 Pattern classification prediction program 36 Void 37 Classification (processing unit)
38 Solder part 39 Crack 40 Display area 41 Area detection training program 42 Area detection prediction program 101 User authentication program 102 Case management program 103 Photographed image upload program 104 Photographed image (1) DB
105 Analysis results (1) DB
106 Analysis Report (1) DB
107 Machine Learning Data (1) DB
108 Machine learning model (pattern classification) DB
109 Machine learning model (area detection) DB
110 Machine learning program 111 Automatic detection program 112 Analysis result editing and viewing program 113 Analysis report automatic creation program 114 Analysis report notification program 115 Analysis report editing and viewing program 133 Model fitting program 134 Image scale adjustment program 135 User editing status confirmation program 137 Area detection result area by candidate model 138 Display of captured image 139 Front/back display switching animation 140 Operation with input device 141 Accept button 160 Distance from X-ray focus to sample 161 Distance from X-ray focus to X-ray detector 162 X-ray focus 163 X-ray detector 164 Enlarged projection image 201 Pattern classification prediction program 202 Area detection prediction program for pattern A 203 Area detection prediction program for pattern B 204 Area detection prediction program for pattern C 301 Captured image (A-B) DB
302 Pattern (AB) DB
303 Data augmentation preprocessing program 304 Machine learning data (AB) DB
305 Machine learning (training) program 306 Photographed image group 307 Preprocessing program 308 Machine learning (prediction) program 309 Pattern classification result group 401 Photographed image (A) DB
402 Mask Image (A) DB
403 Data augmentation preprocessing program 404 Machine learning data (A) DB
405 Machine learning (training) program 406 Photographed image group 407 Preprocessing program 408 Machine learning (prediction) program 409 Analysis result (mask image) group 501 Scale value written in image 502 BGA ball diameter 503 Component length 504 Model selection program 505 Area detection prediction program 506 Machine learning model copy program

Claims (10)

an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint;
a pattern classification program that classifies the acquired image into patterns using a machine learning model;
an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image;
a detection program for detecting a visualized object from the image;
a display device that displays a plurality of images of the visualization object ;
A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object . an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint;
a pattern classification program that classifies the acquired image into patterns using a machine learning model;
an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image;
a detection program for detecting a visualized object from the image;
a display device that displays a plurality of images of the visualization object;
a selection program for selecting an image of a predetermined visualization object from the displayed images of the plurality of visualization objects;
A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object . an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint;
a pattern classification program that classifies the acquired image into patterns using a machine learning model;
an image adjustment program for adjusting at least one of the position, rotation, and scaling of a visualized object of the image;
a detection program for detecting a visualized object from the image;
a selection program that allows a user to view images of the visualization object and select an image of the visualization object ;
A visualization device for voids and cracks in a joint, comprising a confirmation program for confirming the image viewing time of the visualization object . an image acquisition device that acquires at least one of a transmission X-ray image, a cross-sectional image, and an ultrasonic image of the joint;
a pattern classification program that classifies the acquired image into patterns using a machine learning model;
a detection program that detects a first visualization object from the visualization objects in the image;
a display device that displays an image of the first visualization object;
a selection program that selects a predetermined second visualized object image from the image of the first visualized object displayed on the display device;
A visualization device for voids and cracks in joints, comprising a confirmation program for confirming the image viewing time of the first visualization object and the image viewing time of the second visualization object. the joint is a solder joint, and the visualization object is at least one of a void and a crack,
5. The apparatus for visualizing voids and cracks in a joint according to claim 1, 2, 3 or 4, wherein the pattern is based on the shape, form or structure of the joint. a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model;
a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image;
a fourth operation of selecting a predetermined machine learning model from the plurality of machine learning models based on an image of the visualization object selected by a user ;
A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object . a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model;
a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image;
a third operation of displaying images of the visualization objects on a display device and allowing a user to select an image of a predetermined visualization object from the images of the visualization objects;
a fourth operation of selecting a predetermined machine learning model from the plurality of machine learning models based on an image of the visualization object selected by a user ;
A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object . a first operation of classifying at least one of a transmission X-ray image, a cross-sectional image, and an ultrasound image of the joint into a pattern using a machine learning model;
a second operation of adjusting at least one of a position, a rotation, and a scale of a visualization object of the image;
a third operation of displaying images of the visualization objects on a display device and allowing a user to select an image of a predetermined visualization object from the images of the visualization objects;
a fifth operation of determining whether the machine learning model is appropriate based on the time the user viewed the image of the visualization object, and changing or maintaining the machine learning model ;
A method for visualizing voids and cracks in a joint, comprising a sixth operation of confirming the image viewing time of the visualization object . 9. The method for visualizing voids and cracks in a joint according to claim 6, 7 or 8, wherein the pattern is based on the shape, form or structure of the joint . 9. The method for visualizing voids and cracks in a joint according to claim 6, claim 7 or claim 8, wherein the joint is a solder joint, and the visualization object is at least one of a void and a crack.