JP7520802B2 - Radiation image processing device, radiation image processing method, image processing device, learning device, learning data generation method, and program - Google Patents

Description

The present invention relates to a radiological image processing device, a radiological image processing method, an image processing device, a learning device, a method for generating learning data, and a program.

In recent years, radiation detectors equipped with a detection unit for detecting radiation such as X-rays have been widely used in fields such as industry and medicine. In particular, digital radiography (DR) devices that use semiconductor sensors to obtain radiation images have become widespread.

In such digital radiography devices, it is common to perform various types of image processing to improve the image quality of the captured image, one of which is noise reduction processing, which improves the graininess of the captured image and improves the visibility of the diagnostic area. In particular, noise reduction processing that applies machine learning techniques such as deep learning has the potential to significantly improve graininess. For example, Non-Patent Document 1 proposes a technology that achieves good noise reduction effects by performing supervised learning using a set of images to which noise estimated from the noise characteristics of the imaging device has been added.

“Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections”, Xiao-Jiao Mao, Chunhua Shen, and Yu- Bin Yang, ArXiv: 1606.08921, 2016

However, the above-mentioned conventional techniques can have the following problems. In industrial products that are mass-produced, manufacturing variations occur in components such as image sensors, lenses, and phosphors, as well as in the assembly precision of parts, and individual differences in resolution, sensitivity, noise characteristics, etc. are unavoidable.

Therefore, when this type of machine learning technology is applied to industrial products intended for mass production, the characteristics of the product that generated the data used during learning may not match the characteristics of the product that generated the data that is the subject of image processing during inference. In other words, the characteristics of the learning system used for learning may not match the characteristics of the inference system used after shipping as an industrial product, making it impossible to perform suitable image processing. For example, in noise reduction processing, problems such as a reduction in the effectiveness of noise reduction or the occurrence of artifacts in the inference results may occur.

The present invention has been made in consideration of the above problems, and one of its objectives is to provide a radiological image processing device, a radiological image processing method, an image processing device, a learning device, a method for generating learning data, and a program that can apply suitable image processing to images.

In addition to the above object, the present invention also aims to achieve effects that cannot be obtained by conventional techniques, which are derived from the configurations shown in the detailed description of the invention described below.

In order to achieve the object of the present invention, the radiographic image processing device of the present invention is characterized by including: an acquisition means for acquiring a first radiographic image of an object to be inspected captured by a radiographic imaging device; and a generation means for generating a second radiographic image having reduced noise compared to the first radiographic image by inputting the first radiographic image acquired by the acquisition means into a trained model obtained by training using training data including a radiographic image obtained by adding noise simulating system noise of the radiographic imaging device obtained by using a distribution based on manufacturing variability of the radiographic imaging device and noise simulating quantum noise contained in radiation.

According to one embodiment of the present disclosure, it is possible to provide a radiation image processing device, a radiation image processing method, an image processing device, a learning device, a method for generating learning data, and a program that can apply suitable image processing to an image.

1 is a diagram showing an example of a schematic configuration of a radiation imaging system according to a first embodiment; FIG. 1 is a diagram showing an example of a schematic configuration of a radiation detector according to a first embodiment. FIG. 2 is a diagram illustrating an example of a schematic configuration of a control unit according to the first embodiment. FIG. 2 is a diagram showing an example of a schematic configuration of a noise reduction processing unit according to the first embodiment. FIG. 1 is a diagram showing an example of a schematic configuration of a trained model according to a first embodiment. FIG. 1 is a diagram showing an example of a schematic configuration of a trained model according to a first embodiment. 5A to 5C are diagrams for explaining an operation example of a learning process according to the first embodiment. 10 is a flowchart illustrating an example of a learning process according to the first embodiment. 5A and 5B are schematic diagrams for explaining manufacturing variations in system noise of the radiation detector according to the first embodiment. 5A and 5B are schematic diagrams for explaining manufacturing variations in MTF of the radiation detector according to the first embodiment. 11 is a flowchart illustrating an example of an inference process according to the first embodiment. 5A to 5C are diagrams showing examples of images before and after image processing according to the first embodiment. FIG. 11 is a diagram illustrating an example of a schematic configuration of a control unit according to a second embodiment. FIG. 13 is a diagram showing an example of a schematic configuration of a trained model according to a second embodiment.

Below, exemplary embodiments for carrying out the present invention will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used in the drawings to indicate elements that are identical or functionally similar.

In the embodiments of the present disclosure described below, radiation includes alpha rays, beta rays, and gamma rays, which are beams created by particles (including photons) emitted by radioactive decay, as well as beams with the same or greater energy, such as X-rays, particle beams, and cosmic rays. In the following, dark current and electrical noise that are not dependent on the magnitude of the signal in the radiation image are referred to as system noise.

In the following, a machine learning model refers to a learning model based on a machine learning algorithm. Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. Neural networks and deep learning may also be used. Any of the above algorithms that can be used can be used as appropriate and applied to the following embodiments and modifications. Furthermore, learning data refers to a data set used to train a machine learning model, and is composed of a pair of input data input to the machine learning model and correct answer data that is the correct answer for the output result of the machine learning model.

Note that a trained model refers to a machine learning model that follows any machine learning algorithm, such as deep learning, and that has been trained in advance using appropriate training data. However, although a trained model is obtained by training in advance using appropriate training data, this does not mean that no further training is performed, and additional training can also be performed. Additional training can also be performed after the device is installed at the site of use.

Example 1
(Configuration of Radiography System)
Hereinafter, a radiography system according to a first embodiment of the present disclosure will be described with reference to FIGS. 1A and 1B. FIG. 1A shows a schematic configuration of a radiography system 1 according to the present embodiment. In the following description, the object O to be inspected is a human body, but the object O to be inspected by the radiography system according to the present disclosure is not limited to a human body, and may be other animals, plants, and objects to be subjected to non-destructive inspection. In addition, the following description will be given using the radiography system shown in FIG. 1A as an example, but the imaging device that images the object O to be inspected may be an MRI device, an X-ray CT device, a three-dimensional ultrasound imaging device, a photoacoustic tomography device, a PET/SPECT, an OCT device, an optical camera, or the like. That is, the series of processes according to the present embodiment may be applied not only to radiographic images, but also to other medical images and optical images.

The radiation imaging system 1 according to this embodiment includes a radiation detector 10, a control unit 20, a radiation generator 30, an input unit 40, and a display unit 50. The radiation imaging system 1 may also include an external storage device 70, such as a server, that is connected to the control unit 20 via a network 60, such as the Internet or an intranet.

The radiation generator 30 includes a radiation generating source such as an X-ray tube and can emit radiation. The radiation detector 10 can detect the radiation emitted by the radiation generator 30 and generate a radiation image corresponding to the detected radiation. Therefore, the radiation detector 10 can generate a radiation image of the object O to be inspected by detecting the radiation emitted by the radiation generator 30 and passing through the object O to be inspected. Note that the image in this disclosure includes not only the state in which it is displayed on the display unit, but also the state in which it is stored as image data in a database or storage unit.

Here, FIG. 1B shows a schematic configuration of the radiation detector 10 according to this embodiment. The radiation detector 10 is provided with a phosphor 11 and an imaging sensor 12. The phosphor 11 converts radiation incident on the radiation detector 10 into light of a wavelength detectable by the imaging sensor 12. The phosphor 11 may include, for example, CsI or GOS (Gd ₂ O ₂ S). The imaging sensor 12 includes a photoelectric conversion element made of, for example, a-Si or crystalline Si, and can detect light corresponding to the radiation converted by the phosphor 11 and output a signal corresponding to the detected light. The radiation detector 10 can generate a radiation image by performing A/D conversion or the like on the signal output by the imaging sensor 12. Although omitted in FIG. 1B, the radiation detector 10 may include a calculation unit, an A/D conversion unit, and the like.

The control unit 20 is connected to the radiation detector 10, the radiation generator 30, the input unit 40, and the display unit 50. The control unit 20 can acquire a radiation image output from the radiation detector 10 and perform image processing on the radiation image, and control the driving of the radiation detector 10 and the radiation generator 30. Therefore, the control unit 20 can function as an example of an image processing device. The control unit 20 may be connected to an external storage device 70 via an arbitrary network 60 such as the Internet or an intranet, and may acquire a radiation image or the like from the external storage device 70. Furthermore, the control unit 20 may be connected to other radiation detectors, radiation generators, etc. via the network 60. The control unit 20 may be connected to the external storage device 70, etc., by wire or wirelessly.

The input unit 40 includes input devices such as a mouse, keyboard, trackball, and touch panel, and can be operated by an operator to input instructions to the control unit 20. The display unit 50 includes, for example, any monitor, and can display information and images output from the control unit 20, and information input by the input unit 40, etc.

In this embodiment, the control unit 20, input unit 40, display unit 50, etc. are configured as separate devices, but they may be configured as an integrated unit. For example, the input unit 40 and display unit 50 may be configured as a touch panel display. In this embodiment, the control unit 20 configures an image processing device, but the image processing device only needs to be able to acquire a radiation image and perform image processing on the radiation image, and does not need to control the operation of the radiation detector 10 or radiation generator 30.

The control unit 20 may be connected to the radiation detector 10, radiation generator 30, etc. by wire or wirelessly. Furthermore, the external storage device 70 may constitute an image system such as a PACS in a hospital, or may be a server outside the hospital.

(Configuration of the control unit)
Next, a more specific configuration of the control unit 20 (radiation image processing device) will be described with reference to Figures 2A and 2B. Figures 2A and 2B show a schematic configuration of the control unit 20 according to this embodiment. The control unit 20 includes an acquisition unit 21, an image processing unit 22, a display control unit 23, a drive control unit 24, and a storage unit 25.

The acquisition unit 21 can acquire radiological images output by the radiation detector 10 and various information input by the input unit 40. The acquisition unit 21 can also acquire radiological images and patient information from an external storage device 70, etc. That is, the acquisition unit 21 can acquire radiological images of the subject captured by the radiographic device.

The image processing unit 22 is composed of a noise reduction processing unit 26 and a diagnostic image processing unit 27, and can perform image processing according to the present disclosure on the radiographic image acquired by the acquisition unit 21. The noise reduction processing unit 26 is composed of a learning processing unit 261 and an inference processing unit 262. In addition to the configuration of the inference processing unit 262, the learning processing unit 261 is further provided with a manufacturing variation analysis unit 263, a learning data generation unit 264, and a parameter update unit 265. With this configuration, the noise reduction processing unit 26 can learn a machine learning model for performing noise reduction processing, and can apply noise reduction processing suitable for the radiographic image using the machine learning model. In addition, the diagnostic image processing unit 27 can perform diagnostic image processing for converting the image on which noise reduction has been performed by the noise reduction processing unit 26 into an image suitable for diagnosis. Diagnostic image processing includes, for example, gradation processing, emphasis processing, grid stripe reduction processing, and the like.

Next, we will explain the configuration of the learning processing unit 261.

The learning processing unit 261 performs the learning process applied when learning a machine learning model, and in addition to the configuration of the inference processing unit 262, it also has a learning data generation unit 264 and a parameter update unit 265.

When performing the learning process, an image is input to the learning processing unit 261, and learning data is created by the learning data generation unit 264.

Here, we present an example of a set of training data for learning noise reduction processing, in which an image with artificial noise added is used as input data, and an image without artificial noise added is used as correct answer data.

A process is performed to create a set of learning data by adding artificial noise created by simulating the characteristics of a radiological image to the input image. Here, the noise added by the learning data generation unit 264 reflects the amount of noise that may vary due to manufacturing variability, calculated by the manufacturing variability analysis unit 263. Details of the artificial noise to be added will be described later.

The parameter update unit 265 performs processing to update the parameters of the machine learning model held by the inference processing unit 262 based on the calculation results of the inference processing unit 262 and the correct answer data.

Here, the learning processing unit 261 does not necessarily need to be included in the radiography system, and may be configured, for example, on hardware separate from the radiography system. Then, a trained model may be created by performing training in advance using appropriate training data, and only processing by the inference processing unit 262 may be performed in the radiography system. Alternatively, the learning processing unit 261 may be included in the radiography system, allowing additional training to be performed using training data acquired after installation.

The display control unit 23 can control the display of the display unit 50, and can display radiation images before and after image processing by the image processing unit 22, patient information, etc. on the display unit 50. The drive control unit 24 can control the drive of the radiation detector 10 and the radiation generator 30, etc. Therefore, the control unit 20 can control the capture of radiation images by controlling the drive of the radiation detector 10 and the radiation generator 30 using the drive control unit 24.

The storage unit 25 can store programs for implementing various application software including an operating system (OS), device drivers for peripheral devices, and programs for performing the processes described below. The storage unit 25 can also store information acquired by the acquisition unit 21, radiological images processed by the image processing unit 22, and the like. For example, the storage unit 25 can store radiological images acquired by the acquisition unit 21, and radiological images that have been subjected to noise reduction processing described below.

The control unit 20 can be configured using a general computer including a processor, memory, etc., but may be configured as a dedicated computer for the radiation imaging system 1. Here, the control unit 20 functions as an example of an image processing device according to this embodiment, but the image processing device according to this embodiment may be a separate (external) computer communicatively connected to the control unit 20. The control unit 20 and the image processing device may be, for example, a personal computer, and a desktop PC, a notebook PC, or a tablet PC (portable information terminal) may be used. The processor may be a CPU (Central Processing Unit). The processor may be, for example, an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or an FPGA (Field-Programmable Gate Array).

Each function of the control unit 20 may be realized by a processor such as a CPU or MPU executing a software module stored in the memory unit 25. The processor may be, for example, a GPU or FPGA. Each function may be configured by a circuit that performs a specific function, such as an ASIC. For example, the image processing unit 22 may be realized by dedicated hardware such as an ASIC, and the display control unit 23 may be realized by a dedicated processor such as a GPU that is different from the CPU. The memory unit 25 may be configured by any storage medium, such as an optical disk such as a hard disk, or a memory.

(Machine learning model configuration)
Next, an example of a machine learning model constituting the trained model according to the present embodiment will be described with reference to Fig. 3A, Fig. 3B, and Fig. 3C. An example of the machine learning model used by the inference processing unit 262 according to the present embodiment is a multi-layer neural network.

Figure 3A shows a schematic configuration example of a neural network model according to this embodiment. The neural network model 33 shown in Figure 3A is designed to output inference data 32 in which noise has been reduced in response to input data 31 according to previously learned trends. The output radiographic image in which noise has been reduced is based on the learning content in the machine learning process, and the neural network according to this embodiment has learned feature quantities for discriminating between signals and noise contained in the radiographic image.

For example, a convolutional neural network (hereinafter referred to as CNN) may be used for at least a part of the multi-layered neural network. Furthermore, technology related to an autoencoder may be used for at least a part of the multi-layered neural network.

Here, we will explain the case where CNN is used as a machine learning model for noise reduction processing of radiological images. FIG. 3B shows an example of a schematic configuration 33 of a CNN constituting a neural network model according to this embodiment. In the example of the trained model according to this embodiment, when a radiological image 31 is input, an inference image 32 with reduced noise can be output.

The CNN shown in FIG. 3B is composed of multiple layers that process and output a group of input values. The types of layers included in the CNN configuration 33 include a convolution layer, a downsampling layer, an upsampling layer, and a merge layer. Here, 34 is an addition layer, and it is preferable to configure a shortcut that adds input data before output. This allows the CNN to be configured to learn the difference between input data and output data, making it possible to handle systems that target noise.

The convolution layer is a layer that performs convolution processing on a group of input values according to parameters such as the set filter kernel size, number of filters, stride value, dilation value, etc. Note that the number of dimensions of the filter kernel size may also be changed depending on the number of dimensions of the input image.

The downsampling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or combining input value groups. Specifically, an example of such processing is Max Pooling processing.

The upsampling layer is a layer that performs processing to make the number of output values greater than the number of input values by duplicating the input values or adding values interpolated from the input values. Specifically, one example of such processing is upsampling processing by deconvolution.

A synthesis layer is a layer that inputs a group of values, such as the output values of a layer or the pixel values that make up an image, from multiple sources and performs processing to combine them by concatenating or adding them.

Note that different parameter settings for the layers and nodes that make up a neural network may affect the degree to which trends trained from learning data can be reproduced during inference. In other words, in many cases, appropriate parameters differ depending on the implementation form, and can be changed to preferred values as necessary.

In addition to changing the parameters as described above, it is possible to change the configuration of the CNN to obtain better characteristics. Examples of better characteristics include outputting radiological images with reduced noise with higher accuracy, shorter processing times, and shorter training times for machine learning models.

The CNN configuration 34 used in this modified example is a U-net type machine learning model that has an encoder function consisting of multiple layers including multiple downsampling layers, and a decoder function consisting of multiple layers including multiple upsampling layers. In the U-net type machine learning model, the position information (spatial information) that is obscured in multiple layers configured as an encoder is configured so that it can be used in layers of the same dimension (layers that correspond to each other) in multiple layers configured as a decoder (for example, by using skip connections).

Although not shown, as an example of a modified CNN configuration, for example, a layer of an activation function (e.g., ReLu: Rectifier Linear Unit) may be incorporated before or after the convolutional layer, and layers that perform various normalization processes such as batch normalization may be incorporated before or after the above.

Through these steps of the CNN, noise features can be extracted from the input radiological image.

Here, the learning processing unit 261 includes a parameter update unit 265. As shown in FIG. 3C, the parameter update unit 265 calculates a loss function from inference data 32 obtained by applying the neural network model 33 of the inference processing unit 262 to input data 31 in the learning data, and from supervised answer data 35 in the learning data, and performs processing to update the parameters of the neural network model 33 based on the loss function. Here, the loss function indicates the error between the inference data 32 and the supervised answer data 35.

The parameter update unit 265 also updates the filter coefficients of the convolution layer, for example, using the backpropagation method, so as to reduce the error between the inference data 32 represented by the loss function and the ground truth data 35. The backpropagation method is a technique for adjusting the parameters between the nodes of the neural network so as to reduce the above-mentioned error.

For learning, a method of randomly deactivating units (each neuron or each node) that make up the CNN (dropout) may be used.

Furthermore, the trained model used by the inference processing unit 262 may be one generated using transfer learning. In this case, for example, transfer learning may be performed on a machine learning model trained on a radiological image of an object O to be inspected that is of a different type, etc., to generate a trained model to be used in noise reduction processing. By performing such transfer learning, trained models can be efficiently generated even for objects O to be inspected that are difficult to obtain a large amount of training data. The objects O to be inspected that are of a different type, etc., referred to here may be, for example, animals, plants, objects to be subjected to non-destructive inspection, etc.

Here, the GPU can perform efficient calculations by processing more data in parallel. For this reason, when performing learning multiple times using a learning model that uses a CNN as described above, it is effective to perform the processing using a GPU. Therefore, the learning processing unit 261 in this embodiment uses a GPU in addition to a CPU. Specifically, when executing a learning program that includes a learning model, the CPU and GPU work together to perform calculations to perform learning. Note that in the learning process, calculations may be performed only by the CPU or the GPU. In addition, each process of the inference processing unit 262 may also be realized using a GPU, as in the learning processing unit 261.

The configuration of the machine learning model has been explained above, but it is not limited to models using CNN as shown above, and any model similar to machine learning that uses a model that can extract (express) the features of training data such as images by itself through learning can be used.

(Learning process)
Next, the flow of processing by the learning processing unit 261 according to this embodiment will be described.

Here, as a set of learning data used by the learning processing unit 261, an image with artificial noise added is used as input data to learn noise reduction processing, and an image without artificial noise added is used as correct answer data, and a radiographic image in the medical field is shown as an example of the target.

As training data, it is desirable to use a set of noisy radiographic images as input data and radiographic images that are almost noise-free as ground truth data. However, in view of the invasiveness of radiography, radiography must be performed with as low a dose of radiation as possible, and in reality, it is difficult to prepare such training data.

In this embodiment, radiographic images of the human body captured for medical purposes are used as the correct answer data for the training data, and images in which artificial noise equivalent to the noise generated in radiographic images has been added to the radiographic images are used as input data for the training data. By performing training using such training data, it is possible to learn the image features of the radiographic images and the features of the artificial noise.

Based on the above, the processing flow of the learning processing unit 261 in this embodiment will be explained with reference to Figures 4 to 7.

Figure 4 is a flowchart showing the processing flow of the learning processing unit 261 in this embodiment.

In step S401, learning data is input to the learning processing unit 261. The learning data to be input is acquired by the acquisition unit 21, and in this embodiment, an appropriate radiographic image is acquired as the learning data. The acquisition unit 21 may acquire a radiographic image stored in the storage unit 25, or may acquire a radiographic image from the external storage device 70 or the like. The acquisition unit 21 may also acquire a radiographic image output by the radiation detector 10. The learning data in this embodiment is the above-mentioned radiographic image as input data and correct answer data. After being input as the same data to the learning processing unit 261, appropriate artificial noise is added to the input data or the input data and correct answer data inside the learning processing unit, resulting in a state in which a difference occurs between the input data and the correct answer data.

In step S402, the manufacturing variation analysis unit 263 analyzes the manufacturing variation of the radiation detector 10. Here, the manufacturing variation that affects the artificial noise added in step S403 is analyzed, and the influence of the manufacturing variation is calculated. That is, the manufacturing variation analysis unit 263 analyzes the manufacturing variation of the noise-related characteristics of the radiation imaging device. The noise-related characteristics of the radiation imaging device include, for example, the system noise characteristics of the radiation imaging device, the sensitivity of the radiation imaging device, and the MTF characteristics of the phosphor contained in the radiation imaging device. The manufacturing variation analysis unit 263 can analyze the manufacturing variation for at least one of them. Note that the noise-related characteristics described above are merely examples, and are not limited to these as long as they are variables that affect the artificial noise.

In a radiological image, noise contained in the radiological image is broadly divided into quantum noise generated mainly by fluctuations of radiation quanta and system noise generated from the detector, circuitry, etc. Since the radiation detector 10 is configured to convert radiation into visible light by the phosphor 11, it is known that high-frequency attenuation occurs in the radiation signal according to the modulation transfer function (MTF) defined by the resolution of the phosphor 11. Quantum noise generated by fluctuations of radiation quanta also generates high-frequency attenuation due to a similar modulation transfer function. That is, quantum noise corresponds to an example of noise in which high-frequency components are attenuated. On the other hand, system noise generated from the detector, circuitry, etc. is not affected by the phosphor. The noise contained in the radiological image is the sum of system noise that is not dependent on the dose at the time of imaging and quantum noise that changes depending on the dose at the time of imaging. That is, in this embodiment, noise that imitates system noise and noise in which high-frequency components are attenuated are generated according to a distribution based on the manufacturing variation of the radiological imaging device, and are added to the radiological image.

In light of the above, in this embodiment, the noise that is a combination of white noise corresponding to the system noise of the radiation detector 10 and quantum noise affected by the MTF of the phosphor 11 is treated as artificial noise.

Here, regarding the combined ratio of quantum noise and system noise, it is preferable to simulate the characteristics of the radiation detector 10 using, for example, the following relational expressions (Equation 1) to (Equation 4).

Now, assuming that the standard deviation of the quantum noise is σ _q and the standard deviation of the system noise due to the radiation detector 10 is σ _s , the variance σ _all ² of the noise in the radiation image is given by

In addition, if the input image signal is I _sig , the standard deviation σ _q of the quantum noise is proportional to the signal I _sig , and therefore, using the quantum noise coefficient q,

In addition, since the standard deviation σ _s of the system noise is a constant value that is not proportional to the signal I _sig , it can be expressed as follows using the coefficient s of the system noise:

It can be written as:

Based on the above, the artificial noise (addNoise) can be obtained as follows. First, the artificial system noise sNoise simulating the system noise is assumed to be additive white Gaussian noise (AWGN) with a standard deviation of σ _s . Next, the artificial quantum noise qNoise simulating the quantum noise is assumed to be noise that follows a Poisson distribution with a variance of σ _q ² and has an NPS that follows the MTF of the phosphor 11. For example, when the MTF of the phosphor 11 is approximated by a two-dimensional filter as fMTF, it is preferable to create the noise image that follows the Poisson distribution by convolving the fMTF. Note that the Poisson distribution can be approximated to a normal distribution when the variance is sufficiently large, so qNoise may be treated as noise that follows a normal distribution. In this case, the artificial noise (addNoise) is expressed as follows:

It can be expressed as.

It is experimentally known that the distribution of noise occurring in radiological images approximates a normal distribution with a mean and median of approximately 0, and the artificial noise obtained as described above can also be made to have a distribution with a mean and median of approximately 0.

When considering the above system, individual differences occur in the radiation detector 10 due to manufacturing variations, mainly in system noise, sensitivity, and phosphor MTF characteristics.

Figure 5 shows an example of manufacturing variation in the system noise characteristics of the radiation detector 10.

It is generally known that various characteristics are distributed according to a normal distribution due to manufacturing variations, and it is understood that the standard deviation σ _s of the system noise in radiation detectors also has individual differences that approximately follow a normal distribution.

Now, when the system noise characteristic variation follows a normal distribution with a mean μs and a variance _σs ² _M , the manufacturing variation analysis unit 263 is configured to vary the variance of the system noise with a frequency following the normal distribution with a mean μs and a variance _σs ² _M for each trial of S402, and calculate sNoise shown in (Equation 4). That is, it is possible to generate and add noise simulating the system noise of the radiation imaging device according to a distribution based on the manufacturing variation of the radiation imaging device. In particular, it is possible to generate and add noise simulating the system noise according to the distribution of the variation of the system noise characteristic caused by the manufacturing variation of the radiation imaging device. In addition, it is possible to add noise simulating the system noise with a frequency following a distribution calculated from the mean and variance of the variation of the system noise characteristic. It is preferable to calculate the mean μs and the variance _σs ² _M from the manufacturing record obtained in advance. That is, it is possible to generate noise simulating the system noise according to statistical information on the system noise obtained based on the manufacturing record of the radiation imaging device.

As described above, qNoise shown in (Equation 4) is a characteristic that is determined by the sensitivity of the radiation detector 10, the MTF, and the intensity of the radiation input to the radiation detector 10. As with the system noise, the sensitivity and the MTF of the phosphor also have variations that follow a normal distribution due to manufacturing variations, so a characteristic value is randomly selected from the normal distribution for each trial in S402 and used as a parameter for creating artificial noise. That is, it is possible to generate noise in which high-frequency components are attenuated according to the distribution of the characteristics of the MTF of the phosphor contained in the radiation imaging device, which is generated due to manufacturing variations of the radiation imaging device. In addition, it is possible to add noise in which high-frequency components are attenuated with a frequency that follows a distribution calculated from the average and variance of the variations in the characteristics of the MTF of the phosphor. Furthermore, it is possible to add noise in which high-frequency components are attenuated according to the characteristics of the MTF that are randomly selected from a range determined by the distribution. It is preferable to determine a certain range for the range of manufacturing variations, such as by specifying maximum and minimum values based on the general six sigma, or by selecting a value within the standard value if a product standard value is set for each characteristic.

By using the above configuration, it is possible to create artificial noise that takes into account individual differences due to manufacturing variability, and to learn how to reduce it.

Regarding the individual differences due to the above manufacturing variations, for example, when the models are different but the radiation detectors 10 have substantially the same characteristics, such as a product group that has a substantially identical design except for the area of the imaging surface, a configuration may be adopted that takes into account the manufacturing variations, including the differences in models. That is, noise may be created according to statistical information obtained based on the manufacturing track record of multiple manufactured radiographic devices of the same model. Alternatively, noise may be created according to statistical information obtained based on the manufacturing track record of multiple manufactured radiographic devices of different models.

In addition to the individual differences due to manufacturing variations as described above, a configuration may be adopted that takes into account the in-plane characteristic variations within an individual device, as described below.

The radiation detector 10 has a large imaging surface (e.g., approximately 43 cm x 43 cm). For this reason, it is difficult to manufacture a detector with uniform characteristics within the surface, and even the same detector may have slight differences in characteristics within the surface.

Figure 6 is a diagram showing an example of the in-plane distribution of the MTF characteristics of the radiation detector 10. Due to the relationship with the phosphor formation process, etc., it is difficult to make the film thickness of the phosphor constant, and the MTF may not be uniform in the plane, such as a low MTF in the thick central part (A) of the phosphor and a high MTF at the thin end part (D) of the phosphor. In order to deal with such variations, the maximum and minimum values of the MTF in the phosphor plane are measured, and when creating the artificial noise to be added, a value between the maximum and minimum values is randomly selected for each trial of S402. Note that, if a standard value is set as a product for each characteristic, a value within the standard value may be selected. If there is a similar tendency not only for the MTF shown here but also for the characteristics of the system noise and the sensitivity, a similar configuration can be taken. In other words, it is possible to add noise that imitates the system noise according to the variation in the system noise characteristics randomly selected from within the range determined in the distribution.

In step S403, the learning data generation unit 264 adds artificial noise to the input data of the learning data. Here, the addition is performed based on the artificial noise that takes into account the manufacturing variability calculated by the manufacturing variability analysis unit 263 in step S402.

In step S404, appropriate preprocessing is performed on the set of training data. Note that the preprocessing method is not limited to the above. For example, in noise reduction processing, for example, square root transformation or logarithmic transformation can be performed to make quantum noise that follows a Poisson distribution approximately constant regardless of the strength of the input radiation, and a conversion process that allows additive noise to be handled, or a process that sets the average value to 0 can be used. In addition, appropriate preprocessing can be performed depending on the content of image processing, such as normalizing the data with 0-1, or standardizing it to have an average value of 0 and a standard deviation of 1.

In step S405, the inference processing unit 262 performs inference processing using a machine learning model on the input data of the learning data, and outputs inference data to which noise reduction processing has been applied.

In step S406, the parameter update unit 265 compares the inference data with the correct answer data, and calculates a loss function that quantifies the error between the two. In addition, the parameters of the machine learning model (e.g., the filter coefficients of the convolutional layer in a CNN) are updated based on the loss function. Examples of loss functions include the mean absolute error (L1loss) and the mean squared error (L2loss).

In step S407, the learning is judged to be completed. If it is judged to be completed, the flow is terminated. If not, the flow returns to S401, and the flow from S401 to S406 is repeated using other data. The random numbers used in the manufacturing variation analysis performed in S402 are reset for each loop, so that a machine learning model that appropriately corresponds to the manufacturing variation can be learned. The criterion for judging the completion may be, for example, whether a specific number of loops has been executed. Alternatively, it may be judged whether the loss function is equal to or less than a certain value. Alternatively, it may be judged whether overlearning has occurred. Alternatively, it may be judged whether the performance has reached a sufficient level by evaluating the PSNR (Peak Signal to Noise Ratio) or SSIM (Structural SIMilarity), which are indices indicating the performance of noise reduction in the inference image. In other words, it is possible to use an appropriate judgment criterion that is generally used in machine learning.

As described above, the learning processing unit 261 in this embodiment adds artificial noise that simulates the noise characteristics of an actual radiological image while taking into account manufacturing variability. This makes it possible to generate a machine learning model that can output an image with suitably reduced noise when applying processing using machine learning techniques to products that have manufacturing variability associated with mass production, even when the characteristics of the learning system and the inference system do not match.

(Inference Processing Unit)
Next, the flow of processing by the inference processing unit 262 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing a series of image processing steps by the inference processing unit 262 according to this embodiment.

In the series of image processing according to this embodiment, pre-processing is performed on the radiographic image to be processed, and the image is then input to the trained model. After that, post-processing corresponding to the pre-processing is performed on the output from the trained model, and a noise-reduced radiographic image corresponding to the original radiographic image is generated.

When a series of image processing according to this embodiment is started, in step S701, the acquisition unit 21 acquires a radiological image. The acquisition unit 21 may acquire a radiological image generated by the radiation detector 10, or may acquire a radiological image from the storage unit 25, the external storage device 70, etc.

In step S702, preprocessing is performed. This preprocessing is the same as S404 in the learning process.

In step S703, the inference processing unit 262 performs inference processing using the trained model obtained by the learning processing to generate an image with reduced noise. Note that the trained model may be a machine learning model that has been trained in advance, and there is no need for training to be performed each time a series of image processing is performed. Note that it is possible to adjust the strength of the noise reduction processing by blending the input image and the image with reduced noise obtained by performing the inference processing in a specific ratio.

In addition, in this embodiment, the trained model is configured to be provided in the control unit 20, but the trained model may also be provided in an external storage device 70 connected to the control unit 20.

In step S704, post-processing corresponding to the pre-processing performed in step S702 is performed. The post-processing is the inverse processing of the conversions such as normalization and smoothing performed in the pre-processing. The processing in steps S701 to S704 makes it possible to output a radiographic image with reduced noise. Note that if the performance of the image processing unit 22 makes it impossible to process the entire image at once, the image may be divided into small regions of an appropriate size (for example, 256 x 256 pixels) and then processed.

In this way, in the series of image processing according to this embodiment, the inference processing unit 262 processes the image using the learned model created by the learning processing unit 261, thereby making it possible to generate radiographic images with suitable noise reduction in the digital radiography device.

Furthermore, the learning processing unit 261 in this embodiment can take manufacturing variability into account using the manufacturing variability analysis unit 263. Therefore, when applying processing using machine learning technology to products that have manufacturing variability due to mass production, it is possible to output radiographic images with suitably reduced noise even when the characteristics of the learning system and the inference system do not match.

Figure 8 shows an example of a series of radiographic images before and after image processing according to this embodiment. The figure illustrates the results when noise reduction processing by the noise reduction processing unit 26 is applied to an input radiographic image 801. In the radiographic image 802 after noise reduction when no measures against manufacturing variability are taken and the characteristics of the learning system and the inference system do not match due to manufacturing variability, noise-like artifacts occur. On the other hand, it can be seen that the radiographic image 803 after noise reduction when measures against manufacturing variability in this patent are taken achieves suitable noise reduction without generating artifacts.

(Modification of the learning process)
In the above-described embodiment, the influence of the manufacturing variation analyzed by the manufacturing variation analysis unit 263 is reflected in the artificial noise added by the learning data generation unit 264, and this is added and learned, thereby ensuring redundancy with respect to the manufacturing variation.

However, as an alternative embodiment, the following configuration is also possible:

In this embodiment, the influence of the manufacturing variability analyzed by the manufacturing variability analysis unit 263 is treated separately and is applied as a regularization term of the loss function in the parameter update process. In other words, the change in the characteristics of the imaging device caused by the manufacturing variability of the imaging device may be added as a regularization term in the learning.

When the inference process using the network under training is f _netstudy , the input data to which artificial noise has been added is I _in , the correct answer data is I _ans , and the regularization parameter is w, the loss function L is given by, for example, (Equation 5).

Let I _in (i) denote the value of the i-th pixel in the image I _in , and the number of pixels in the image is n. The regularization parameter w is a term determined based on the variation in characteristics caused by manufacturing variations as shown in Figs. 5 and 6.

Even with the above configuration, when applying machine learning technology processing to products that have manufacturing variations associated with mass production, it is possible to generate a machine learning model that can output images with optimal noise reduction.

Example 2
Although the configuration for noise reduction processing of radiographic images has been described as image processing, in the present invention, the processing to be learned is not limited to noise reduction processing of radiographic images. In other words, any image processing that can be realized by machine learning using appropriate correct answer data can be applied. The image processing unit 22 in the configurations of Figures 1A, 1B, 2A, and 2B can have a configuration having an arbitrary machine learning processing unit 91 as shown in Figure 9. The other configurations are the same as those described in Figures 1A, 1B, 2A, and 2B.

Below, as an example of another machine learning process, a configuration for super-resolution processing will be described with reference to FIG. 10. FIG. 10 shows a schematic example of the configuration for learning a machine learning neural network model according to this embodiment.

Here, a set of low-resolution and high-resolution data is used as the training data. From the original data 106, the training image generation unit 264 performs an input data creation process 106 (here, a process of lowering the resolution of the original data 106) and a correct answer data creation process 107 to create a set of training data.

The neural network model 102 of the inference processing unit 262 is applied to the low-resolution input data 101 in the training data, a loss function is calculated from the inference data 103 with improved resolution and the correct answer data 104 in the training data, and the parameters of the neural network model 33 are updated based on the loss function. Here, the loss function indicates the error between the inference data 103 and the correct answer data 104.

In super-resolution processing, learning must be performed to reproduce high-resolution data from low-resolution data, but the resolution of the input data changes due to manufacturing variations. To address this, the manufacturing variation analysis unit 263 analyzes the above-mentioned resolution variations and reflects them in the processing of the learning image generation unit 264. For example, when creating the input data 101 in the input data creation process 105 and when creating the correct answer data 104 in the correct answer data creation process 107, a spatial filter is created to change the resolution within the range expected due to manufacturing variations, and a resolution adjustment process is performed on the original data. In other words, learning data is generated that includes a radiographic image whose resolution has been adjusted using a spatial filter generated based on the manufacturing variations of the radiography device. As a result, the inference processing using the trained neural network is equivalent to the processing shown in FIG. 7.

As explained above, even in super-resolution processing, it is possible to generate a machine learning model that can output an image with optimally restored resolution even in products with manufacturing variations associated with mass production, where the characteristics of the learning system and the inference system do not match.

Other examples of the machine learning processing unit 91 include various types of segmentation processing, frequency enhancement processing, and image style conversion such as the night view mode of a camera, and can be applied to any machine learning processing. In addition, characteristics that vary due to manufacturing variations are not limited to the phosphor and lens resolution, noise characteristics, and sensitivity as described above. For example, any characteristics of various devices can be handled, such as color characteristics, the number and location of missing pixels, voltage values and dark current values during device operation, and individual differences in circuit components.

(Variation 1)
Regarding the machine learning model used by the arithmetic processing unit 266, as the configuration of the CNN, a variational auto-encoder (VAE), a fully convolutional network (FCN), or a combination of any layer configuration such as SegNet and DenseNet can be used. In addition, for example, a capsule network (CapsNet) may be used. Here, in a general neural network, each unit (each neuron or each node) is configured to output a scalar value, so that, for example, spatial information regarding the spatial positional relationship (relative position) between features in an image is reduced. As a result, for example, learning can be performed to reduce the effects of local distortion and translation of the image. On the other hand, in a capsule network, each unit (each capsule) is configured to output spatial information as a vector, so that, for example, spatial information is retained. This makes it possible to perform learning that takes into account, for example, the spatial relationship between features in an image.

(Variation 2)
Furthermore, the learning data for the various trained models is not limited to data obtained using the radiation detector itself that actually captures images, and may be data obtained using the same type of radiation detector or the same kind of radiation detector, depending on the desired configuration. Note that the trained models for noise reduction processing according to the above-mentioned embodiments and modified examples are considered to extract, for example, the magnitude of the luminance values of the radiation image, the order, gradient, position, distribution, continuity, etc. of the bright and dark areas as part of the feature quantities, and use them in the estimation processing related to the generation of a noise-reduced radiation image.

The trained models according to the above-mentioned embodiments and modified examples can be provided in the control unit 20. The trained models may be configured, for example, as software modules executed by a processor such as a CPU, MPU, GPU, or FPGA, or may be configured as circuits that perform specific functions such as an ASIC. These trained models may also be provided in another server device connected to the control unit 20. In this case, the control unit 20 can use the trained models by connecting to a server or the like that has the trained models via any network such as the Internet. Here, the server that has the trained models may be, for example, a cloud server, a fog server, or an edge server.

Other Examples
The present invention can also be realized by providing software (programs) that realize one or more functions of the various embodiments and modifications described above to a system or device via a network or a storage medium, and having a computer in the system or device read and execute the programs. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

In this case, the processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). The processor or circuitry may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

The present invention has been described above with reference to examples and modifications, but the present invention is not limited to the above examples and modifications. The present invention also includes inventions that have been modified without going against the spirit of the present invention, and inventions that are equivalent to the present invention. In addition, the above-mentioned examples and modifications can be combined as appropriate without going against the spirit of the present invention.

10 Radiation detector 20 Control unit (image processing device)
22 Image processing unit

Claims (13)

an acquisition means for acquiring a first radiographic image of the object to be inspected captured by the radiographic imaging device;
a generating means for generating a second radiographic image having reduced noise compared to the first radiographic image by inputting a first radiographic image acquired by the acquiring means into a trained model obtained by training using training data including a radiographic image obtained by adding noise simulating system noise of the radiographic image apparatus, which is obtained by using a distribution based on manufacturing variability of the radiographic image apparatus, and noise simulating quantum noise contained in radiation;
A radiation image processing device comprising: The radiation image processing device according to claim 1, characterized in that the generating means obtains noise simulating the system noise by using a distribution indicating the variation in system noise characteristics caused by manufacturing variations in the radiation imaging device. The radiation image processing device according to claim 2, characterized in that the generating means obtains the distribution indicating the variance of the system noise characteristics by using the average and variance of a plurality of system noise characteristics. The radiation image processing device according to claim 3, characterized in that the generating means obtains noise that imitates the system noise by using a system noise characteristic that is randomly selected from the plurality of system noise characteristics. The radiation image processing device according to any one of claims 1 to 4, characterized in that the generating means obtains the noise simulating quantum noise by using a distribution indicating the variation in MTF characteristics within the surface of a phosphor provided in the radiation imaging device, which is caused by manufacturing variations in the radiation imaging device. The radiation image processing device according to claim 5, characterized in that the generating means obtains, as the noise simulating the quantum noise, a noise in which the high frequency components contained in the quantum noise are attenuated, by using a distribution showing the variation in the characteristics of the MTF. The radiation image processing device according to claim 5 or 6, characterized in that the generating means obtains a distribution indicating the variance of the MTF characteristics by using the average and variance of the characteristics of a plurality of MTFs. The radiation image processing device according to claim 7, characterized in that the generating means obtains the noise simulating the quantum noise by using MTF characteristics randomly selected from the characteristics of the plurality of MTFs. 9. The radiation image processing device according to claim 1, wherein the generating means obtains a distribution based on manufacturing variability of the radiation imaging device by using statistical information obtained based on a manufacturing record of a plurality of manufactured radiation imaging devices of one model. 10. The radiation image processing device according to claim 1, wherein the generating means obtains a distribution based on manufacturing variability of the radiation imaging device by using statistical information obtained based on manufacturing records of a plurality of different models of imaging device. an acquiring step of acquiring a first radiographic image of the object to be inspected captured by a radiographic apparatus;
a generating step of generating a second radiographic image having reduced noise compared to the first radiographic image by inputting the first radiographic image acquired in the acquiring step into a trained model obtained by training using training data including a radiographic image obtained by adding noise simulating system noise of the radiographic image apparatus, which is obtained by using a distribution based on manufacturing variability of the radiographic image apparatus, and noise simulating quantum noise contained in radiation;
A radiation image processing method comprising: A program for causing a computer to function as each of the means of the radiation image processing device according to any one of claims 1 to 10. a radiation imaging device for capturing a radiation image;
A radiation image processing system comprising: the radiation image processing apparatus according to claim 1 , communicably connected to the radiation imaging apparatus.

Images