JP7658249B2 - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and an image processing program.

There is known an ultrasound diagnostic device that transmits an ultrasound beam to a subject and detects the time and intensity of the transmitted ultrasound beam as it reflects off tissues and blood flow within the body and returns, thereby generating a color Doppler image of the state inside the subject (see, for example, Patent Documents 1, 2, and 3).

Color Doppler images are generally generated by overlaying a B-image (Brightness mode Imaging), which is a grayscale representation of a tomographic image inside the body, with a flow image (Color Flow Imaging or Tissue Doppler Imaging) that shows blood flow information (blood flow velocity, blood flow direction, power, velocity variance) and the movement of body tissues such as the myocardial wall in color.

For ease of explanation, the signal that generates the B image will be referred to as the "B signal" and the signal that generates the flow image will be referred to as the "flow signal" below.

Japanese Patent Application Publication No. 11-169368 JP 2008-284287 A JP 2017-042606 A

In recent years, due to the need to more clearly represent the texture of internal structures and tissues in a manner that matches the ease of viewing for users and the color characteristics of display devices (such as LCDs), B images are often represented as color images (hereinafter also referred to as "colored B images") rather than grayscale images. For example, such B images are images in which the signal (e.g., signal intensity) of the reflected ultrasonic wave from inside the subject is converted into color information using a color map. Note that a color image means an image in which the pixel value of each pixel in the image is represented as a combination of values (i.e., density) of two or more color components such as in the RGB color space.

However, when a flow image is synthesized with such a colored B image to generate a color Doppler image, the color impression of both images changes in the synthesized color Doppler image, and the visibility of the structural characteristics of tissues and the characteristics of the spread and velocity of blood flow expressed by color tends to deteriorate.

In particular, since color Doppler images combine images showing two different conditions, blood flow and a tomographic image inside the body, it is necessary to make it easy to visually distinguish the signal strength of the components contained in these signals. For example, even in areas where the speed and power of the flow signal are low, it is preferable to display the continuously changing blood flow without interruption and to stably display the exact position of thin blood vessels.

Figures 1 to 3 show examples of color Doppler images generated using three types of image synthesis methods related to conventional technology.

Figure 1 shows an example of a color Doppler image generated by combining a flow image with a grayscale B image. As can be seen by referring to Figure 1, when the B image is a grayscale image, there is no color mixing, resulting in a color Doppler image in which the characteristics of both the B image and the flow image are relatively easy to see. However, in this case, it is not possible to add color to the B image or adjust the display color of the B image (for example, adjust the hue or saturation).

Figure 2 shows an example of a color Doppler image generated by image synthesis of a flow image so as to be overlaid on a colored B image. As can be seen from Figure 2, in a color Doppler image generated by this method, the B image is hidden in the area where the flow image is displayed. This causes a problem in that the visibility of the tissue structure represented by the B image is poor, and the relationship between the behavior of the blood flow shown on the flow image and the behavior of the tissue shown on the B image cannot be seen.

In addition, there is also a method in which, when overwriting a colored B image with a flow image, whether to display a B image or a flow image for each pixel forming the image is determined according to the echo intensity and color intensity, and the pixel value of either the B image or the flow image is selected for each pixel based on the result of the determination, to generate a color Doppler image. However, even with this method, the configuration is such that only one of the B image or the flow image is displayed for each pixel region, and the above-mentioned problems cannot be resolved.

Figure 3 shows an example of a color Doppler image generated by superimposing a colored B image and a flow image by transparent synthesis (also called alpha blending). In this example, the colored B image and the flow image are weighted and added at a predetermined transparency ratio (i.e., α value) to generate a color Doppler image. As can be seen from Figure 3, in such a color Doppler image, the impression of the color of the colored B image and/or the flow image changes depending on the α value, and the visibility of the structural features of tissues and the characteristics of the spread and speed of blood flow expressed by chromaticity (representing a color expressed by hue and saturation; the same applies below) is reduced.

The present disclosure has been made in consideration of the above problems, and aims to provide an image processing device, an image processing method, and an image processing program that can improve the visibility of the features expressed by each of two medical images in a composite image for display that is generated by image-compositing two colored medical images relating to the same part of a subject.

The present disclosure, which solves the above-mentioned problems, is
1. An image processing device that synthesizes first and second medical images related to the same part of a subject, the first and second medical images being expressed as color images, the image processing device comprising:
a base composite image generating unit that generates a base composite image by performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image; and
a contribution determining unit that determines, for each pixel region, a contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a color correction unit that generates a composite image for display by adding color components of pixel values of each pixel region of the first medical image to pixel values of each pixel region of the base composite image with weights according to the contribution rates;
The image processing device includes:

In other respects,
1. An image processing method for synthesizing first and second medical images of a same part of a subject, the first and second medical images being expressed as color images, the method comprising the steps of:
a process of performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image to generate a base synthetic image;
determining, for each pixel region, a degree of contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a process of generating a composite image for display by adding a color component of a pixel value of each pixel region of the first medical image to a pixel value of each pixel region of the base composite image with a weight according to the contribution degree;
The image processing method includes the steps of:

In other respects,
A program for causing a computer to execute image processing for image synthesis of first and second medical images relating to the same part of a subject, the first and second medical images being represented as color images, the program comprising:
a process of performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image to generate a base synthetic image;
determining, for each pixel region, a degree of contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a process of generating a composite image for display by adding a color component of a pixel value of each pixel region of the first medical image to a pixel value of each pixel region of the base composite image with a weight according to the contribution degree;
It is an image processing program having the following structure.

The image processing device according to the present disclosure can improve the visibility of the features expressed by each of the two medical images in a composite image for display, which is generated by image-compositing two colored medical images of the same part of a subject.

FIG. 13 is a diagram showing an example of a color Doppler image generated by combining a flow image with a B image expressed in grayscale. FIG. 13 is a diagram showing an example of a color Doppler image generated by image synthesis of a flow image so as to be overlaid on a colored B image. FIG. 1 is a diagram showing an example of a color Doppler image generated by superimposing a colored B image and a flow image by transparent synthesis (also called alpha blending); FIG. 1 is a diagram showing the configuration of an ultrasonic diagnostic apparatus according to an embodiment of the present invention. FIG. 1 is a diagram showing a configuration of an image processing unit according to an embodiment of the present invention. FIG. 1 is a diagram showing an example of an image generated by processing in each part of an image processing unit; FIG. 13 is a diagram showing an example of a B image generated by a B image generating unit; FIG. 1 is a diagram showing an example of a flow image generated by a flow image generating unit; FIG. 13 is a diagram showing an example of a luminance component image of a B image generated by a division unit; FIG. 13 is a diagram showing an example of a color component image of a B image generated by a division unit; FIG. 13 is a diagram showing an example of a contribution degree image generated by a contribution degree determination unit; FIG. 13 is a diagram showing an example of a base composite image generated by a base composite image generating unit; FIG. 1 is a diagram showing an example of a color contribution component image generated by a color correction unit; FIG. 13 is a diagram showing an example of a composite image for display generated by a color correction unit; FIG. 13 is a diagram showing an example of a color map for a B image; FIG. 1 shows an example of a color map for a flow image. FIG. 13 is a diagram showing an example of a contribution degree determination table; FIG. 1 is a diagram showing a configuration of a color correction unit according to an embodiment of the present invention. FIG. 1 is a diagram showing the configuration of an ultrasound diagnostic apparatus (image processing unit) according to a first modified example. FIG. 13 is a diagram showing a color map for a B image to which a B image generating unit according to a first modified example refers; FIG. 13 is a diagram showing the configuration of an ultrasound diagnostic apparatus (image processing unit) according to Modification 2.

A preferred embodiment of the present disclosure will be described in detail below with reference to the attached drawings. In this specification and drawings, components having substantially the same functions are designated by the same reference numerals to avoid redundant description.

In this disclosure, the term "pixel region" refers to a region of one pixel, or a region that is made up of multiple pixels forming one section.

[Overall configuration of ultrasonic diagnostic device]
The configuration of an ultrasound diagnostic device according to one embodiment of the present invention (hereinafter, referred to as "ultrasound diagnostic device U") will be described below with reference to Fig. 4. Note that in this embodiment, as an example of an image processing device of the present disclosure, a mode is shown in which the device is applied to an application in which a B image and a flow image generated by the ultrasound diagnostic device U are image-combined to generate a color Doppler image. However, the subject of image composition by the image processing device of the present disclosure may be other medical images, such as an elasticity image generated by the ultrasound diagnostic device U.

Figure 4 is a diagram showing the configuration of an ultrasound diagnostic device U according to this embodiment.

The ultrasound diagnostic device U includes an ultrasound probe 1, a transmitter 2, a receiver 3, a scanner 4, a B signal generator 5, a flow signal generator 6, an image processor 7, and a display 8. The "image processing device" of the present disclosure is applied to the image processor 7 of the ultrasound diagnostic device U.

The ultrasound diagnostic device U is used to visualize the shape, properties, or dynamics inside a subject as ultrasound images and perform image diagnosis.

The ultrasonic probe 1 converts the voltage pulse generated by the transmitting unit 2 into an ultrasonic beam and transmits it into the subject, and receives ultrasonic echoes reflected within the subject, converts them into electrical signals, and outputs them to the receiving unit 3. The ultrasonic probe 1 is composed of, for example, a plurality of transducers (piezoelectric elements) arranged in a matrix, and a channel switching unit (selector) for switching and controlling the driving state of the plurality of transducers on and off individually or in block units (hereinafter referred to as "channels").

The transmitter 2 is a transmission circuit that transmits a voltage pulse as a drive signal to the ultrasonic probe 1. The transmitter 2 includes, for example, a high-frequency pulse oscillator and a pulse setting unit. The voltage pulse generated by the high-frequency pulse oscillator is adjusted to the voltage amplitude, pulse width, and timing set by the pulse setting unit, and is transmitted to each channel of the ultrasonic probe 1.

The transmitter 2 has a pulse setting section for each of the multiple channels of the ultrasound probe 1, and is capable of setting the voltage amplitude, pulse width, and timing of the voltage pulse for each of the multiple channels. For example, the transmitter 2 can change the target depth by setting appropriate delay times for the multiple channels, and can also generate different pulse waveforms when generating B images and when generating flow images (for example, one voltage pulse is sent when generating B images and four voltage pulses are sent when generating flow images).

The receiving unit 3 is a receiving circuit that receives and processes the received signals related to the ultrasonic echoes generated by the ultrasonic probe 1. The receiving unit 3 includes a preamplifier for each channel, an AD converter for each channel, and a receiving beamformer. The preamplifier amplifies the weak received signals. The AD converter converts the amplified received signals (analog signals) into digital signals. The receiving beamformer combines the signals of multiple channels into one by phasing and adding the received signals (digital signals) of each channel, and outputs the combined signals to the B signal generating unit 5 or the flow signal generating unit 6. Hereinafter, this output signal is referred to as the "received echo signal".

The scanning unit 4 controls the ultrasound probe 1, the transmitting unit 2, and the receiving unit 3. The scanning unit 4 controls a channel switching unit provided in the ultrasound probe 1 to switch between the multiple channels to be driven. The scanning unit 4 then drives (scans) the multiple channels in sequence, causing the ultrasound probe 1 to transmit ultrasound beams into the subject in the order along the scanning direction. In this way, the scanning unit 4 causes the B signal generating unit 5 and the flow signal generating unit 6 to generate two-dimensional data along the scanning direction and depth direction.

In addition to the above, the signal processing circuit is provided with a transmission/reception switching unit (not shown) that switches the signal system with the ultrasonic probe 1, and a processing system switching unit (not shown) that switches the destination to which the received echo signal is sent from the receiving unit 3. The scanning unit 4 controls the switching of the transmission/reception switching unit to switch the signal system when a voltage pulse is sent from the transmitting unit 2 to the ultrasonic probe 1 and when a received signal generated by the ultrasonic probe 1 is sent to the receiving unit 3. The scanning unit 4 also controls the switching of the processing system switching unit to send the received echo signal from the receiving unit 3 to the B signal generating unit 5 when generating a B image, and to send the received echo signal from the receiving unit 3 to the flow signal generating unit 6 when generating a flow image.

However, this configuration is merely an example, and it is of course not necessary to provide a transmission/reception switching unit or a processing system switching unit in the signal processing circuit. For example, the B signal generating unit 5 and the flow signal generating unit 6 may generate a B signal and a flow signal from the same received echo signal. In that case, it becomes possible to generate a B image and a flow image, which will be described later, in a synchronized manner.

The B signal generating unit 5 acquires the received echo signal from the receiving unit 3 and generates a B signal. The B signal is a signal that indicates the temporal change in the signal intensity of the ultrasonic echo in the depth direction (the transmission direction of the ultrasonic beam). When the ultrasonic probe 1 scans with the ultrasonic beam, the B signal generating unit 5 sequentially accumulates the B signals obtained at each scanning position in a line memory and generates two-dimensional data that constitutes a frame unit of the B image.

The B signal generating unit 5 is configured to include, for example, an envelope detection circuit, a dynamic filter, and a logarithmic compression circuit. The envelope detection circuit detects the signal strength by envelope detection of the received echo signal. The logarithmic compression circuit performs logarithmic compression on the signal strength of the received echo signal detected by the envelope detection circuit. The dynamic filter is a bandpass filter whose frequency characteristics change according to the depth, and removes noise components contained in the received echo signal.

The flow signal generating unit 6 acquires the received echo signal from the receiving unit 3 and generates a flow signal. A flow signal is a signal generated by frequency analysis of ultrasonic echoes from blood flow or moving body tissue, and represents their velocity, power, and velocity variance (Turbulence). The flow signal is calculated, for example, based on the phase difference of ultrasonic echoes from the same depth position of ultrasonic pulses transmitted continuously by the ultrasonic probe 1. The flow signal generating unit 6, like the B signal generating unit 5, sequentially accumulates flow signals obtained at each scanning position in a line memory when the ultrasonic probe 1 scans with an ultrasonic beam, and generates two-dimensional data that constitutes a frame unit of a flow image.

The flow signal generating unit 6 is configured to include, for example, an MTI filter, a quadrature detection circuit, and an autocorrelation calculation unit. The MTI (Moving Target Indication) filter is a low-pass filter that performs processing to remove clutter components (ultrasonic echoes from tissues) from stationary tissues from the received echo signal. The quadrature detection circuit mixes a reference signal that is in phase with the transmitted ultrasonic wave and a reference signal that is out of phase with the transmitted ultrasonic wave by π/2 with respect to the received echo signal to generate a quadrature detection signal (hereinafter referred to as an "IQ signal"). The autocorrelation calculation unit extracts a flow signal (velocity, power, variance) by autocorrelation calculation based on the IQ signals of ultrasonic echoes from the same depth position of consecutively transmitted ultrasonic pulses.

The image processing unit 7 receives frame-by-frame B signals and flow signals for constructing image data from the B signal generating unit 5 and the flow signal generating unit 6, respectively. The image processing unit 7 generates a B image based on the input B signals, and generates a flow image based on the input flow signals, synthesizes these images, and outputs them to the display unit 8 as a color Doppler image (details will be described later).

Here, the frame-by-frame B signal and flow signal refer to a single, integrated unit of signals required to construct a single image. However, the B signal and flow signal may be signals for a partial region of a single image. For example, the flow image may be superimposed only on a specified region of interest in the B image.

The B signal and the flow signal input from the B signal generating unit 5 and the flow signal generating unit 6 to the image processing unit 7 may be subjected to coordinate conversion processing or data interpolation processing via a DSC (Digital Scan Converter).

The display unit 8 is a monitor that displays the composite image for display (in this embodiment, a color Doppler image) output from the image processing unit 7.

The B signal generating unit 5, the flow signal generating unit 6, and the image processing unit 7 are realized by a digital arithmetic circuit composed of, for example, a DSP (Digital Signal Processor). Each of these components has a buffer memory, and intermediate data in the process of each arithmetic process and two-dimensional data in frame units generated in each arithmetic process are stored in the buffer memory.

[Detailed configuration of image processing unit]
Hereinafter, the detailed configuration of the image processing unit 7 according to this embodiment will be described with reference to FIGS.

Figure 5 is a diagram showing the configuration of the image processing unit 7 according to this embodiment. Note that the arrows in Figure 5 represent the flow of signal processing.

Figure 6 is a diagram showing an example of an image generated in the processing of each part of the image processing unit 7. Figures 7 to 14 are diagrams showing each image shown in Figure 6 individually. Figure 7 is a diagram showing an example of a B image generated by the B image generating unit 71. Figure 8 is a diagram showing an example of a flow image generated by the flow image generating unit 72. Figure 9 is a diagram showing an example of a luminance component image of the B image generated by the dividing unit 73. Figure 10 is a diagram showing an example of a color component image of the B image generated by the dividing unit 73. Figure 11 is a diagram showing an example of a contribution image generated by the contribution determination unit 75. Figure 12 is a diagram showing an example of a base composite image generated by the base composite image generating unit 74. Figure 13 is a diagram showing an example of a color contribution component image generated by the color correction unit 76. Figure 14 is a diagram showing an example of a composite image for display generated by the color correction unit 76.

The image processing unit 7 includes a B image generating unit 71, a flow image generating unit 72, a dividing unit 73, a base composite image generating unit 74, a contribution determining unit 75, and a color correcting unit 76.

The B image generating unit 71 generates a B image by acquiring the B signals generated by the B signal generating unit 5 on a frame-by-frame basis and converting the B signals corresponding to each pixel region into pixel values. In this case, the B image generating unit 71 generates the B image using, for example, a function (hereinafter referred to as a "color map for B images") that converts the signal intensity of the B signal into pixel values, which is stored in advance in a storage unit (for example, a ROM possessed by the image processing unit 7). The B image generating unit 71 converts the B signal into pixel values such that the luminance increases as the signal intensity of the B signal increases. In other words, image regions with high luminance in the B image are regions in which the tissue structure, etc. within the subject are clearly expressed, and are regions that characterize the tissue structure within the subject.

FIG. 15 is a diagram showing an example of a color map for a B image. In this embodiment, the B image is generated as a colored image by the color map for the B image as shown in FIG. 15. That is, the pixel value of each pixel region of the B image is not expressed by only one-bit luminance, but is, for example, a value expressed in 256 gradations for each of the RGB components (i.e., a color space vector for the RGB components). In the color map for the B image, for example, the signal intensity of the B signal is converted into the value of each of the RGB components by applying it to three functions f ₀ -R, f ₀ -G, and f ₀ -B that convert the signal intensity of the B signal into a value expressed in 256 gradations for each of the RGB components. Note that the pixel value of a pixel region where the signal intensity of the B signal is large is usually expressed in a state where all of the RGB components are large.

The flow image generating unit 72 acquires the flow signals generated by the flow signal generating unit 6 on a frame-by-frame basis, and generates a flow image by converting the flow signals corresponding to each pixel region into pixel values. In this case, the flow image generating unit 72 generates the flow image using a function (hereinafter referred to as a "color map for flow images") that converts flow signals into pixel values, which is stored in advance in a storage unit (e.g., a ROM possessed by the image processing unit 7). Here, the flow image generating unit 72 determines pixel values based on two components of flow signals, velocity and power. Typically, the flow image generating unit 72 converts flow signals into pixel values such that the luminance increases as the power of the flow signal (e.g., blood flow rate) increases.

Figure 16 is a diagram showing an example of a color map for a flow image. Note that the velocity component on the horizontal axis shown in Figure 16 is such that the center of the graph represents a velocity of 0, the left side of the graph represents the velocity moving away from the center, and the right side of the graph represents the velocity moving toward the center.

The pixel value of the flow image is, for example, a color space vector related to RGB components, and is a value expressed in 256 gradations for each of the RGB components. In the color map for the flow image, for example, the direction of the speed toward the ultrasound probe 1 is set to be red with a large R component (area D in FIG. 16), and conversely, the direction away from the ultrasound probe 1 is set to be blue with a large B component (area B in FIG. 16). Also, in the color map for the flow image, the higher the power, the larger all the RGB color components are set (area B in FIG. 16) (area D in FIG. 16). On the other hand, when the power is low, it is highly likely that the component is a background noise component, so the ratio of the RGB color components is set to be brown, approximately 12:10:5 (area A in FIG. 16) (area E in FIG. 16). Furthermore, when the velocity is low, it is highly likely that this is a motion artifact caused by unwanted Doppler signals from objects other than blood flow, so the ratio of RGB color components is similarly set to brown, approximately 12:10:5 (area C in Figure 16).

The division unit 73 divides the pixel values of each pixel region of the B image into a luminance component and a color component, and generates an image in which only the luminance component is extracted from the colored B image (hereinafter referred to as the "luminance component image"), and an image in which only the color component is extracted from the colored B image (hereinafter referred to as the "color component image"). The division unit 73 generates the luminance component image and the color component image so that an image obtained by adding the luminance component image and the color component image becomes the original colored B image.

The division unit 73 calculates the luminance component of the pixel value of each pixel region of the B image (i.e., the pixel value of the luminance component image) as the average value of each RGB component of the pixel value of each pixel region of the B image, for example, as shown in the following equation (1).
I _B1 = avg (I _B0R , I _B0G , I _B0B )...(1)
(where _IB1 is the pixel value of the luminance component image, and avg( _IB0R , _IB0G , _IB0B ) is the average value of the R component _IB0R of the pixel value of the B image, the G component _IB0G of the pixel value of the B image, and the B component _IB0B of the pixel value of the B image)

Then, the division unit 73 calculates the color components of the pixel values of each pixel region of the B image (i.e., the pixel values of the color component image) by subtracting the luminance components of the pixel values of each pixel region of the B image calculated using equation (1) from the RGB components of the pixel values of each pixel region of the B image, for example, as shown in the following equations (2), (3), and (4).
I _B2R = I _B0R - _{I B1} ...(2)
I _B2G = I _B0G - _{I B1} ...(3)
I _B2B = I _B0B - _{I B1} ...(4)
(where _IB2R is the R component of the color component image, _IB2G is the G component of the color component image, _IB2B is the B component of the color component image, _IB0R is the R component of the pixel value of the B image, _IB0G is the G component of the pixel value of the B image, _IB0B is the B component of the pixel value of the B image, and _IB1 is the pixel value of the luminance component image)

The base composite image generating unit 74 generates a base composite image by adding the luminance component of the pixel value of each pixel region of the B image to the pixel value of each pixel region of the flow image. For example, the base composite image generating unit 74 associates each pixel region of the B image (luminance component image) with each pixel region of the flow image, and calculates the pixel value of each pixel region of the base composite image using the following formulas (5) to (8). Note that the weighting coefficients w _B1 and w _F in formula (5) are set to appropriate values (for example, w _B1 : 0.5, w _F : 0.5) so as to maintain the visibility of the tissue structure and the spread of blood flow represented by the B image and the flow image, respectively.

In this case, the base composite image generating unit 74 calculates the luminance component of the pixel value of the base composite image by adding the luminance component of the pixel value of the B image and the luminance component of the pixel value of the flow image, and then calculates each of the RGB components of the pixel value of this base composite image by adding the color component of the pixel value of the flow image (the color component from which the luminance component has been subtracted in advance, using processing similar to that of the above equations (2), (3), and (4)) to the luminance component of the pixel value of this base composite image.

I _Out1a = I _B1・w _B1 +I _Fa・w _F …(5)
(where I _Out1a is the luminance component of the pixel value of the base composite image, I _B1 is the pixel value of the luminance component image, I _Fa is the luminance component of the pixel value of the flow image, w _B1 is the weighting factor for the B image, and w _F is the weighting factor for the flow image).

I _Out1R = I _FR + I _Out1a ...(6)
I _Out1G = I _FG + I _Out1a ...(7)
I _Out1B = I _FB + I _Out1a ...(8)
(wherein, I _Out1R is the R component of the pixel value of the base composite image, I _Out1G is the G component of the pixel value of the base composite image, I _Out1B is the B component of the pixel value of the base composite image, I _FR is the value obtained by subtracting the luminance component from the R component of the pixel value of the flow image, I _FG is the value obtained by subtracting the luminance component from the G component of the pixel value of the flow image, I _FB is the value obtained by subtracting the luminance component from the B component of the pixel value of the flow image, w _B1 is the weighting coefficient for the B image, and w _F is the weighting coefficient for the flow image)

The image synthesis process by the base synthetic image generating unit 74 is based on the same idea as the conventional color Doppler image generation method described with reference to FIG. 1. That is, when generating the base synthetic image, the B image to be synthesized is represented only by brightness, so that no color mixing occurs on the base synthetic image, and the characteristics of both the B image and the flow image (for example, the characteristics of the tissue structure within the subject and the characteristics of the spread of blood flow) can be visually recognized.

The contribution determination unit 75 determines, for each pixel region, the contribution (hereinafter simply referred to as "contribution") regarding the color that each pixel region of the B image should give to each pixel region of the base composite image, based on the luminance component of the pixel value of each pixel region of the B image (i.e., the pixel value of each pixel region of the luminance component image) and the luminance component of the pixel value of each pixel region of the flow image. The contribution determination unit 75, for example, compares the luminance of the B image and the luminance of the flow image for each corresponding pixel region, and determines the contribution for each pixel region based on that relationship.

The contribution degree is determined for each pixel area of the image size of the B image, for example, in units of one pixel. The contribution degree acts like a mask image that is applied to the color component image of the B image (hereinafter also referred to as the "contribution degree image").

The contribution determination unit 75 determines the contribution of each pixel region from the luminance component of the pixel value of each pixel region of the B image and the luminance component of the pixel value of each pixel region of the flow image, for example, using a contribution determination table stored in advance in a storage unit (for example, a ROM possessed by the image processing unit 7).

Fig. 17 is a diagram showing an example of a contribution degree determination table. For example, as shown in the contribution degree determination table of Fig. 17, the contribution degree of each pixel region is calculated as a scalar value of 0≦ _wBc ≦1 to be multiplied with each of the RGB components of the B image. For example, the contribution degree takes a value of 1.0 in a pixel region that displays only the B image, and takes a value of 0.0 in a pixel region that displays only the blood flow of the flow image. The contribution degree is determined to a value close to 1.0 in a pixel region where the impression of the B image is greater than the impression of the flow image, and is determined to a value close to 0.0 in a pixel region where the impression of the color flow image is greater than the impression of the B image. Specifically, as shown in the contribution degree determination table of Fig. 17, the contribution degree is determined based on the relationship between the luminance component of the pixel value of the B image and the luminance component of the pixel value of the flow image, and the larger the luminance component of the pixel value of the B image relative to the luminance component of the pixel value of the flow image, the larger the contribution degree is determined to be.

However, unlike typical mask images, the contribution image does not have a binary value of 0 or 1, but has a gradation between 0.0 and 1.0. This allows the colors of the B image and the flow image to be reflected in the pixel regions that express their respective characteristics. For example, a large contribution (i.e., weight) is given to pixel regions that should strongly express the colors of the B image, and a small contribution (i.e., weight) is given to pixel regions that do not, determining the degree to which the colors of each pixel region of the color component image (B image) are expressed in the final composite image for display.

B images generally represent internal structures and the texture of tissues, and therefore have characteristic image information related to tissue structure over a wide range of the image. On the other hand, flow images generally have characteristic image information related to the speed and volume of blood flow only in localized regions of the image where blood flow is present, and have no characteristic image information in other regions. Therefore, by generating the "degree of contribution," which indicates how much the color of the B image affects the composite image for display, so that it is low in regions where blood flow is present and high in regions where blood flow is not present, it becomes possible to reflect the color of the B image over a wide range in the final composite image for display, while retaining the color of the flow image in regions where blood flow is present.

In addition, in areas where the speed and power of the flow signal are low, the contribution is increased so that the color of the B image becomes stronger. This allows the characteristics of the tissue structure represented by the B image in such areas to be clearly displayed, while the continuously changing blood flow represented by the flow image is displayed without interruption, and the exact position of thin blood vessels is displayed stably.

The color correction unit 76 adds the color components of the pixel values of each pixel region of the B image to the pixel values of each pixel region of the base composite image with weights according to the contributions, to generate a composite image for display. As a result, in the composite image for display, the colors of the B image are restored to image regions where the impression of the B image is greater than the impression of the color flow image.

Figure 18 is a diagram showing the configuration of the color correction unit 76 according to this embodiment. For example, the color correction unit 76 first performs a contribution image weighting process (i.e., masking process) on the color component image of the B image to generate a color contribution component image of the B image. The color correction unit 76 then adds the color components of the pixel values of each pixel region of the color contribution component image of the B image to the pixel values of each pixel region of the base composite image to generate a composite image for display that is ultimately to be displayed on the display unit 8.

The color correction unit 76 calculates the pixel values of each pixel region of the composite image for display using, for example, the following equations (9), (10), and (11).
I _Out2R = I _B2R・w _B2 + I _Out1R …(9)
I _Out2G = I _B2G・w _B2 + I _Out1G … (10)
I _Out2B = I _B2B・w _B2 + I _Out1B … (11)
(wherein I _Out2R is the R component of the pixel value of the composite image for display, I _Out2G is the G component of the pixel value of the composite image for display, I _Out2B is the B component of the pixel value of the composite image for display, I _Out1R is the R component of the pixel value of the base composite image, I _Out1G is the G component of the pixel value of the base composite image, I _Out1B is the B component of the pixel value of the base composite image, I _B2R is the R component of the pixel value of the color component image, I _B2G is the G component of the pixel value of the color component image, I _B2B is the B component of the pixel value of the color component image, and w _B2 is the contribution of the RGB components of the color component image to each color)

In this way, the image processing unit 7 according to this embodiment generates a base composite image (in which both are reflected by the magnitude of brightness) that reflects the spatial information of the entire object represented by each of the B image and the flow image (for example, tissue structure, spread of blood flow, etc.), and then, for each image region, strongly reflects the color tone of the image between the B image and the flow image that has a larger local feature value of the object on the base composite image to create a composite image for display.

This makes it possible to generate a composite image for display that maintains the visibility of the spatial information of the entire object represented by each of the B image and flow image (e.g., tissue structure, blood flow spread, etc.), while improving the visibility of the local features of each of the B image and flow image (e.g., areas with large tissue movement or areas with fast blood flow speeds, etc.) (see Figure 14).

In addition, the image synthesis method used by the image processing unit 7 according to this embodiment prevents the flow image from being overlaid on the B image, which would otherwise hide the B image. Therefore, in the color Doppler image generated by the image processing unit 7 according to this embodiment, it is possible to visually confirm the relationship between the behavior of the blood flow shown in the flow image and the behavior of the tissue shown in the B image. Furthermore, in such a color Doppler image, even in areas where the speed and power of the flow signal are low, the blood flow that changes successively is displayed without interruption, and the exact position of thin blood vessels is stably displayed.

(Variation 1)
Fig. 19 is a diagram showing the configuration of an ultrasound diagnostic apparatus U (image processing unit 7) according to Modification 1. Fig. 20 is a diagram showing a B image color map referred to by a B image generating unit 71 according to Modification 1 (see Figs. 20B and 20C).

Usually, in the ultrasound diagnostic device U, colored B images are used not only to generate color Doppler images, but also for standalone display. Therefore, in the ultrasound diagnostic device U according to the above embodiment, the B image generating unit 71 uses a color map for B images (see FIG. 20A) to directly convert B signals (i.e., the signal intensity of the reflected wave from within the subject) into pixel values of each color component, and generates a colored B image from the B signals.

However, in this case, when performing image synthesis processing of the B image and the flow image, as in the ultrasound diagnostic device U according to the above embodiment, it is necessary to perform processing to divide the B image generated as a color image by the B image generation unit 71 into a color component image and a luminance component image by the division unit 73.

In the ultrasound diagnostic apparatus U according to this modification, from the viewpoint of reducing the computational load when generating a color Doppler image, a configuration is adopted in which the B image generating unit 71 generates a luminance component image of the B image using a first function f _b (hereinafter also referred to as "luminance component image generating map") (see FIG. 20B) that converts the signal intensity of the B signal into luminance, and generates a color component image of the B image using second functions f _c -B, f _c -G, f _c -R (hereinafter also referred to as "color component image generating map") (see FIG. 20C) that convert the signal intensity of the B signal into chromaticity. This makes it possible to omit the configuration of the division unit 73.

The luminance component image generation map and color component image generation map referenced by the B image generation unit 71 in this modified example are set, for example, as shown in the following formulas (12), (13), (14), and (15) so as to correspond to the color map for the B image in the above embodiment.

y=avg(R,G,B)...(12)
(where y is the luminance of the luminance component image generation map, and avg(R,G,B) is the average value of the RGB components of the color map for the B image)

CR=Ry...(13)
CG=Gy...(14)
CB=B-y...(15)
(where CR is the R component of the color component image generation map, CG is the G component of the color component image generation map, CB is the B component of the color component image generation map, R is the R component of the color map for the B image, G is the G component of the color map for the B image, and B is the B component of the color map for the B image)

The image data format for the B image in this modified example is a data format that allows brightness and chromaticity to be stored separately.

According to the ultrasound diagnostic device U of this modified example, at the stage of generating the B image, it is possible to generate the luminance component image of the B image and the color component image of the B image separately. This makes it possible to omit the process of dividing the luminance component image of the B image and the color component image of the B image from the color image of the B image, and to reduce the computational load when generating a color Doppler image.

(Variation 2)
FIG. 21 is a diagram showing the configuration of an ultrasound diagnostic apparatus U (image processing unit 7) according to the second modification.

The image processing unit 7 in this modified example has multiple image synthesis units for synthesizing two colored medical images relating to the same part of the subject, and by cascading the multiple image synthesis units, it is possible to synthesize three or more medical images relating to the same part of the subject.

Figure 21 shows a structure in which a first image synthesis unit 70a and a second image synthesis unit 70b are connected in a two-stage cascade. The first image synthesis unit 70a and the second image synthesis unit 70b are each composed of a division unit 73, a base synthesis image generation unit 74, a contribution determination unit 75, and a color correction unit 76. The first image synthesis unit 70a and the second image synthesis unit 70b each synthesize two input medical images using a method similar to that described in the above embodiment.

Here, the first image synthesis unit 70a receives the first medical image and the second medical image. The first image synthesis unit 70a then synthesizes the first medical image and the second medical image to output a first synthetic image for display. The second image synthesis unit 70b receives the first synthetic image for display output from the first image synthesis unit 70a and a third medical image. The second image synthesis unit 70b then synthesizes the third medical image and the first synthetic image for display to output a second synthetic image for display. The ultrasound diagnostic device U then displays, for example, the second synthetic image for display on the display unit 8.

In this way, by connecting in multiple stages, it becomes possible to composite three or more medical images. For example, if a B image is composited with an image highlighting nerve tissue and then a flow image, a medical image can be provided in which three types of information about the inside of the body - the internal structure, the location of the nerve tissue, and blood flow information - can be visually confirmed just by looking at one medical image.

(Variation 3)
The ultrasound diagnostic device U according to the above embodiment is configured to determine the degree of contribution of color that each pixel region of the B image should give to each pixel region of the base composite image based on the magnitude relationship between the luminance component of the pixel value of the B image and the luminance component of the pixel value of the flow image. This configuration is useful in that it makes it possible to distinguish, for each pixel region, a pixel region that gives a strong impression of the B image from a pixel region that gives a strong impression of the flow image, and to determine the degree of contribution in terms of gradations.

From the viewpoint of reducing the computational load, the above comparison process for determining the contribution degree may be omitted. That is, the contribution degree determination unit 75 may determine the contribution degree based only on the image information (luminance component or color component) of the flow image. In this case, the contribution degree determination unit 75 may determine the contribution degree so that, for example, the contribution degree is small for areas where the signal strength of the flow signal is large, and the contribution degree is large for areas where the signal strength of the flow signal is small.

However, in this case, there is a risk that the visibility of the tissue structure of image B may be partially degraded in the final composite image for display. Therefore, the image synthesis method according to this modified example is a method that is adopted when the aim is to reduce the computational load even at the expense of degradation in image quality.

(Variation 4)
In general, in determining whether a feature of a B image or a flow image is present in a given region on a color Doppler image, it is more useful to compare the luminance components than to compare only one color component (e.g., the R component) because the luminance component represents the signal intensity of each of the B signal and the flow signal.

However, from the perspective of ensuring visibility, when it is desired to avoid combining specific colors or to emphasize the presence of specific colors, it may be more useful to compare the color components of the B image and the flow image rather than comparing the luminance components of the B image and the flow image.

From this perspective, the contribution determination unit 75 may be configured to determine the contribution based on the chromaticity in the first color space vector of the B image and the chromaticity in the second color space vector of the flow image. In this case, the contribution determination unit 75 may be configured to determine the contribution by comparing the chromaticity in the same color space vector between the B image and the flow image, or may be configured to determine the contribution by comparing the chromaticity in different color space vectors.

(Variation 5)
In the above embodiment, the signal intensity of the B signal is converted into RGB components to configure the entire image area of the B image as a color image. However, the B image applied to the image synthesis process according to the present disclosure may be one in which only a part of the entire image area is configured as a color image.

For example, the B image input to the image processing unit 7 may be obtained by coloring the area of the puncture needle detected in a grayscale B image generated by converting the signal intensity of the B signal into luminance.

The B image input to the image processing unit 7 may also be obtained by coloring the area of nerve tissue detected in a grayscale B image generated by converting the signal intensity of the B signal into luminance.

Even when such an image is used as the B image, in the image synthesis method according to the prior art, when the image is synthesized with the flow image, color mixing occurs in some image regions, and there is a risk of degrading the visibility of the B image and/or the flow image. However, the image synthesis method according to the present disclosure can suppress such degradation of visibility and generate a composite image for display that improves the visibility of the characteristics of the B image and the flow image.

(Variation 6)
In the above embodiment, the contribution degree determining section 75 compares the luminance of the B image and the luminance of the flow image for each corresponding pixel region on a pixel-by-pixel basis, and determines the contribution degree for each pixel region.

However, the contribution determination unit 75 may compare the luminance of the B image and the luminance of the flow image for each corresponding pixel region, where the pixel regions are each made up of a plurality of adjacent pixels, and determine the contribution for each pixel region to generate a contribution image. In this case, the contribution determination unit 75 may determine the contribution for each pixel region, for example, by using the average value of the luminance in the pixel regions to be compared.

In addition, the contribution degree determination unit 75 may determine the contribution degree for each characteristic pattern appearing in the B image and/or for each characteristic pattern appearing in the flow image, for example.

(Variation 7)
In the above embodiment, the base composite image generating unit 74 generates the base composite image by adding the luminance component of the pixel value of each pixel region of the B image to the pixel value of each pixel region of the flow image.

However, the base composite image generating unit 74 may use other functions for image synthesis as long as it generates a base composite image by superimposing the luminance component image of the B image on the flow image. For example, it may use an operation that compares the luminance components of both images and adopts the larger value (such as a function max(a, b)), or an operation that adopts the smaller value (such as a function min(a, b)). However, in order to clearly express the spatial information of the entire object represented by each of the B image and the flow image, it is preferable that both the luminance component of the B image and the luminance component of the flow image are reflected in the base composite image.

(Variation 8)
In the above embodiment, the chromaticity of the B image and the flow image is expressed in the RGB color space. However, the expression format of the chromaticity of the B image and the flow image is not limited to this, and the chromaticity of the B image and the flow image may be expressed in the HSV color space, the HSL color space, the YUV color space, or the like.

(Variation 9)
In the above embodiment, the contribution degree determination unit 75 expresses the contribution degree as a scalar value, and the contribution degree for each color of the RGB components of the color component image is set to the same value. However, the contribution degree may be expressed as a vector in which a different weight can be set for each component of the color space vector. By doing so, it is possible to make the color of the blood flow particularly easy to see.

(Variation 10)
In the above embodiment, image synthesis of a B image and a flow image generated by an ultrasound diagnostic device U has been shown as an example of an application target of the image processing device according to the present disclosure. However, the image processing device according to the present disclosure can perform image synthesis on various other colored medical images. For example, such medical images may be medical images captured by an X-ray imaging device, medical images captured by an MRI inspection device, and medical images captured by a thermography imaging device.

(Modification 11)
In the above embodiment, the image processing unit 7 is realized by a hardware configuration. However, a part or all of the configuration may be realized by a software configuration under program control of a microcomputer or the like.

Specific examples of the present invention have been described in detail above, but these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples given above.

The image processing device according to the present disclosure can improve the visibility of the features expressed by each of the two medical images in a composite image for display, which is generated by image-compositing two colored medical images of the same part of a subject.

Reference Signs List U Ultrasound diagnostic device 1 Ultrasound probe 2 Transmitter 3 Receiver 4 Scanner 5 B signal generator 6 Flow signal generator 7 Image processor 70a First image synthesizer 70b Second image synthesizer 71 B image generator 72 Flow image generator 73 Divider 74 Base synthesized image generator 75 Contribution degree determiner 76 Color corrector 8 Display unit

Claims (11)

1. An image processing device that synthesizes first and second medical images related to the same part of a subject, the first and second medical images being expressed as color images, the image processing device comprising:
a base composite image generating unit that generates a base composite image by performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image; and
a contribution determining unit that determines, for each pixel region, a contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a color correction unit that generates a composite image for display by adding color components of pixel values of each pixel region of the first medical image to pixel values of each pixel region of the base composite image with weights according to the contribution rates;
An image processing device comprising: The contribution degree determination unit compares a luminance component or a color component of a pixel value of each pixel region of the second medical image with a luminance component or a color component of a pixel value of each pixel region of the first medical image for each corresponding pixel region, and determines the contribution degree for each pixel region.
The image processing device according to claim 1 . The contribution degree determiner determines the contribution degree in a gradational manner such that the contribution degree increases as the luminance component of the pixel value of each pixel region of the first medical image relative to the luminance component of the pixel value of each pixel region of the second medical image increases.
The image processing device according to claim 2 . The base composite image generating unit adds a luminance component of a pixel value of each pixel region of the first medical image to a pixel value of each pixel region of the second medical image to generate the base composite image.
The image processing device according to claim 1 . a division unit that divides pixel values of each pixel region of the first medical image into a luminance component and a color component and generates a luminance component image and a color component image of the first medical image;
The image processing device according to claim 1 . a first medical image generating unit that generates the first medical image based on first sample values related to in-vivo information at each position in the subject obtained by the first imaging process;
a second medical image generating unit that generates the second medical image based on second sample values related to in-vivo information at each position in the subject obtained by the second imaging process;
The image processing device according to claim 1 , further comprising: the first medical image generation unit generates a luminance component image of the first medical image using a first function that converts the first sample value into luminance, and generates a color component image of the first medical image using a second function that converts the first sample value into chromaticity;
The image processing device according to claim 6. The image processing device is applied to an ultrasound diagnostic device,
the first medical image is a B image obtained by a first ultrasonic scan of the inside of the subject by the ultrasonic diagnostic device,
The second medical image is a flow image obtained by a second ultrasonic scan of the subject by the ultrasonic diagnostic device.
The image processing device according to claim 1 . a plurality of image synthesis units each including the base synthesis image generation unit, the contribution determination unit, and the color correction unit;
By cascading a plurality of the image synthesis units, it is possible to synthesize three or more medical images relating to the same part of the subject.
The image processing device according to claim 1 . 1. An image processing method for synthesizing first and second medical images of a same part of a subject, the first and second medical images being expressed as color images, the method comprising the steps of:
a process of performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image to generate a base synthetic image;
determining, for each pixel region, a degree of contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a process of generating a composite image for display by adding a color component of a pixel value of each pixel region of the first medical image to a pixel value of each pixel region of the base composite image with a weight according to the contribution degree;
An image processing method comprising the steps of: A program for causing a computer to execute image processing for image synthesis of first and second medical images relating to the same part of a subject, the first and second medical images being represented as color images, the program comprising:
a process of performing image synthesis so as to superimpose a luminance component image, in which only a luminance component is extracted from the first medical image, on the second medical image to generate a base synthetic image;
determining, for each pixel region, a degree of contribution of color that each pixel region of the first medical image should give to each pixel region of the base composite image based on at least a luminance component or a color component of a pixel value of each pixel region of the second medical image;
a process of generating a composite image for display by adding a color component of a pixel value of each pixel region of the first medical image to a pixel value of each pixel region of the base composite image with a weight according to the contribution degree;
An image processing program having the following steps:

Images