JP7611571B2 - IMAGE EXTRACTION METHOD, IMAGE EXTRACTION DEVICE, IMAGE EXTRACTION PROGRAM, AND RECORDING MEDIUM - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. Publication name: "2020 Graduation Research Abstracts, pages 124 and 125" Publication date: February 10, 2021 Publisher: Kansai University, School of Informatics

This disclosure relates to an image extraction method for generating an image by extracting a partial area contained in an image, an image extraction device for implementing the image extraction method, an image extraction program, and a recording medium.

Technology for cutting out any region from an image is used in photo and video processing, etc. For example, Patent Document 1 discloses a method for extracting a desired region using a graph cut method. Patent Document 2 discloses a method for sequentially adding pixels adjacent to a certain reference pixel to the pixels contained in each region using a region growing method. In particular, Patent Document 2 discloses a method for dividing each axis (R axis, G axis, B axis) of the RGB color space into s grids and extracting a desired region using a foreground color cluster and a background color cluster.

JP 2015-121901 A JP 2010-79477 A

Patent Document 1 uses a graph cut method, but due to algorithm constraints, it is difficult to extract the regions of multiple objects or estimate a dynamic color model. In addition, the graph cut method requires that the boundaries of the region to be extracted be determined from a large number of combinations, which takes time for processing.

Patent Document 2 uses a region growing method, but there are cases where similar colors are used in both the foreground and background color clusters, i.e., so-called color conflicts. For this reason, for pixels containing similar colors where color conflicts have occurred, it may be erroneously determined whether the pixel is in the foreground or the background. In particular, when extracting the regions of multiple objects within one screen, a color cluster is generated for each region. For this reason, many color conflicts occur, and accurate cutouts may not be possible.

Here, when extracting the regions of multiple objects, the image is not divided into two values, foreground and background, but is divided into multiple values, region 1, region 2, ..., region N, and each region is extracted.

In order to solve the above problem, an image extraction method according to one aspect of the present disclosure is an image extraction method for extracting a specific region from an image, and includes a receiving step of receiving a designation of a first reference pixel that is a pixel of a part of a first region included in the image, a reference pixel setting step of setting the first reference pixel as a first pixel, a probability distribution estimation step of estimating a first probability distribution that is a probability that a pixel on the image is included in the first region at coordinates in the color space based on color information in the color space of the first reference pixel, a neighboring pixel analysis step of determining whether the first neighboring pixel is a pixel included in the first region based on color information of a first neighboring pixel adjacent to the first pixel and the first probability distribution, and if it is determined that the first neighboring pixel is a pixel included in the first region, adding the first neighboring pixel to the first pixel and resetting the first neighboring pixel and the first pixel as a new first pixel, and an extraction step of extracting the reset first pixel as a first extracted image.

An image extraction device according to another aspect of the present disclosure includes a reference pixel setting unit that receives a designation of a first reference pixel that is a pixel that is part of a first region included in an image, and sets the first reference pixel as a first pixel; a probability distribution estimation unit that estimates a first probability distribution that is a probability that a pixel on the image is included in the first region at coordinates in the color space based on color information in the color space of the first reference pixel; an adjacent pixel analysis unit that determines whether the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and adds the first adjacent pixel to the first pixel and resets the first adjacent pixel and the first pixel as a new first pixel when it is determined that the first adjacent pixel is a pixel included in the first region; and an extracted image generation unit that extracts the reset first pixel as a first extraction region.

The image extraction device according to each aspect of the present disclosure may be realized by a computer. In this case, the image extraction program that causes the computer to operate as each unit (software element) of the image extraction device to realize the image extraction device on the computer, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present disclosure.

According to one aspect of the present disclosure, the accuracy of image extraction can be further improved.

1 is a flowchart for explaining an image extraction method according to a first embodiment of the present disclosure. 1 is a block diagram of an image extraction device according to a first embodiment of the present disclosure. 1 is a schematic diagram showing a process for illustrating a method for specifying a reference pixel according to the first embodiment of the present disclosure. 4 is a schematic diagram showing an example of color coordinates in RGB color space of each reference pixel according to the first embodiment of the present disclosure. FIG. 1A to 1C are schematic diagrams illustrating examples of probability distributions in an RGB color space according to the first embodiment of the present disclosure. FIG. 2 is a process diagram showing a method for analyzing adjacent pixels according to the first embodiment of the present disclosure. FIG. 11 is another schematic process diagram showing the method for analyzing adjacent pixels according to the first embodiment of the present disclosure. 1 is a schematic diagram showing how each probability distribution in an RGB color space is modified according to the first embodiment of the present disclosure; FIG. 1A to 1C are diagrams illustrating an image according to an embodiment of the present disclosure, an extraction process for the image, and an extraction result for the image. FIG. 11 is a schematic diagram showing an example of each color bucket in an RGB color space according to a comparative example. 11A and 11B are diagrams illustrating a comparison between an image extraction result according to an embodiment of the present disclosure and an image extraction result according to a comparative example. 11 is a schematic diagram showing the steps of a method for specifying a reference pixel according to a second embodiment of the present disclosure. FIG. 11 is a schematic diagram showing the steps of a method for specifying a reference pixel according to a third embodiment of the present disclosure. FIG.

[Embodiment 1]
<Image Extraction Device Overview>
The image extraction method according to the present embodiment is a method for extracting a specific region from an image and generating an image including the specific region. In particular, the image extraction method according to the present embodiment is a method for extracting a first extraction region including the first region and a second extraction region including the second region from an image having a first region and a second region, and generating a first extraction image including the first extraction region and a second extraction image including the second extraction region. The image extraction method will be described with reference to Figs. 1 and 2. Fig. 1 is a flowchart showing the image extraction method according to the present embodiment. Fig. 2 is a block diagram showing an image extraction device 2 and a display device 4 for executing the image extraction method.

The image extraction device 2 according to this embodiment includes a control unit 6. The image extraction device 2 may further include a memory 8, a communication unit 10, an operation unit 12, an input/output interface 14, and a system bus 16. The control unit 6 includes a reference pixel setting unit 20, a probability distribution estimation unit 22, an adjacent pixel analysis unit 24, and an extracted image generation unit 26. The control unit 6 may further include a display control unit 18 and a probability distribution correction unit 28.

The image extraction device 2 according to this embodiment displays an image on the display device 4 using a method described in detail below, and generates an extracted image by extracting a partial area from the image in response to a user operation.

The display device 4 is a device that displays images, and includes, for example, a display, a smartphone, or a tablet terminal. Note that in FIG. 2, the image extraction device 2 and the display device 4 are illustrated as separate entities, but this is not limited thereto, and the image extraction device 2 may include the display device 4 as a display unit.

The control unit 6 generates an extracted image by extracting a partial area from the image based on the information input to the image extraction device 2. The detailed operation of each part of the control unit 6 will be described in more detail later in conjunction with the explanation of the image extraction method according to this embodiment. Note that the control unit 6 may include a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and each of these may operate as each part of the control unit 6.

The memory 8 at least temporarily stores data necessary for the operation of each unit of the image extraction device 2, or data input from each unit. The data stored in the memory 8 is read out as appropriate from each unit of the image extraction device 2.

The communication unit 10 is a communication device that transmits at least a portion of the data read from the memory 8 to an external device of the image extraction device 2, receives data transmitted from the external device, and records the data in the memory 8. For example, in this embodiment, the communication unit 10 is capable of mutual data communication with the display device 4.

The operation unit 12 is operated, for example, by a user of the image extraction device 2. For example, the operation unit 12 may include a keyboard or the like that is used to input data from the outside for image extraction by the image extraction device 2. For example, the operation unit 12 may be used to input data for making settings for image extraction, etc.

The input/output interface 14 performs interface processing between the memory 8, the communication unit 10, and the operation unit 12 and each part of the control unit 6. Each part of the control unit 6 and the input/output interface 14 are connected via a system bus 16.

<Specifying the reference pixel>
In the image extraction method according to the present embodiment, first, a user designation of a reference pixel that is to be used as a reference for image extraction is accepted from an image to be extracted. The process of accepting the designation of the reference pixel will be described with reference to FIG.

In the image extraction method according to this embodiment, for example, it is assumed that the image 30 shown in FIG. 3 is first displayed on the display device 4 (step S2). The image extraction device 2 may, for example, control the display control unit 18 of the control unit 6 to cause the display device 4 to display the image 30.

For example, image 30 shown in FIG. 3A has a first object region 32 in which a specified object is depicted as a foreground region 31, which is a first region. Image 30 also has a background region 36 in which a specified background is depicted as a second region different from the first region. In this embodiment, a method for generating a first extracted image including first object region 32, which is a first region, and a second extracted image including background region 36, which is a second region, from image 30 will be described as an example.

Following step S2, a reception process is performed in which, for example, from a user, a designation of a first reference pixel that is a pixel of the first region and a second reference pixel that is a pixel of the second region is received (step S4). Step S4 is executed, for example, by the reference pixel setting unit 20 receiving, from the user, a designation of the first reference region 38 and the second reference region 40 shown in FIG. 3B. The first reference region 38 is designated as the first object region 32, which is the first region, and the second reference region 40 is designated as the background region 36, which is the second region.

In this embodiment, the image 30 may further have, as the foreground region 31, which is the first region, in addition to the first object region 32, a second object region 34 in which another specified object is depicted, as shown in FIG. 3(c). In this case, in step S4, the first reference region 38 may be designated for each of the first object region 32, which is the first region, and the second object region 34, as shown in FIG. 3(d). Thus, in this embodiment, when a region is divided into multiple regions by other regions, each reference region is designated for each of the multiple regions.

The first reference area 38 and the second reference area 40 are specified, for example, by the user writing the first reference area 38 and the second reference area 40 directly on the image 30 displayed on the display device 4 using a mouse or a touch pen. Note that in this embodiment, the first reference area 38 and the second reference area 40 are both specified as linear shapes, but this is not limited thereto, and they may be specified as areas within a specified shape, such as a rectangle or a circle.

As a result, in step S4, the pixels in the first reference region 38 are set as first reference pixels. Also, in step S4, the pixels in the second reference region 40 are set as second reference pixels. The first and second reference pixels are set by the reference pixel setting unit 20 analyzing the specified first reference region 38 and second reference region 40, and identifying which pixels in the image 30 each reference region contains.

Next, in this embodiment, the first reference pixel is set as the first pixel, and the second reference pixel is set as the second pixel (step S6). Step S6 is executed, for example, by the reference pixel setting unit 20 following the setting of the first reference pixel and the second reference pixel.

<Color coordinates and probability distribution>
Next, the probability distribution estimation unit 22 analyzes the color information of each of the first and second reference pixels, and identifies the coordinates in the color space of each reference pixel (step S8). An example of the color coordinates of each of the first and second reference pixels will be described with reference to Fig. 4. Fig. 4 is a diagram showing the color coordinates of each of the first and second reference pixels in the RGB color space. In the RGB color space shown in Fig. 4, first color coordinates 42 obtained from the color information of each of the first reference pixels and second color coordinates 44 obtained from the color information of each of the second reference pixels are plotted.

Next, the probability distribution estimation unit 22 estimates a first probability distribution and a second probability distribution in the color space from the color coordinates of the first reference pixel and the second reference pixel (step S10). Here, the first probability distribution and the second probability distribution are distributions that represent the probability that a pixel on the image 30 having a color corresponding to each coordinate on the color space is included in the color of the pixels included in the first region and the second region, respectively.

In other words, the first probability distribution is the probability that a pixel on image 30 is included in the first region in the coordinates in the color space, estimated based on the color information in the color space of the first reference pixel. The second probability distribution is the probability that a pixel on image 30 is included in the second region in the coordinates in the color space, estimated based on the color information in the color space of the second reference pixel.

The first probability distribution and the second probability distribution will be described with reference to FIG. 5, along with their estimation methods. FIG. 5 is a diagram showing a first probability distribution 46 and a second probability distribution 48 in RGB color space. Note that in FIG. 5, for convenience of illustration, the first probability distribution 46 and the second probability distribution 48 are expressed as distributions on a certain plane in the RGB color space, but in reality, they are expressed as three-dimensional distributions in the RGB color space.

In general, for example, in an image having both a foreground region 31 and a background region 36, the color of a pixel located in the foreground region 31 is highly likely to be similar to the color of the pixels located around that pixel. On the other hand, the colors of the pixels located in the foreground region 31 and the pixels located in the background region 36 are less likely to be similar to each other.

Generally, the distance between the color coordinates of two points on the RGB color space can indicate how similar the colors corresponding to those color coordinates are. Specifically, the closer the distance between the color coordinates of two points on the RGB color space, the more similar the colors corresponding to those color coordinates are to each other.

Therefore, for example, if many first color coordinates 42 are plotted around a certain color coordinate in the RGB color space, it can be said that there is a high probability that the color coordinates of the colors of the pixels included in the first region are located around the specific color coordinates. Therefore, the first probability distribution 46 is determined so that it is high around the coordinates where many first color coordinates 42 are plotted, and low around the coordinates where the number of plots of the first color coordinates 42 is low. For the same reason, the second probability distribution 48 is determined so that it is high around the coordinates where many second color coordinates 44 are plotted, and low around the coordinates where the number of plots of the second color coordinates 44 is low.

Specifically, as shown in FIG. 5, the first probability distribution 46 has, from the surrounding side toward the coordinate where at least one of the first probability distributions 46 has a high probability, a first low probability region 46A, a first medium probability region 46B, and a first high probability region 46C, in this order. Also, the second probability distribution 48 has, from the surrounding side toward the coordinate where at least one of the second probability distributions 48 has a high probability, a second low probability region 48A, a second medium probability region 48B, and a second high probability region 48C, in this order.

The first probability distribution 46 and the second probability distribution 48 are determined, for example, by a Gaussian Mixture Model (GMM) in the RGB color space. For example, the first probability distribution 46 and the second probability distribution 48 have a distribution in which multiple spatial Gaussian distributions in the RGB color space are superimposed. However, this embodiment is not limited to this, and the first probability distribution 46 and the second probability distribution 48 are not particularly limited as long as they are distributions that express the probability of the color coordinates of pixels in the first region and pixels in the second region.

Note that hereafter in this specification, when the term "probability distribution" is used simply, it refers to the distribution of the probability that a pixel in an image having a color corresponding to each coordinate in color space is included in the color of the pixels contained in a certain region.

<Adding pixels>
Next, an adjacent pixel analysis step is performed in which one of the pixels adjacent to the pixel already set as the pixel of each region is added to the pixel of one of the regions. The adjacent pixel analysis step will be described with further reference to Figures 6 and 7. Figures 6 and 7 are schematic diagrams of the step that show, in an enlarged scale, pixels around the already set first and second pixels among the pixels of the image 30.

As shown in each of Figures 6 and 7, image 30 has a structure in which a plurality of approximately rectangular pixels 50 are arranged along the row and column directions. Here, pixels 50 each include a first pixel 52 that is set as a pixel belonging to the first region, and a second pixel 54 that is set as a pixel belonging to the second region.

In the adjacent pixel analysis process, first, a first adjacent pixel 56, which is a pixel adjacent to the first pixel 52 in the row direction or column direction, is identified. In addition, a second adjacent pixel 58, which is a pixel adjacent to the second pixel 54 in the row direction or column direction, is identified (step S12). Depending on the positions of the first pixel 52 and the second pixel 54, a conflicting pixel 60 may be generated that is adjacent to both the first pixel 52 and the second pixel 54. The conflicting pixel 60 is treated as an adjacent pixel that is included in both the first adjacent pixel 56 and the second adjacent pixel 58. Next, the color information of each of the first adjacent pixel 56 and the second adjacent pixel 58 is analyzed, and the coordinates in the RGB color space are calculated (step S14).

Next, the pixel most likely to be included in the first pixel 52 is determined from the first adjacent pixels 56. Also, the pixel most likely to be included in the second pixel 54 is determined from the second adjacent pixels 58 (step S16). In this embodiment, an example will be described in which step S16 is achieved by calculating and comparing a cost C _N (s, t) defined in the following equation (1) for each adjacent pixel.

In formula (1), one of the pixels in each region is pixel s, and one of the pixels adjacent to pixel s is pixel t. Note that N represents the region N that contains pixel s.

D(s,t) is the Euclidean distance in the RGB color space between the color of pixel s and the color of pixel t, and represents the similarity between the colors of pixel s and pixel t. The smaller D(s,t) is, the more similar the colors of pixel s and pixel t are, and the higher the probability that pixel t is included in region N. C(s) is the value of the cost C _N (s,t) of pixel s when pixel s is set in each region. If pixel s is the reference pixel, C(s) is set to 0.

W ₁ (s, t) is a weight for D(s, t) and is defined by the following equation (2).

In formula (2), l _M (t) represents the probability that the color of pixel t is included in the colors of pixels included in region M. For example, l _M (t) is calculated by comparing the coordinates of the color of pixel t in the RGB color space with the probability distribution in region M. If the color of pixel t is highly likely to be included in the colors of region M, in other words, if the value of l _M (t) is high, pixel t is highly likely to be included in region M. Note that |W ₀ | in formula (2) is the absolute value of the minimum value of l _M (t) obtained by searching l _M (t) for all pixels t, and is a term for normalizing W ₁ (s, t) so that it does not become a negative value. From formulas (1) and (2), the higher the similarity between the colors of pixel s and pixel t, and the higher the probability that the color of pixel t is included in the colors of region N, the lower the cost C _N (s, t).

W ₂ (s, t) is a weight for C(s) and is defined by the following equation (3).

From equation (3), if the probability that the color of pixel t is included in the colors of region N is higher than the probability that the color of pixel t is included in the colors of regions other than region N, the value of _W2 (s,t) will be 0, and otherwise the value of _W2 (s,t) will be 1. In other words, _W2 (s,t) is a term provided in order to not take into account the cost _CN (s,t) when pixel s is set as a pixel in a certain region when there is a high probability that the color of pixel t is included in the colors of region N.

From the above, it can be said that the lower the calculated cost C _N (s, t) of each pixel t, the higher the probability that the pixel t is included in the region N including pixel s. Therefore, in step S16, for example, the cost C _N (s, t) is calculated for each of the first adjacent pixel 56 and the second adjacent pixel 58, and the pixel with the lowest cost C _{N (s, t) is identified. In this way, the first pixel candidate 62, which is the pixel with the lowest cost C N} (s, t) among the first adjacent pixel 56 and the second adjacent pixel 58, is identified. Here, for example, if there are multiple pixels with the lowest cost C _N (s, t), multiple pixel candidates may be identified, and for example, a second pixel candidate 64 different from the first pixel candidate 62 may be further identified.

For example, when there are multiple first pixels 52 adjacent to a certain first adjacent pixel 56, in step S16, each of the first pixels 52 may be treated as pixel s and multiple costs C _N (s, t) may be calculated. In this case, the average value of the multiple calculated costs C _N (s, t) may be regarded as the cost C _N (s, t) of the first adjacent pixel 56.

Next, the adjacent pixel is added to the pixels included in the region that is most likely to include the adjacent pixel, and the pixel is reset (step S18). Specifically, the first pixel candidate 62 is added to the pixel in the region that is adjacent to the first pixel candidate 62 and is most likely to include the first pixel candidate 62. For example, as shown in FIG. 7, when the first pixel candidate 62 is adjacent only to the first pixel 52, the first pixel candidate 62 is added to the first pixel 52. When the first pixel candidate 62 is a competitive pixel 60 adjacent to both the first pixel 52 and the second pixel 54, the first pixel candidate 62 is added to the pixel in the region that has a lower cost C _N (s, t). Accordingly, the first adjacent pixel 56 adjacent to the added first pixel 52 is added.

In other words, in the adjacent pixel analysis process in this embodiment, it is determined whether the first adjacent pixel 56 is a pixel included in the first region based on the color information of the first adjacent pixel 56 and the first probability distribution 46. Furthermore, in the adjacent pixel analysis process in this embodiment, if it is determined that the first adjacent pixel 56 is a pixel included in the first region, the first adjacent pixel 56 is added to the first pixel 52, and the first adjacent pixel 56 and the first pixel 52 are re-defined as a new first pixel 52.

In addition, in the adjacent pixel analysis process in this embodiment, it is determined whether the second adjacent pixel 58 is a pixel included in the second region based on the color information of the second adjacent pixel 58 and the second probability distribution 48. Furthermore, in the adjacent pixel analysis process in this embodiment, if it is determined that the second adjacent pixel 58 is a pixel included in the second region, the second adjacent pixel 58 is added to the second pixel 54, and the second adjacent pixel 58 and the second pixel 54 are re-defined as a new second pixel 54.

If the second pixel candidate 64 is identified in step S16, the second pixel candidate 64 may also be added in step S18 to a pixel adjacent to the second pixel candidate 64 and in a region that is most likely to include the second pixel candidate 64. For example, as shown in FIG. 7, if the second pixel candidate 64 is adjacent only to the second pixel 54, the second pixel candidate 64 is added to the second pixel 54.

As described above, the adjacent pixel analysis process is executed, one of the adjacent pixels is added to the pixels contained in each region, and the pixels contained in each region are reset. Note that steps S12 to S18 of the adjacent pixel analysis process are executed by the adjacent pixel analysis unit 24 by analyzing each pixel 50.

As described above, the neighboring pixels to be added to the pixels of each region are determined based on the probability indicating which of the regions the neighboring pixels are included in. Therefore, the neighboring pixel analysis process can automatically determine which pixel the conflicting pixel 60 should be added to. In other words, in the neighboring pixel analysis process, if the first neighboring pixel 56 and the second neighboring pixel 58 are the same pixel, it is determined whether the conflicting pixel 60 should be added to the first pixel 52 or the second pixel 54 based on the first probability distribution 46 and the second probability distribution 48. Therefore, in this embodiment, the probability that the conflicting pixel 60 is set to a pixel included in an inappropriate region is more effectively reduced.

<Determining when to end pixel addition and correcting the probability distribution>
Next, it is determined whether or not there is any pixel that has not been set to any of the regions among the pixels 50 (step S20). If it is determined in step S20 that there is a pixel that has not been set to any of the regions, it is then determined whether or not the number of pixels that have been set to the region to which they belong has increased by a certain amount or more since the probability distribution in each region was determined (step S22).

When step S22 is executed for the first time, step S22 counts the number of pixels each region contains that has increased since step S10 was executed. Step S22 counts the increase in the number of pixels each region contains, based on the number of pixels each region contains when step S10 was executed, until that number reaches a certain number or more.

If the increase in the number of pixels contained in each region is not equal to or greater than a certain level in step S22, step S12 is executed again. Therefore, each time steps S12 to S22 are repeated, the number of pixels contained in each region increases. If the increase in the number of pixels contained in each region is equal to or greater than a certain level in step S22, a probability distribution correction process is executed to correct the probability distribution in color space for each region (step S24).

In step S24, the probability distribution in each region is modified based on the color information of each pixel that each region currently contains. In other words, in step S24, the first probability distribution 46 is modified based on the color information in the RGB color space of the reset first pixel 52. In step S24, the second probability distribution 48 is modified based on the color information in the RGB color space of the reset second pixel 54. The modification of each probability distribution in step S24 is performed by the same method as the estimation of each probability distribution in step S10.

Once step S24 has been executed, in the next and subsequent steps S22, the increase in the number of pixels contained in each region is counted based on the number of pixels contained in each region when step S24 was executed the previous time, and a determination is made as to whether step S24 needs to be executed. In other words, step S24 is executed if a prescribed number or more of step S18 have been executed since the most recent execution of step S10 or step S24.

The correction of the probability distribution in each region in step S24 will be described in more detail with reference to FIG. 8. S24-A shown in FIG. 8(a) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 before step S24 is executed for the first time. S24-B shown in FIG. 8(b) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 after the correction of the probability distribution in step S24 is executed once. S24-C shown in FIG. 8(c) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 after the correction of the probability distribution in step S24 is executed twice. S24-D shown in FIG. 8(d) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 when the first probability distribution 46 and the second probability distribution 48 are estimated from all pixels of the image 30.

In step S10, a first probability distribution 46 and a second probability distribution 48 are estimated from the respective color information of a first reference pixel that is part of the first region and a second reference pixel that is part of the second region. Therefore, there is a high probability that the first probability distribution 46 and the second probability distribution 48 estimated in step S10 differ from the ideal first probability distribution 46 and the second probability distribution 48 estimated from all pixels of the image 30.

The estimation of the probability distribution in step S24 uses an increased number of reference pixels compared to the estimation of the probability distribution in step S10. Therefore, the probability distribution corrected in step S24 is more likely to approximate the ideal probability distribution estimated from all pixels of image 30. Furthermore, the more times step S24 is performed, in other words, the more the probability distribution is corrected, the more likely the probability distribution will approximate the ideal probability distribution.

Therefore, each time the number of set pixels increases by a certain amount or more, step S24 is executed to modify the probability distribution, so that the analysis of each adjacent pixel is executed based on a probability distribution that is closer to an ideal probability distribution. Therefore, by executing the probability distribution modification process of step S24, the probability that adjacent pixels are added to pixels in an appropriate area is further improved. Note that step S24 is executed by the probability distribution modification unit 28, but is not limited to this, and may be executed by the probability distribution estimation unit 22.

In this embodiment, step S24 is performed after a specified number of pixels or more have been set since the previous estimation or modification of the probability distribution. If the difference in the number of color coordinates used to estimate the probability distribution is small, the fluctuation in the probability distribution is small even if the probability distribution is modified. For this reason, it is more effective to modify the probability distribution when the number of reference pixels has increased to a certain extent. Therefore, in this embodiment, the speed of image extraction can be improved while more effectively modifying the probability distribution than by performing step S24 each time the number of set pixels increases.

<Image extraction>
Steps S12 to S24 are repeated, and when all pixels 50 in the image 30 have been set as pixels in any of the regions, the process proceeds from step S20 to an extraction step (step S26).

In step S26, an image including the pixels contained in each region is extracted as an extracted image. Specifically, in step S26, the first pixel 52 is extracted as the first extracted image, and the second pixel 54 is extracted as the second extracted image. The extraction process is performed by the extracted image generating unit 26 identifying the first pixel 52 and the second pixel 54, and extracting the images including these pixels.

As a result of the above, ideally, a first extracted image including pixels in the foreground region 31 and a second extracted image including pixels in the background region 36 are generated. Therefore, in this embodiment, an image in which the foreground region 31 is extracted and an image in which the background region 36 is extracted are obtained from the image 30.

In the image extraction method according to this embodiment, the region in which each pixel is included is determined based on the probability distribution in each region. Therefore, the image extraction method according to this embodiment reduces the probability that each pixel is set as a pixel in an inappropriate region, improving the accuracy of extracting the pixels included in each region from an image. This image extraction method can be realized using the image extraction device 2 according to this embodiment.

In addition, in this embodiment, all pixels 50 in image 30 are set to either the first pixel 52 or the second pixel 54. Therefore, it is possible to more reliably set which region each pixel belongs to than by setting the pixels contained in one region and then generating an extracted image. Therefore, in this embodiment, the accuracy of generating the first extracted image and the second extracted image is further improved.

<Evaluation of Examples and Comparative Examples>
The effect of the image extraction method according to the present embodiment is evaluated by comparing an example carried out according to the present embodiment with a comparative example carried out according to the image extraction method according to the comparative embodiment. The process of generating an extracted image from a specific image in the example will be described with reference to FIG.

In the example, an extracted image was generated from image 66 shown in FIG. 9(a) based on the image extraction method according to this embodiment. Image 66 is an image including an object region 68 and a background region 70. In the example, in step S4, a first reference region 38 and a second reference region 40 were specified as shown in FIG. 9(a).

Then, based on the image extraction method according to this embodiment, an attempt was made to generate from image 66 a first extracted image including object region 68 as foreground region 31, and a second extracted image including background region 70. In the example, step S24 was executed when 40% and 80% of all pixels in image 66 were added to either region.

The first extracted image 72A shown in FIG. 9 (b-1) is an extracted image including the first pixel 52 immediately before the first step S24, in other words, at the time when 40% of all pixels in the image 66 are set to the first pixel 52 or the second pixel 54. The second extracted image 74A shown in FIG. 9 (c-1) is an extracted image including the second pixel 54 immediately before the first step S24. The first extracted image 72B shown in FIG. 9 (b-2) is an extracted image including the first pixel 52 immediately before the second step S24, in other words, at the time when 80% of all pixels in the image 66 are set to the first pixel 52 or the second pixel 54. The second extracted image 74B shown in FIG. 9 (c-2) is an extracted image including the second pixel 54 immediately before the second step S24. The first extracted image 72 shown in (b-3) of FIG. 9 is an extracted image that includes the first pixel 52 and was generated in step S26. The second extracted image 74 shown in (c-3) of FIG. 9 is an extracted image that includes the second pixel 54 and was generated in step S26.

The first extracted image 72A, the second extracted image 74A, the first extracted image 72B, and the second extracted image 74B shown in FIG. 9 show how the first pixel 52 and the second pixel 54 are gradually added in sequence in this embodiment. Also, in this embodiment, the first extracted image 72 including the first pixel 76 at a position corresponding to the foreground region 31 and the second extracted image 74 including the second pixel 78 at a position corresponding to the background region 70 are obtained.

Next, an image extraction method according to a comparative embodiment will be described. The image extraction method according to the comparative embodiment differs from the image extraction method according to the present embodiment in that it includes, instead of step S10, a process of generating color buckets in a color space for each region. This process will be described with reference to FIG. 10.

S10-C in FIG. 10 is a schematic diagram showing an example of color buckets in an RGB color space generated in a comparative embodiment. In the comparative embodiment, after identifying the color coordinates in the color space of each reference pixel, color buckets are generated in each region from the information on the color coordinates. In the comparative embodiment, the R axis, G axis, and B axis in the RGB color space are divided into multiple grids, the RGB color space is divided into multiple sections, and color buckets are generated in some of the sections. The color buckets in each region are generated in the section where the color coordinates of each reference pixel are located most frequently.

In the comparative embodiment, for example, as shown in FIG. 10, a first color bucket 46D, which is a color bucket in the first region, is generated in a section of the RGB color space that contains many first color coordinates 42. Also, in the comparative embodiment, for example, as shown in FIG. 10, a second color bucket 48D, which is a color bucket in the second region, is generated in a section of the RGB color space that contains many second color coordinates 44. Here, in a section that has both the first color coordinates 42 and the second color coordinates 44, a competing color bucket CB that belongs to both the first color bucket 46D and the second color bucket 48D may be generated.

In the comparative example, a cost C _N (s, t) is calculated for each adjacent pixel, and the pixel with the lowest cost C _N (s, t) is added to one of the pixels included in each region, with the exception that W ₁ (s, t) and W ₂ (s, t) of the cost C _N (s, t) are changed to the following equations (4) and (5), respectively.

In formula (4) and formula (5), q(t) represents a color with a reduced quantum number of pixel t, and corresponds to the color coordinate of pixel t. _KM represents a section in the RGB color space in which the color bucket in region M is generated. Therefore, from formula (4), W ₁ (s, t) according to the comparative embodiment is 1 when the color coordinate of pixel t is included only in the color bucket of region N, 4 when the color coordinate of pixel t is included only in the color bucket of a region other than region N, and 2 otherwise. Also, from formula (5), W ₂ (s, t) according to the comparative embodiment is 0 when the color coordinate of pixel t is included in the color bucket of region N, and 1 otherwise.

Furthermore, in the comparative embodiment, steps S22 and S24 are not executed, and if there is a pixel in step S20 that is not set as one of the pixels contained in any of the regions, step S12 is executed again.

Except for the points mentioned above, the image extraction method according to the comparative embodiment is performed in the same manner as the image extraction method according to the present embodiment.

In the comparative example, each extracted image was generated from image 66 by specifying the same reference area as first reference area 38 and second reference area 40 shown in FIG. 9, using the image extraction method according to the comparative embodiment. The extracted images generated in the comparative example and the extracted images generated in the example are compared with reference to FIG. 11.

Figure 11 (a) is a diagram showing a first extracted image 72C generated in the comparative example. Figure 11 (b) is a diagram showing a first extracted image 72 generated in the example. Note that in Figures 11 (a) and (b), in order to more clearly compare the accuracy of image extraction, second pixels 78C and 78, which correspond to the background region 70, are overlaid on the first extracted image 72C and the first extracted image 72, respectively, with reduced density.

The first extracted image 72C generated in the comparative example has a first pixel 76C that corresponds to the object region 68. However, the first extracted image 72C has an insufficient region E in which some of the pixels located in the object region 68 are not set as the first pixel 76C. It is presumed that the insufficient region E has arisen because the colors of the pixels around the insufficient region E are included in the competing color bucket CB, causing some of the pixels located in the object region 68 to be erroneously set as the second pixel 78C.

On the other hand, the first extracted image 72 generated in the example does not have an insufficient area E, and the setting of the first pixel 76 is generally successful. Thus, compared to the image extraction method of the comparative embodiment, the image extraction method of this embodiment can generate an extracted image with greater accuracy.

[Embodiment 2]
<Generation of Third Extracted Image>
Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be given to components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

In this embodiment, too, a method for generating an extracted image from image 30 will be described as an example. However, this embodiment differs from the previous embodiment in that in addition to the first extracted image and the second extracted image, a third extracted image is also generated.

In the image extraction method according to this embodiment, for example, first, the image 30 shown in FIG. 12 is displayed on the display device 4 by the same method as step S2 in the previous embodiment. Note that the image 30 shown in FIG. 12 (a) is the same as the image 30 shown in FIG. 3 (c). However, in this embodiment, the first region has only the first object region 32, and the second object region 34 is included in the third region. Furthermore, in this embodiment, a method of generating a first extracted image including the first object region 32 as the first region, a second extracted image including the background region 36 as the second region, and a third extracted image including the second object region 34 as the third region from the image 30 will be described as an example. In other words, the image extraction method according to this embodiment is a method of further extracting a third extracted region including the third region from an image having the third region, and generating a third extracted image including the third extracted region.

In this embodiment, in step S4 in the previous embodiment, in addition to the designation of the first and second reference pixels, the designation of a third reference pixel is accepted. The designation of the third reference pixel is performed by designating a third reference region 80 in the second object region 34, which is the third region, and designating a pixel contained in the third reference region 80 as the third reference pixel, as shown in FIG. 12(b). The designation of the third reference pixel is performed using the same method as the designation of the first and second reference pixels.

In the present embodiment, the steps are sequentially performed by modifying each step of the previous embodiment. In step S6 of the present embodiment, the third reference pixel is further set to the third pixel. In steps S8 and S10 of the present embodiment, a third probability distribution is further estimated, which expresses the probability that a pixel on the image 30 having a color corresponding to each coordinate on the color space is included in the color of the pixel included in the third region. In steps S12 to S18 of the present embodiment, the cost C _N (s, t) is further calculated for a third adjacent pixel adjacent to the third pixel, and each pixel is set. In step S24 of the present embodiment, the third probability distribution is further corrected. In step S26 of the present embodiment, a third extracted image including the third pixel is further generated. Each of steps S6 to S26 of the present embodiment is performed by the same method as each of steps S6 to S26 of the previous embodiment, except for the above-mentioned points. The image extraction method of the present embodiment is performed by the same device as the image extraction device 2 of the previous embodiment.

In this embodiment, in addition to the first extracted image and the second extracted image, a third extracted image including a third pixel can be generated. Therefore, according to the image extraction method according to this embodiment, for example, from an image having multiple object regions in a background region, multiple extracted images having different object regions and an extracted image having a background region can be obtained.

In this embodiment, the area in which each pixel is included is determined based on the probability distribution in each area. Therefore, according to the image extraction method of this embodiment, even if the number of types of areas to be extracted increases, the probability that each pixel will be set as a pixel of an inappropriate area decreases, improving the accuracy of extracting the pixels contained in each area from the image.

In this embodiment, the image 30 may further have regions different from any of the first to third regions, including, for example, a fourth region, a fifth region, etc. In this case, in this embodiment, the image may be extracted using four or more types of reference pixels, including not only the first reference pixel, the second reference pixel, and the third reference pixel, but also the fourth reference pixel, the fifth reference pixel, etc. Even in this case, the image extraction method according to this embodiment can further generate an extracted image including pixels of the relevant region.

In this embodiment, the first region and the third region include the first object region 32 and the second object region 34 of the image 30, respectively. Therefore, in the image extraction method according to this embodiment, it is possible to generate a plurality of extracted images in which a plurality of object regions are individually extracted from the image. However, this is not limited to this, and in this embodiment, the third region may include a portion of the background region 36. In this case, in the image extraction method according to this embodiment, it is possible to generate a plurality of extracted images in which a portion of the background region is each extracted from the image.

[Embodiment 3]
<Generation of First Extracted Image Only>
Another embodiment of the present disclosure will be described below. In this embodiment, too, an example of a method for generating an extracted image from an image 30 will be described. However, this embodiment differs from the above-described embodiments in that only a first extracted image is generated.

In the image extraction method according to this embodiment, for example, first, the image 30 shown in FIG. 13(a) is displayed on the display device 4 by the same method as step S2 in each of the above-mentioned embodiments. Note that the image 30 shown in FIG. 13(a) is the same as the image 30 shown in FIG. 3(a). However, in this embodiment, a method of generating only a first extracted image including a first object region 32 as a foreground region 31, which is the first region, from the image 30 will be described as an example.

In this embodiment, only the designation of the first reference pixel is accepted in step S4 in the previous embodiment. Therefore, as shown in FIG. 13(b), in this embodiment, only the first reference region 38 is designated for the image 30.

From this point on, in this embodiment, steps that are partially modified from the steps in the previous embodiment are executed in sequence. In step S6 in this embodiment, only the first reference pixel is set to the first pixel 52. In steps S8 and S10 in this embodiment, only the first probability distribution 46 is estimated. In steps S12 to S18 in this embodiment, only the first adjacent pixel 56 is identified, a cost C _N (s, t) is calculated for the first adjacent pixel 56, and only the first pixel 52 is reset. In step S24 in this embodiment, only the first probability distribution 46 is modified. In step S26 in this embodiment, only the first extracted image including the first pixel 52 is generated.

Here, in the image extraction method according to this embodiment, in order to generate only the first extracted image including the first region, not all pixels 50 of the image 30 are added to the first pixel 52, but only some of the pixels 50 are added to the first pixel 52. For this reason, it is considered that step S20 according to this embodiment can be changed to, for example, the following process.

For example, in step S20 according to the present embodiment, it is determined whether the cost C _N (s, t) of each first adjacent pixel 56 is equal to or greater than a certain specified value. In this case, if it is determined in step S20 that the costs C _N (s, t) of all the first adjacent pixels 56 are equal to or greater than the specified value, the process proceeds to execution of step S26. Step S20 is useful in cases where it is possible to determine that all the first pixels 52 have been identified if the color difference between the first and second regions of the image 30 is clear and the costs C _N (s, t) of all the first adjacent pixels 56 are equal to or greater than the specified value.

Alternatively, for example, in step S20 according to this embodiment, it is determined whether the total number of first pixels 52 is equal to or greater than a certain specified value. In this case, if it is determined in step S20 that the total number of first pixels 52 is equal to or greater than the specified value, the process proceeds to execution of step S26. Step S20 is useful when the ratio of the area of the first region to the total area of the image 30 is known, at least approximately.

By the above step S20, in the image extraction method according to this embodiment, only a portion of the pixels 50 of the image 30 can be added to the first pixels 52. This reduces the number of pixels 50 for which the cost C _N (s, t) should be calculated, and therefore the image extraction method according to this embodiment can shorten the time required to generate the first extracted image.

Each of steps S6 to S26 according to this embodiment is executed by the same method as each of steps S6 to S26 according to the previous embodiment, except for the points described above. The image extraction method according to this embodiment is executed by the same device as the image extraction device 2 according to the previous embodiment.

In this embodiment, a first extracted image including at least the first pixel 52 can be generated by simply specifying the first reference pixel. Therefore, according to the image extraction method of this embodiment, the user does not need to specify the second reference pixel, and the first extracted image can be generated more easily.

Also in this embodiment, whether each pixel is included in the first pixel 52 is determined based on the first probability distribution 46 in the first region. Therefore, according to the image extraction method of this embodiment, the probability that a pixel 50 that should be added to the first pixel 52 is not added to the first pixel 52 is reduced. In addition, according to the image extraction method of this embodiment, the probability that a pixel 50 that should not be added to the first pixel 52 is added to the first pixel 52 is reduced. Therefore, according to the image extraction method of this embodiment, a first extracted image including the first pixel 52 can be generated from the image 30 more simply and accurately.

In this embodiment, the first region includes the first object region 32. Therefore, in the image extraction method according to this embodiment, an extracted image can be generated in which only the object region is extracted from the image. However, the present disclosure is not limited to this, and in this embodiment, the first region may not include the first object region 32, but may include at least a portion of the background region 36. In this case, in the image extraction method according to this embodiment, an extracted image can be generated in which only at least a portion of the background region is extracted from the image. Furthermore, the first region may include the first object region 32 and the second object region 34.

<Example of software implementation>
The functions of the image extraction device (hereinafter, referred to as the "device") according to each of the above-described embodiments can be realized by an image extraction program for causing a computer to function as the device. In this case, the functions of the device can be realized by a program for causing a computer to function as each unit included in the control unit 6, for example.

In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing each of the functions described in each of the above embodiments.

The program may be recorded on one or more computer-readable recording media, not on a temporary basis. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, the scope of this disclosure also includes an integrated circuit in which a logic circuit that functions as each of the control blocks is formed. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

The processes described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI may run on the control device, or on another device (such as an edge computer or a cloud server).

This disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure.

2 Image extraction device 6 Control unit 20 Reference pixel setting unit 22 Probability distribution estimation unit 24 Adjacent pixel analysis unit 26 Extracted image generation unit 28 Probability distribution correction unit

Claims (11)

An image extraction method for extracting a specific region from an image, comprising the steps of:
a receiving step of receiving a designation of a first reference pixel which is a pixel that is part of a first region included in the image;
a reference pixel setting step of setting the first reference pixel as a first pixel;
a probability distribution estimation step of estimating a first probability distribution, which is a probability that a pixel on the image is included in the first region at a coordinate in the color space, based on color information in the color space of the first reference pixel;
an adjacent pixel analysis step of determining whether the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and adding the first adjacent pixel to the first pixel when it is determined that the first adjacent pixel is a pixel included in the first region, and resetting the first adjacent pixel and the first pixel as a new first pixel;
an extraction step of extracting the reset first pixel as a first extracted image ;
and a probability distribution modifying step of modifying the first probability distribution based on the reset color information of the first pixel in the color space . the image has a second region different from the first region;
The receiving step further includes receiving a designation of a second reference pixel which is a pixel that is a part of the second region;
The reference pixel setting step further includes setting the second reference pixel as a second pixel;
the probability distribution estimation step further comprises estimating a second probability distribution, which is a probability that a pixel on the image is included in the second region at a coordinate in the color space, based on color information in the color space of the second reference pixel;
the adjacent pixel analyzing step further includes judging whether the second adjacent pixel is a pixel included in the second region based on color information of a second adjacent pixel adjacent to the second pixel and the second probability distribution, and when it is determined that the second adjacent pixel is a pixel included in the second region, adding the second adjacent pixel to the second pixel, and resetting the second adjacent pixel and the second pixel as a new second pixel;
2. The image extraction method according to claim 1, wherein the extraction step further comprises extracting the reset second pixel as a second extracted region. The image extraction method according to claim 2, wherein the adjacent pixel analysis step is repeatedly performed until all pixels in the image are set to either the first pixel or the second pixel. The image extraction method according to claim 2 or 3, wherein in the adjacent pixel analysis step, if the first adjacent pixel and the second adjacent pixel are the same pixel, it is determined whether to add the pixel to the first pixel or the second pixel based on the first probability distribution and the second probability distribution. the image has a third region that is different from both the first region and the second region;
The receiving step further includes receiving a designation of a third reference pixel which is a pixel that is a part of the third region;
The reference pixel setting step further includes setting the third reference pixel as a third pixel;
the probability distribution estimation step further comprises estimating a third probability distribution, which is a probability that a pixel on the image is included in the third region at a coordinate in the color space, based on color information in the color space of the third reference pixel;
the neighboring pixel analyzing step further comprises determining whether the third neighboring pixel is a pixel included in the third region based on color information of a third neighboring pixel adjacent to the third pixel and the third probability distribution, and when it is determined that the third neighboring pixel is a pixel included in the third region, adding the third neighboring pixel to the third pixel, and resetting the third neighboring pixel and the third pixel as a new third pixel;
The image extraction method according to claim 2 , wherein the extraction step further comprises extracting the reset third pixel as a third extraction region. The image extraction method according to any one of claims 2 to 5, further comprising a probability distribution correction step of correcting the second probability distribution based on the color information in the color space of the reset second pixel. The image extraction method according to claim 5, further comprising a probability distribution correction step of correcting the third probability distribution based on the color information in the color space of the reset third pixel. The image extraction method according to claim 1 , wherein the probability distribution correction step is executed when the adjacent pixel analysis step has been executed a specified number of times or more since the most recent execution of the probability distribution estimation step or the probability distribution correction step. a reference pixel setting unit that receives a designation of a first reference pixel that is a pixel of a part of a first region included in an image, and sets the first reference pixel as a first pixel;
a probability distribution estimation unit that estimates a first probability distribution, which is a probability that a pixel on the image is included in the first region at a coordinate in the color space, based on color information on the color space of the first reference pixel;
an adjacent pixel analysis unit that determines whether or not the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and adds the first adjacent pixel to the first pixel when it is determined that the first adjacent pixel is a pixel included in the first region, and resets the first adjacent pixel and the first pixel as a new first pixel;
an extraction image generating unit that extracts the reset first pixel as a first extraction region ;
a probability distribution correction unit that corrects the first probability distribution based on the reset color information of the first pixel in the color space . 10. An image extraction program for causing a computer to function as the image extraction device described in claim 9 , the image extraction program for causing the computer to function as the reference pixel setting unit, the probability distribution estimation unit, the adjacent pixel analysis unit, the extracted image generation unit, and the probability distribution correction unit . A computer-readable recording medium having the image extraction program according to claim 10 recorded thereon.

Images