JP7254440B2 - Image processing device, imaging device, image processing method, and program - Google Patents

Description

The present invention relates to an imaging apparatus that determines leaf color based on color information of a captured image.

BACKGROUND ART Conventionally, a leaf color determination technique is known as one of techniques for determining the growth state of a plant such as paddy rice. Leaf color determination is a technique for determining the numerical value of the leaf color (leaf color value) of a plant using a color reference plate called a leaf color plate. Japanese Patent Application Laid-Open No. 2004-200001 discloses a system for obtaining a leaf color value using representative values such as average values of RGB values and gray values of pixels in a predetermined range of captured image data.

JP 2017-12138 A

However, in the system disclosed in Patent Literature 1, the leaf color value is obtained using a representative value such as an average value, so the accuracy and resolution of leaf color determination are not sufficient. Further, in this system, if the determination index of the leaf color value is subdivided in order to improve the resolution of the leaf color determination, the precision of the leaf color determination will be lowered, and the calculation load will be applied and the system will be complicated.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing apparatus, an imaging apparatus, an image processing method, and a program capable of easily realizing high-precision and high-resolution leaf color determination.

An image processing apparatus as one aspect of the present invention determines a plurality of first leaf color values for a plurality of pixel regions based on a plurality of pieces of color information for each of a plurality of pixel regions in an image of a plant . 1 determination means; and a second determination means for determining a second leaf color value for a determination target area including the plurality of pixel areas in the image based on the plurality of first leaf color values.

An imaging apparatus as another aspect of the present invention includes an imaging element that photoelectrically converts an optical image formed by an imaging optical system and outputs image data, and the image processing apparatus.

According to another aspect of the present invention, there is provided an image processing method, wherein a first determination means determines a plurality of first color information related to the plurality of pixel regions in the image of the plant , based on the plurality of color information of each of the plurality of pixel regions. determining a leaf color value; and based on the plurality of first leaf color values, determining a second leaf color value for a determination target region including the plurality of pixel regions in the image.

A program as another aspect of the present invention causes a computer to execute the image processing method.

Other objects and features of the invention are described in the following embodiments.

Advantageous Effects of Invention According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and a program capable of easily realizing high-precision and high-resolution leaf color determination.

1 is a block diagram of an image processing device according to a first embodiment; FIG. 4 is a flowchart of an image processing method according to the first embodiment; FIG. 4 is an explanatory diagram of an image processing method according to the first embodiment; FIG. 2 is a configuration diagram of an image processing system according to a second embodiment; FIG. FIG. 11 is a block diagram of an imaging device according to a third embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
First, an image processing apparatus and an image processing method according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 3. FIG. FIG. 1 is a block diagram of an image processing device 10. As shown in FIG. FIG. 2 is a flow chart of an image processing method. FIG. 3 is an explanatory diagram of the image processing method.

As shown in FIG. 1, the image processing unit 104 has an acquisition unit 104a, a first determination unit 104b, and a second determination unit 104c. ). Acquisition means 104 a acquires an image (captured image) 20 . Based on the color information of each of the plurality of first regions (pixels 22) included in the image 20, the first determination unit 104b determines a plurality of leaf color values (first color value, first Leaf color value) 112 is determined. Based on the plurality of leaf color values 112, the second determination means determines a leaf color value (second color value, second leaf color value) 130 for a second region (determination target region 21) including a plurality of first regions (pixels 22). judge.

First, in step S1 of FIG. 2, the image processing apparatus 10 sets a determination target area (second area) 21 of the image in accordance with a user's operation. Note that the image processing apparatus 10 may automatically set the determination target area 21 of the image according to a predetermined algorithm. In the present embodiment, the determination target area 21 is an area including paddy rice in the image 20 as shown in FIG. Therefore, the determination target region 21 includes a plurality of pixels 22 having color information of paddy rice.

Subsequently, in step S<b>2 , the acquisition unit 11 of the image processing device 10 acquires color information of each pixel 22 . In the present embodiment, the color information is information (luminance value 111) on red (R), green (G), and blue (B) of each pixel 22, but is not limited to this, and may be any other color. Information may be used.

Subsequently, in step S<b>3 , the first determination means 12 of the image processing device 10 determines the leaf color value (first leaf color value) 112 of the pixel 22 . In this embodiment, the first determination means 12 determines the leaf color value 112 for each of all the pixels 22 included in the determination target region 21 set in step S1. Note that in the present embodiment, the first determination unit 12 acquires color information of each of the plurality of pixels 22 as the plurality of first regions (that is, each of the plurality of first regions is one of the pixels included in the second region). (corresponding to one pixel 22), but is not limited to this. The plurality of first regions may be pixel regions including a plurality of pixels included in the second region (that is, the first regions are regions including a plurality of pixels). In this case, the first determination means 12 can determine the plurality of leaf color values 112 for each of the plurality of pixel regions based on the average value of the color information of each of the plurality of pixels.

Subsequently, in step S<b>4 , the second determination unit 13 of the image processing device 10 determines the leaf color value (second leaf color value) 130 of the determination target area (second area) 21 . The leaf color value 130 is the leaf color value of the determination target region 21 finally output from the image processing device 10 . At this time, the second determination means 13 determines the plurality of first leaf color values (leaf color value 112) and the number of first regions corresponding to each of the plurality of first leaf color values (the number of pixels 22 included in the determination target region 21). ) and a histogram (frequency calculation result) 120 is created. In the histogram 120, the horizontal axis represents the first leaf color value (the range of leaf color values determined by the first determination means 12; in this embodiment, leaf color values 1 to 9 are set), and the vertical axis represents the frequency (each leaf color number of pixels 22 indicating a value).

Then, the second determination means 13 determines the second leaf color value (finally output leaf color value 130) based on the histogram 120. FIG. In this embodiment, the second determination means 13 determines the centroid value of the histogram 120 as the second leaf color value. The centroid value corresponds to a value obtained by adding the multiplied values of each leaf color value and the number of pixels indicating that leaf color value for all leaf color values and dividing the added value by the total number of pixels. However, the present embodiment is not limited to this, and the second determination means 13 may set a value determined based on the histogram 120, such as the average value or mode value of the histogram 120, as the second leaf color value. .

In this embodiment, the first determination means 12 can derive the leaf color value (first leaf color value) 112 using the color information of each pixel 22 of the image 20 acquired by the imaging device. The first determination means 12 can determine the leaf color value 112 using principal component analysis, calibration curve analysis, neural network, machine learning, or the like. In this embodiment, it is particularly preferable to use machine learning. Machine learning involves labeling obtained information in association with correct information and applying it to machine learning tools. In this method, the acquired information is linked to the correct information by automatically forming filter information regarding how the machine combines the acquired information to lead to the correct answer. The filter information can be learned in advance and held in the image processing device 10 . With this method, it is possible to improve accuracy by preparing a sufficient number of combinations of obtained information and correct information. Therefore, it is possible to robustly evaluate the leaf color value 112 even if the luminance value 111 changes due to, for example, changes in environmental conditions or lighting conditions. In this manner, the first determination means 12 can determine the leaf color value 112 using machine learning based on previously learned filter information.

Preferably, the filter information used by the first determination means 12 includes, in addition to the leaf color value 112, a noise filter for determining noise other than leaf colors such as sky and soil. As a result, the second determination means 13 can determine the leaf color value 130 only for leaf color values excluding sky, soil, and the like. For example, when there is a high-density photograph within the angle of view, soil, water, sky, or the like may be partially captured. If such an object is determined as a "leaf color value", there is a possibility that an error will occur in the subsequent determination result by the second determination means 13. Therefore, by providing a noise filter and excluding color information related to an object that is clearly an object other than paddy rice from the determination target, it is possible to obtain a more accurate determination result.

As described above, in this embodiment, in addition to the first determination means 12 that determines the leaf color of the pixel 22, the second determination means 13 that determines the leaf color value 130 using the histogram 120 of the determination target area 21 of the image 20 is provided. have. A final leaf color value 130 is calculated using not only the leaf color value of the pixel 22 in the image 20 but also the histogram 120 of the determination target region 21 . By setting the determination target region 21 that is at least a part of the image 20, it is possible to obtain the leaf color value 130 in the community. Leaf color values are also generally subjectively evaluated by the human eye using leaf color plates. The strict leaf color value in the determination target region 21 is not the leaf color value 112 determined based on the luminance value 111, but the determination result of the determination target region 21 by the second determination means 13 is a more accurate value.

In this embodiment, the second determination means 13 creates a histogram 120 based on the determination result of the first determination means 12 and finally determines the leaf color value 130 . By determining the final leaf color value (second leaf color value) 130 after determining the provisional leaf color value (first leaf color value) 112, the leaf color value of the object in the determination target region 21 is evaluated with higher accuracy. be able to. Contrary to the order of this embodiment, the first determination means 12 may perform leaf color determination after creating the histogram 120 for the luminance value 111 . In this case, since the number of calculations is reduced and the load is lightened, it is suitable for high-speed processing. Either order may be used in the present embodiment, but the former is suitable for determining leaf color values with higher accuracy.

The image 20 has characteristics derived from the imaging device that the luminance value changes due to the change in transmittance of the imaging lens (imaging device) and vignetting with respect to the angle of view. Therefore, in this embodiment, it is preferable to specify the location of the pixel 22 by specifying the determination target region 21 and correct the value of the luminance value 111 or the value of the leaf color value 112 . Especially for vignetting, the amount of change is determined according to the imaging lens, and the degree of change is determined according to the location of the determination target area 21 of the image 20 . In addition, the leaf color value 112 targeted in this embodiment varies depending not only on the spectral distribution but also on the absolute value of the amount of light. Therefore, since changes in the amount of light directly affect deviations in the leaf color values, it is possible to determine the leaf color values 112 with higher accuracy by performing correction in accordance with the designation of the determination target region 21 .

Determining the leaf color value 130 with respect to the determination target area 21 is effective in improving the resolution of the leaf color value. Here, a case where it is desired to evaluate the leaf color value below the decimal point is taken as an example. When trying to determine the leaf color value with high resolution using the first determination means 12, it is necessary to have a criterion for determining the leaf color value 112 from the luminance value 111 with high accuracy. Under such conditions, it is necessary to prepare sufficient learning data corresponding to the machine learning described above, or it is necessary to make highly accurate corrections to changes according to the weather and environment. On the other hand, when the second determination means 13 is used, the first determination means 12 is set to a level for roughly recognizing the leaf color, and the second determination means 13 is used to evaluate the leaf color value 130 according to the frequency of the determination target area 21. can be calculated with high resolution. For the leaf color value 130 determined in this way, it is not necessary to refine the determination criteria of the first determination means 12 described above. Moreover, since the second determination means 13 is used, there is an advantage that the final output is not much different from the tool in which the determination criteria of the first determination means 12 are finely set. This is because the second determination means 13 finally performs processing similar to averaging processing, so the fineness of information before averaging does not have much effect.

Further, in this embodiment, when the first determination means 12 uses machine learning, the first determination means 12 can determine (output) the leaf color value and its probability. In machine learning, it is common to output the "likelihood" of an answer before determining the final output value. For example, with respect to the brightness value 111, the first determination means 12 sets the probability of the leaf color value 3 to 25%, the probability of the leaf color value 4 to 40%, and the probability of the leaf color value 5 to 20%. to output As the final determination result of the first determination means 12, it is common to output the leaf color value 4 indicating the maximum probability, but the leaf color value 4 may be output including the probability information of 40%. . In this case, the first determination unit 12 determines the leaf color value of each pixel 22 and the probability of each pixel 22 in the determination target region 21 and outputs the probability to the second determination unit 13 . Therefore, the second determination unit 13 can calculate the determination result taking into consideration not only the frequency within the determination target region 21 but also the frequency within the pixel 22 . As a result, the second determination means 13 can perform leaf color determination including not only the leaf color distribution in the determination target region 21 but also the probability of the leaf color of the pixel 22, and can determine the leaf color value 130 with higher accuracy. becomes possible.

In the present embodiment, the leaf color value 112 is acquired using an imaging device that captures visible light, but the present invention is not limited to this, and the leaf color value 112 is acquired using invisible light such as ultraviolet light or infrared light. It is also possible to obtain For example, it is known that there is a high correlation between a Normalized Difference Vegetation Index (NDVI) using near-infrared light and a leaf color value. Therefore, the leaf color value can be determined using the NDVI acquired from the imaging device.

(Second embodiment)
Next, an image processing system (image processing apparatus) according to a second embodiment of the present invention will be described with reference to FIG. FIG. 4 is a configuration diagram of an image processing system 80 in this embodiment. The image processing system 80 includes an image processing device 81 (hardware as an information processing device), a display device 82 (monitor), and an input device 83 (input means such as a keyboard). This embodiment will explain a case where the image processing method of the first embodiment is operated on an image processing system 80 (software of a personal computer). The image processing device 81 of this embodiment has the functions of the acquisition unit 11, the first determination unit 12, and the second determination unit 13 of the image processing apparatus 10 described with reference to FIG.

First, in order to operate the image processing method of the first embodiment in the image processing apparatus 81, software (image processing program) for executing the image processing method is installed in the image processing apparatus 81 (personal computer). storage unit (not shown). In addition, data related to machine learning and neural networks that have been learned in advance are stored in a predetermined folder in the storage unit provided in the image processing device 81 . Data related to machine learning and neural networks can be installed from media 84 (storage media) such as CD-ROM and DVD-ROM. Alternatively, these data and software that operates using the data may be downloaded via a network (Internet) and installed.

Next, the image processing device 81 activates the installed software and performs image processing on the captured image. The image processing device 81 automatically acquires information about various shooting conditions (shooting conditions) from Exif information or the like, and performs image processing. It is preferable that the user can change various settings while checking the image after image processing on the display device 82 . For example, the user can specify or change the area on the display device 82 from which the second leaf color value is obtained. A detailed description of the image processing is omitted because it has been described in detail in the first embodiment.

(Third embodiment)
Next, with reference to FIG. 5, a schematic configuration of an imaging device according to a third embodiment of the present invention will be described. FIG. 5 is a block diagram of the imaging device 100 according to this embodiment. The imaging device 100 includes an image processing unit 104 (image processing device) capable of executing the image processing method of the first embodiment.

In the imaging device 100, an object (not shown) forms an image on an imaging device 102 via an imaging optical system 101 (optical system) including a diaphragm 101a (or light blocking member) and a focus lens 101b. In this embodiment, the imaging optical system 101 is an interchangeable lens (lens device) that can be attached to and detached from an imaging apparatus body having an imaging element 102 . However, the present embodiment is not limited to this, and can also be applied to an image pickup apparatus in which the image pickup apparatus main body and the image pickup optical system 101 are integrally configured.

The aperture value (F number) is determined by the aperture 101a or the light blocking member. The imaging device 102 photoelectrically converts a subject image (optical image) formed via the imaging optical system 101 and outputs image data. Image data (electrical signals) output from the image sensor 102 is output to the A/D converter 103 . The A/D converter 103 converts an electrical signal (analog signal) input from the image sensor 102 into a digital signal and outputs the digital signal (captured image) to the image processing unit 104 . The imaging device 102 and the A/D converter 103 constitute imaging means for photoelectrically converting an optical image (subject image) formed by the imaging optical system 101 and outputting a captured image.

The image processing unit 104 processes the digital signal output from the A/D converter 103 (the image generated based on the image data output from the image sensor 102) in the state detection unit 107 and the storage unit 108. Predetermined image processing is performed using the information. The image processing unit 104 has an acquisition unit 104a, a first determination unit 104b, and a second determination unit 104c.

Acquisition means 104a acquires an image (an image based on image data formed by the imaging optical system 101, that is, a captured image). The first determining means 104b determines a plurality of first leaf color values for each of the plurality of first regions based on color information of each of the plurality of first regions included in the image. A second determination means determines a second leaf color value for a second region including the plurality of first regions based on the plurality of first leaf color values.

The storage unit 108 (storage means) stores data such as filter information related to machine learning used when the image processing unit 104 performs image processing (pixel classification processing). However, in this embodiment, data such as filter information related to machine learning may be stored in the internal memory of the image processing unit 104 instead of the storage unit 108 provided outside the image processing unit 104 .

The state detection unit 107 acquires information about shooting conditions (shooting time, focal length, F-number, shooting distance, image height, etc.) at the time of shooting. The system controller 110 adds information about shooting conditions to an image (shot image) and stores it in the storage unit 108 or the image recording medium 109 . The system controller 110 controls the image processing unit 104 , the display unit 105 , the imaging optical system control unit 106 , the state detection unit 107 and the image recording medium 109 . The imaging optical system control unit 106 controls the operation of the imaging optical system 101 . A state detection unit 107 detects the state of the imaging optical system 101 from the information obtained by the imaging optical system control unit 106 . When a desired image is selected from the image recording medium 109 after photographing, the image processing unit 104 can perform the image processing of the first embodiment on that image.

Also, it is known that the brightness of acquired information changes depending on the imaging optical system 101 and the imaging device 102 . In particular, depending on the location of the image 20, a phenomenon in which the luminance is lowered due to the vignetting of the image pickup optical system 101 and the incident angle characteristics of the image pickup device 102 is observed. On the other hand, it is known that the determination result of the leaf color value changes depending on the luminance without a large change in chromaticity. In other words, while the leaf color to be determined is dominated by luminance, there is a serious problem that the luminance changes depending on the imaging optical system 101 and the imaging element 102 that acquire the image 20 .

As described above, when leaf color value determination is performed by machine learning, it is difficult to analyze the effects of the imaging optical system 101 and the imaging device 102 in terms of image space. On the other hand, it is known how much the luminance changes for each region of the output image 20 . Therefore, in each embodiment, it is preferable to correct the second leaf color value according to the second region specified by the user. In particular, taking into consideration the phenomenon that the luminance decreases toward the periphery of the image 20, the phenomenon that the leaf color value is determined to increase toward the periphery of the image 20 is corrected. As a result, it is possible to correct the distribution of leaf colors in the image 20 with high accuracy, although it is simple. In each embodiment, vignetting and the effect of luminance change due to oblique incidence characteristics are taken into consideration, but the phenomenon is not limited to this as long as it affects the determination of the leaf color value.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that can easily achieve high-accuracy and high-resolution leaf color determination.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

For example, each embodiment is not limited to leaf color values, which are color values related to leaves of plants, but can also be applied to color values of flowers, fruits, and the like.

10 image processing device 12 first determination means 13 second determination means

Claims (15)

a first determination means for determining a plurality of first leaf color values for the plurality of pixel regions based on the plurality of color information for each of the plurality of pixel regions in the plant image;
and second determining means for determining, based on the plurality of first leaf color values, a second leaf color value for a determination target region including the plurality of pixel regions in the image. . The second determination means determines the second leaf color value based on a histogram showing the relationship between the plurality of first leaf color values and the number of the pixel regions corresponding to the plurality of first leaf color values. 2. The image processing apparatus according to claim 1, wherein the determination is made. 3. The image processing apparatus according to claim 2, wherein said second determination means determines a centroid value, an average value, or a mode value of said histogram as said second leaf color value. 4. The image processing apparatus according to claim 1, wherein each of said plurality of pixel regions corresponds to one pixel included in said determination target region. 5. The image processing apparatus according to claim 4 , wherein said first determination means determines said first leaf color value for each of all pixels included in said determination target area. each of the plurality of pixel regions is a pixel region including a plurality of pixels included in the determination target region;
3. The first determination means determines the first leaf color value for each of the plurality of pixel regions based on an average value of the color information of each of the plurality of pixels. 4. The image processing device according to any one of 3 . The first determination means determines the first leaf color value with a first resolution,
7. The image processing apparatus according to claim 1, wherein said second determination means determines said second leaf color value with a second resolution higher than said first resolution. 8. The image processing apparatus according to claim 1, wherein the color information is information about luminance values of R, G, and B of each of the plurality of pixel regions. 9. The image processing apparatus according to claim 1, wherein said first determination means determines said first leaf color value using machine learning based on filter information. 10. The image processing apparatus according to claim 9 , wherein the filter information includes a noise filter for excluding pixel areas other than the pixel area corresponding to the plant from the plurality of pixel areas . 11. The image processing apparatus according to claim 9 , wherein said first determination means further determines the probability of said plurality of first leaf color values. The image processing apparatus according to any one of claims 1 to 10 , wherein the second determination means corrects the second leaf color value according to the determination target area specified by the user. . an imaging device that photoelectrically converts an optical image of a plant formed by an imaging optical system and outputs image data;
first determination means for determining a plurality of first leaf color values for the plurality of pixel regions based on the plurality of color information for each of the plurality of pixel regions in an image based on the image data;
and second determining means for determining a second leaf color value for a determination target area including the plurality of pixel areas in the image based on the plurality of first leaf color values. determining, by a first determining means, a plurality of first leaf color values for the plurality of pixel regions based on the plurality of color information for each of the plurality of pixel regions in the image of the plant ;
and determining, by a second determination means, a second leaf color value for a determination target region including the plurality of pixel regions in the image based on the plurality of first leaf color values. Image processing method. determining a plurality of first leaf color values for the plurality of pixel regions based on the plurality of color information for each of the plurality of pixel regions in the plant image;
and determining, based on the plurality of first leaf color values, a second leaf color value for a determination target region including the plurality of pixel regions in the image.

Images