JP7749384B2 - Forest Image Synthesis System - Google Patents

Description

This invention relates to technology related to the synthesis of satellite images and aerial photographs, and more specifically to a forest image synthesis system that can generate a composite image after removing clouds from the images.

In recent years, the area of natural forests on Earth has been decreasing, with the United Nations reporting that the area of natural forests lost since 2015 is approximately 100,000 ^km2 per year, which means that an area of forest equivalent to the size of Tokyo is lost every week. Deforestation, also known as deforestation, is caused by a variety of factors, including the construction of infrastructure such as roads, the development of farmland, logging to obtain raw materials (wood, paper, pulp, palm oil, natural rubber, etc.), and natural fires.

Deforestation puts wild animals at risk of extinction, which in turn causes the collapse of existing ecosystems and may even have an impact on human life. Deforestation also has a significant impact on global warming, a pressing issue facing the Earth today. Trees continue to grow by absorbing carbon dioxide from the air, and it is said that forests absorb approximately 29% of the carbon dioxide emitted by human activity, meaning that forests store large amounts of carbon dioxide. Therefore, cutting down trees not only reduces the forest's ability to absorb carbon dioxide from the atmosphere, but also releases the carbon dioxide stored in the trees.

Carbon dioxide is a greenhouse gas that traps heat emitted from the Earth. Therefore, if the amount of carbon dioxide released into the atmosphere increases, the Earth's temperature will rise, which will lead to a decrease in Arctic sea ice and rising sea levels, as well as increased flooding, hurricanes, and heat waves, as well as prolonged droughts.

As mentioned above, deforestation occurs for a variety of reasons, including unplanned logging. While necessary minimum logging is acceptable, unnecessary and non-urgent logging should be avoided, and future logging plans should be controlled as logging progresses. Therefore, managing the logging situation in forests is extremely beneficial, and it is desirable to plan future tree planting and logging based on this management. Using images taken regularly is an effective way to manage forest conditions. Satellite images taken by satellites equipped with optical sensors and aerial photographs taken by cameras mounted on aircraft can be used to check changes in forests over time from above, allowing for management of forest conditions.

However, satellite images and aerial photographs can sometimes capture obstacles covering forests, and since satellite images are taken regardless of the weather, it is not uncommon for images to include clouds. Naturally, forests directly below clouds are not captured, making it impossible to grasp the extent of deforestation in that area. Furthermore, when matching an image with clouds remaining with an image without clouds, appropriate processing may not be performed, and it is known that combining an image containing clouds with another image results in a noticeable "patchwork" effect. As a result, various techniques have been developed to remove (mask) clouds from images. For example, Patent Document 1 proposes a technique for joining satellite images after overlaying satellite image data to minimize cloud coverage.

Japanese Patent Application Laid-Open No. 2015-215252

As mentioned above, various cloud removal techniques have been proposed to date, including the SCL (Scene Classification Map)-based masking method, which excels at detecting and masking thin cirrus clouds; the cloud scoring masking method, which minimizes excessive cloud removal; the cloud-shadow shift method, which can reduce (i.e., mask out) small cloud patches that are likely to result in errors; and the FMASK method, among many other cloud removal techniques. However, none of these cloud removal techniques can completely extract and remove clouds (i.e., with absolutely no mis-extraction or omissions). While each technique has its own advantages, such as being able to "remove even fog-like objects" or "highly effective at distinguishing between shadows and artificial objects and clouds," each technique also has its own weaknesses, such as an inability to properly extract clouds under certain image conditions.

Normally, areas where clouds have been removed are filled in with images taken at other times. In other words, multiple images of the same area taken at different times are used, clouds are removed from the images, and then these images are combined. For example, if there are a considerable number of images, such as those taken regularly throughout the year, it is unlikely that the same area will always be covered in clouds, so even if clouds have been removed, the area can be filled in with one of the images.

However, when trying to understand forest changes over a relatively short period of time, it may not be possible to fill in the areas where clouds have been removed with other images. For example, when using images from several days to a week, it is possible that clouds will be present in the same area in all of the images, in which case the images acquired during that period will not be able to cover the issue. As a result, forest changes are understood from images with clouds removed, and naturally, the more information that can be obtained from the images, the more accurately forest changes can be understood; in other words, the smaller the area of the image that is deemed cloudy is, the better. On the other hand, as mentioned above, conventional cloud removal techniques remove different types of clouds; in other words, the amount of information that can be obtained from images varies depending on the cloud removal technique used. This means that the accuracy of understanding forest changes depends on the cloud removal technique.

The objective of the present invention is to solve the problems of the past, namely to provide a forest image synthesis system that can remove clouds more accurately than conventional technology and can generate a more appropriate synthetic image even from a small number of images acquired over a relatively short period of time.

The present invention focuses on the use of two or more cloud removal technologies to remove (mask) clouds from an image, in other words, to remove clouds while complementing the inherent advantages and disadvantages of each technology, and is an invention based on a previously unseen concept.

The forest image synthesis system of the present invention is a system that creates a composite image based on multiple forest images obtained by photographing the same area of forest at different times, and is equipped with a cloud removal means and an image synthesis means. The cloud removal means is a means for performing cloud removal processing on each of the multiple forest images to generate cloud-removed images in which clouds have been removed from the forest images, while the image synthesis means is a means for performing image synthesis on cloud-removed image sets (image sets consisting of multiple cloud-removed images obtained by the cloud removal means) to generate a provisional composite image. The cloud removal means generates multiple types of cloud-removed image sets by performing different types of cloud removal processing on the cloud-removed image sets. The image synthesis means generates multiple provisional composite images based on the multiple types of cloud-removed image sets, and generates a completed composite image by performing image synthesis on the multiple provisional composite images.

The forest image synthesis system of the present invention can also generate cloud-removed images by removing secondary removal pixels from each forest image. In this case, the cloud removal means sets primary removal pixels (pixels that reflect clouds) for each forest image, and sets primary removal pixels common to all forest images as secondary removal pixels.

The forest image synthesis system of the present invention can also generate a provisional synthesis image based on representative pixel values. In this case, the image synthesis means statistically processes the pixel values of valid pixels (pixels excluding secondary removal pixels) in the cloud-removed images that make up the cloud-removed image set, thereby determining the representative pixel value for each valid pixel.

The forest image synthesis system of the present invention can also determine the representative pixel value by performing statistical processing after excluding the pixel values of valid pixels that have been designated as primary removal pixels.

The forest image synthesis system of the present invention can also generate a completed composite image based on representative pixel values. In this case, the image synthesis means removes secondary removal pixels from the pixels that make up the provisional composite image, and then calculates the representative pixel value by statistically processing the pixel values of the pixels in multiple provisional composite images.

The forest image synthesis system of the present invention can also be configured so that the image synthesis means determines the median value of multiple pixel values as the representative pixel value.

The forest image synthesis system of the present invention can also be configured so that the image synthesis means performs statistical processing of vegetation indices as pixel values.

The forest image synthesis system of the present invention has the following advantages.
(1) It is possible to remove clouds from images more accurately than conventional techniques without being affected by the characteristics (advantages and disadvantages) of cloud removal techniques.
(2) Even if only a small number of images are acquired in a relatively short period of time, a composite image can be generated more appropriately than in the prior art.
(3) As a result, even if the photographs are taken over a short period of time, it is possible to accurately grasp the changes in the forest conditions.

A model diagram that schematically shows a forest image set consisting of forest images acquired at seven different times. FIG. 1 is a model diagram illustrating a procedure for generating a composite image according to the present invention. 1 is a block diagram showing the main configuration of a forest image synthesis system according to the present invention. FIG. 10 is a model diagram that schematically shows a primary cloud-removed image obtained as a result of the cloud removal means performing cloud removal processing on the forest image. FIG. 10 is a model diagram that schematically shows a secondary cloud-removed image that is generated by the cloud removal means on the forest image based on the primary cloud-removed image. FIG. 10 is a graph illustrating the median value obtained from pixel values obtained from seven forest images. FIG. 10 is a model diagram illustrating pixels that are the subject of statistical processing. FIG. 1 is a flowchart showing the main processing flow of the forest image synthesis system of the present invention.

An example embodiment of the forest image synthesis system of the present invention will be explained with reference to the accompanying drawings.

1. Overall Overview The present invention uses satellite images or aerial photographs (hereinafter referred to as "forest images") obtained by photographing forests to obtain images suitable for confirming changes in forests over time, and is a technology for generating a composite image based on multiple forest images (hereinafter referred to as "forest image sets") taken of the same area at different times, as shown in Figure 1. Note that Figure 1 shows a forest image set consisting of forest images taken daily for one week (first to seventh periods), but this is not limiting and a forest image set consisting of two or more forest images can also be used.

Figure 2 is a model diagram that schematically illustrates the procedure for generating a composite image according to the present invention. As shown in this figure, the present invention performs cloud removal processing on forest images before combining the images. However, two or more different types of cloud removal processing are performed on the same forest image set. In this figure, three different types of cloud removal processing (first cloud removal processing to third cloud removal processing) are performed on all (seven in the figure) forest images that make up the forest image set. This cloud removal processing can employ various conventional techniques, such as the SCL-based masking method, cloud scoring masking method, cloud-shadow shift method, and FMASK method, as previously described.

When multiple types of cloud removal processes are performed on the forest images that make up a forest image set, multiple combinations (hereinafter referred to as "cloud-removed image sets") of cloud-removed images (hereinafter referred to as "cloud-removed images") are obtained. For example, in Figure 2, three types of cloud removal processes are performed on seven forest images, resulting in three types of cloud-removed image sets (first cloud-removed image set to third cloud-removed image set). Once the cloud-removed image sets are obtained, each cloud-removed image set is combined to generate a composite image (hereinafter referred to as a "tentative composite image"). In Figure 2, three types of cloud removal processes are performed, resulting in three types of tentative composite images (first tentative composite image to third tentative composite image). Then, by further combining multiple types (three types in the figure) of tentative composite images, the final composite image (hereinafter referred to as a "completed composite image") is generated.

2. Forest Image Synthesis System Figure 3 is a block diagram showing the main components of the forest image synthesis system 100 of the present invention. As shown in this figure, the forest image synthesis system 100 is configured to include a cloud removal means 101 and an image synthesis means 102, and can also be configured to include an output means 103 such as a display or printer, and a forest image storage means 104.

The cloud removal means 101 and image synthesis means 102 that make up the forest image synthesis system 100 can be manufactured as dedicated units, or a general-purpose computer device can be used. This computer device is equipped with a processor such as a CPU, memory such as ROM and RAM, and may also include input means such as a mouse and keyboard, and a display (i.e., output means 103), and can be configured, for example, as a personal computer (PC) or server.

The forest image storage means 104 that stores the forest image set can use the storage device of a general-purpose computer (e.g., a personal computer), or can be built on a database server. If built on a database server, it can be placed on a local network (LAN: Local Area Network), or it can be a cloud server that stores data via the Internet.

Below, we will provide a detailed explanation of each of the main elements that make up the forest image synthesis system 100 of the present invention.

(Cloud removal means)
The cloud removal means 101 is a means for reading a forest image set from the forest image storage means 104 and performing cloud removal processing on the multiple forest images that make up the forest image set. However, as described above, this cloud removal means 101 performs two or more different types of cloud removal processing on the same forest image set. Note that the cloud removal means 101 generates an image in which the cloud removal processing has been performed on each forest image (hereinafter referred to as a "primary cloud-removed image"), and also generates an image in which clouds have been removed by further processing based on this primary cloud-removed image (hereinafter referred to as a "secondary cloud-removed image"). The procedure by which the cloud removal means 101 generates the primary cloud-removed image to the secondary cloud-removed image will be described below with reference to Figures 4 and 5.

First, the cloud removal means 101 performs cloud removal processing on each forest image (four forest images in Figure 4) that makes up the forest image set, thereby setting pixels that reflect clouds (hereinafter referred to as "primary removal pixels"), and these primary removal pixels are designated as removal targets (i.e., mask targets). Figure 4 is a model diagram that schematically illustrates the primary cloud-removed image obtained as a result of the cloud removal means 101 performing cloud removal processing on the forest image, i.e., as a result of masking the primary removal pixels. Of all pixels (pixels A1 to F8), pixels shaded in gray are designated as primary removal pixels (e.g., pixel C2). Note that while primary removal pixels are set for each pixel, they can also be set for individual pixels (e.g., pixel F3 in the primary cloud-removed image from the first period in the figure), or they can be set only for a certain number of pixel clusters (above a predetermined threshold).

Once the primary cloud-removed image is generated, the cloud removal means 101 generates a secondary cloud-removed image based on this primary cloud-removed image. Specifically, the primary removal pixels common to all primary cloud-removed images constituting the forest image set are set as secondary removal pixels, and these secondary removal pixels are designated as removal targets (i.e., mask targets). Figure 5 is a model diagram that schematically illustrates the secondary cloud-removed image obtained as a result of masking the secondary removal pixels. Of all pixels (pixels A1 to F8), pixels shaded in black are designated as secondary removal pixels (e.g., pixel D7). For example, pixel sets consisting of pixels C2, C3, D2, and D3, and pixel sets consisting of pixels D7, D8, E6, and E7 are designated as secondary removal pixels because they are designated as primary cloud-removed images in the four primary cloud-removed images (primary cloud-removed images from the first period to the fourth period). On the other hand, pixels A2 and D1, which are set as primary cloud-removed images in the primary cloud-removed image for the second time period, are not set as primary cloud-removed images in the primary cloud-removed images for the other time periods, and therefore are not set as secondary removed pixels. Furthermore, pixel F3, which is set as primary cloud-removed images in the primary cloud-removed image for the first time period, is also set as primary cloud-removed images in the primary cloud-removed image for the fourth time period, but is not set as primary cloud-removed images in the primary cloud-removed images for the second or third time periods, and therefore is also not set as a secondary removed pixel.

In this way, the cloud removal means 101 removes (masks) only primary removal pixels that are common to all primary cloud-removed images as secondary removal pixels; in other words, it removes only pixels that are likely to reflect clouds. This ensures that more pixels are not removed as secondary removal pixels (hereinafter referred to as "valid pixels"), meaning that a larger area of the image can be secured without being masked, thereby obtaining more information for checking changes in the forest over time. Note that, as a result of performing cloud removal processing on a forest image, pixels that are not designated as primary removal pixels (e.g., pixel A1) are of course valid pixels, but pixels that are designated as primary removal pixels but not as secondary removal pixels (e.g., pixel E8) are also valid pixels.

The cloud removal means 101 forms a cloud-removed image set by generating secondary cloud-removed images for all forest images that make up the forest image set (seven forest images in the example of Figure 2), and also forms multiple cloud-removed image sets (three types of cloud-removed image sets in the example of Figure 2) by performing two or more types of cloud removal processing.

(Image synthesis means)
The image composition means 102 is a means for generating a provisional composite image by combining forest images (secondary cloud-removed images) that make up a cloud-removed image set, and for generating a completed composite image by combining a plurality of provisional composite images. Below, the procedure for generating a provisional composite image (hereinafter referred to as "primary image composition") and the procedure for generating a completed composite image (hereinafter referred to as "secondary image composition") will be described in detail.

In primary image synthesis, a "provisional synthesized image" is generated by synthesizing all secondary cloud-removed images that make up the cloud-removed image set (four secondary cloud-removed images in the example shown in Figure 5). Specifically, first, of all the pixels that make up the secondary cloud-removed image, pixels that will not be removed (i.e., masked) as secondary removal pixels are selected, i.e., valid pixels. Then, a pixel value that represents that valid pixel (hereinafter simply referred to as a "representative pixel value") is determined by statistically processing the pixel values assigned to the same valid pixel. For example, pixel A8 shown in Figure 5 is a valid pixel, and its representative pixel value is determined by statistically processing the pixel value of pixel A8 in the secondary removal pixels at the first time point, the pixel value of pixel A8 at the second time point, the pixel value of pixel A8 at the third time point, and the pixel value of pixel A8 at the fourth time point.

Here, pixel values refer to various physical quantities obtained from the pixels that make up the forest image set, and can be RGB, grayscale, or a vegetation index. This vegetation index is an index that grasps the vegetation situation (amount and vitality of plants) by taking advantage of the characteristics of light reflection by plants, and is a physical quantity obtained by a calculation formula that uses satellite data as input values. A well-known example of a vegetation index is the Normalized Difference Vegetation Index (NDVI).

When statistically processing pixel values, various conventional methods can be used, such as calculating a simple average, a weighted average, a mode, or a median. The inventors of the present invention have discovered that when generating a composite image (particularly a completed composite image) using a set of forest images acquired over a relatively short period of time (e.g., one week), it is appropriate to use statistical processing that calculates the median of pixel values as the representative pixel value. For example, in Figure 6, pixel values (NDVI) obtained from seven forest images are plotted in order of observation time (i.e., photographing time), and in this case, the pixel value obtained in the fifth observation (NDVI = 4.0) is calculated as the median.

As explained above, in primary image synthesis, statistical processing is performed on each pixel value assigned to the same effective pixel. However, as previously mentioned, pixels that are designated as primary removed pixels but not as secondary removed pixels (pixels shaded in gray in Figure 5) are also effective pixels, and therefore the pixel values of these primary removed pixels that are not secondary removed pixels (hereinafter referred to as "pure primary removed pixels") can also be subject to statistical processing. However, since it is known that combining an image containing clouds with another image results in a noticeable "patchwork" effect and reduces the accuracy of forest condition assessment, it is desirable not to include these pure primary removed pixels in the statistical processing. For example, in CASE 1 in Figure 7, the four target pixels (pixel A1 at the first time point to pixel A1 at the fourth time point) do not include any pure primary removal pixels, so the representative pixel value is calculated from the pixel values of all pixels. However, in CASE 2, pixel C4 at the third time point is considered a pure primary removal pixel, so the representative pixel value is calculated from the pixel values of the three pixels excluding this (pixel C4 at the first time point, pixel C4 at the second time point, and pixel C4 at the fourth time point). Similarly, in CASE 2, pixel F3 at the first time point and pixel F3 at the fourth time point are considered pure primary removal pixels, so the representative pixel value is calculated from the pixel values of the two pixels excluding these (pixel F3 at the second time point and pixel F3 at the third time point).

Once the representative pixel value is calculated for each effective pixel, a "tentative composite image" is generated in which the secondary removal pixels are masked by assigning the representative pixel value to each effective pixel. Of course, as shown in Figure 2, as many provisional composite images are generated as there are cloud removal processes performed (three in Figure 2).

In secondary image synthesis, a "completed composite image" is generated by synthesizing multiple provisional composite images. Specifically, as with primary image synthesis, a representative pixel value is determined by statistically processing each pixel value assigned to the same pixel, and the completed composite image is generated by assigning the representative pixel value to that pixel. The pixel value used in this process should preferably be the same pixel value (e.g., NDVI) as that used in primary image synthesis. Meanwhile, the statistical processing method for pixel values can be the same as that used in primary image synthesis (e.g., a method of determining the median), or a different method from that used in primary image synthesis, such as a method of determining the mode, can also be used. Furthermore, when statistically processing pixel values, it is desirable to exclude secondary removal pixels (i.e., only valid pixels).

(Processing flow)
The main processing of the forest image synthesis system 100 will be described in detail below with reference to Figure 8. Figure 8 is a flow diagram showing the flow of the main processing of the forest image synthesis system 100 of the present invention. In this flow diagram, the action to be performed is shown in the center column, the items required for that action are shown in the left column, and the items resulting from that action are shown in the right column. In this case, the forest image set is composed of n forest images, and m types of cloud removal processing are performed.

As shown in Figure 8, first, a forest image set is read from the forest image storage means 104, and the cloud removal means 101 generates cloud-removed images (primary cloud-removed image to secondary cloud-removed image) by performing cloud removal processing on the forest images that make up the forest image set (Step 201). Once the cloud removal means 101 has repeatedly generated cloud-removed images n times to obtain a cloud-removed image set, the image synthesis means 102 generates a "provisional synthesized image" by synthesizing all of the secondary cloud-removed images that make up the cloud-removed image set (Step 202).

The cloud removal means 101 generates a cloud-removed image set m times, and the image synthesis means 102 repeatedly synthesizes the secondary cloud-removed images to obtain m provisional composite images. Then, these m provisional composite images are synthesized to generate a "completed composite image" (Step 203). The completed composite image generated here can be confirmed by outputting it to the output means 103, such as a display or printer.

The forest image synthesis system of the present invention can be effectively used in managing the conditions of various forests, including natural forests as well as artificial forests. Considering that the present invention reduces the risk of extinction of wildlife due to deforestation and has the effect of curbing global warming, an urgent issue at present, the present invention can be said to be an invention that not only has industrial applications but is also expected to make a significant contribution to society.

100 Forest image synthesis system of the present invention 101 Cloud removal means (of forest image synthesis system) 102 Image synthesis means (of forest image synthesis system) 103 Output means (of forest image synthesis system) 104 Forest image storage means (of forest image synthesis system)

Claims (7)

A system for creating a composite image based on multiple forest images obtained by photographing the same forest area at different times, comprising:
a cloud removal means for performing cloud removal processing on each of the forest images to generate cloud-removed images in which clouds have been removed from the forest images;
an image synthesis means for generating a provisional synthesis image by performing image synthesis on a cloud-removed image set consisting of a plurality of the cloud-removed images obtained by the cloud removal means;
the cloud removal means sets primary removal pixels that reflect clouds for each forest image, sets the primary removal pixels that are common to all the forest images as secondary removal pixels, and generates the cloud-removed image by removing the secondary removal pixels from each of the forest images;
The cloud removal means generates a plurality of types of cloud-removed image sets by performing different types of cloud removal processing on a plurality of the forest images ,
the image synthesis means calculates a representative pixel value for each valid pixel excluding the secondary removed pixel by statistically processing pixel values of the valid pixels in the cloud-removed images constituting the cloud-removed image set, and generates the provisional synthesized image based on the representative pixel value;
The image composition means generates a plurality of the tentative composite images based on a plurality of types of the cloud-removed image sets, and generates a completed composite image by performing image composition on the plurality of the tentative composite images.
A forest image synthesis system characterized by: The effective pixels include the first-removed pixels that are not set as the second-removed pixels.
2. The forest image synthesis system according to claim 1. the image synthesis means calculates the representative pixel value by performing statistical processing after excluding the pixel values of the effective pixels that have been determined as the primary removed pixels.
2. The forest image synthesis system according to claim 1 . the image synthesis means, after excluding the second-removed pixels from among the pixels constituting the provisional synthesized image, calculates the representative pixel value by statistically processing the pixel values of the pixels relating to the plurality of provisional synthesized images, and generates the complete synthesized image based on the representative pixel value.
4. The forest image synthesis system according to claim 1 , wherein the forest image synthesis system is a system for synthesizing a forest image. the image synthesis means generates the completed composite image based on the representative pixel values obtained by a statistical processing method different from the statistical processing used to generate the provisional composite image.
5. The forest image synthesis system according to claim 4 . the image synthesis means determines a median value of the plurality of pixel values as the representative pixel value;
6. A forest image synthesis system according to claim 1 . the image synthesis means statistically processes the vegetation index as the pixel value;
7. A forest image synthesis system according to claim 1 .