JP7531374B2 - Image processing device, control method and program - Google Patents

Description

This disclosure relates to an image processing device, etc.

A technology has been proposed for image forming devices to obtain appropriate output when the input document has a background.

For example, a technology has been proposed that separates image data of a document into multiple objects including a background object of the document, detects line images that appear in the image data, and treats a pixel of the line image as a background object when the pixel exists in the background object and the pixel value of the pixel of the line image is equal to or less than a threshold value determined based on a representative value of the background (see, for example, Patent Document 1).

In addition, a technique has been proposed that removes pixels determined to be background pixels, thereby preserving the pixel values of image data that are not background pixels (see, for example, Patent Document 2).

JP 2017-175270 A JP 2016-066872 A

However, when faint characters or figures that are expressed with a density close to that of the background are included on top of the background, the area containing the faint characters or figures is likely to be determined to be a background area, and there was a problem in that removing the background would erase the information of the faint characters or figures.

In view of the above-mentioned problems, the present disclosure aims to provide an image processing device and the like that can appropriately output either an image that has undergone background removal processing or an image that has not undergone background removal processing.

The first embodiment for solving the above-mentioned problem is as follows:
An image processing device including an acquisition unit that acquires first image data and a control unit,
The control unit is
determining a level of background removal based on the input first image data;
performing a background removal process on the first image data based on the level of background removal to generate second image data having been subjected to background removal;
Calculating a difference between the first image data and the second image data;
When the difference is equal to or greater than a threshold value, the first image data is output.
When the difference is less than a threshold value, the second image data is output.

The second embodiment for solving the above-mentioned problem is as follows:
determining a level of background removal based on the input first image data;
performing a background removal process on the first image data based on the level of background removal to generate second image data that has been background removed;
calculating a difference between the first image data and the second image data;
outputting the first image data when the difference is equal to or greater than a threshold, and outputting the second image data when the difference is less than the threshold;
The control method includes:

The third embodiment for solving the above-mentioned problem is as follows:
On the computer,
A function of determining a level of background removal based on the input first image data;
a function of performing background removal processing on the first image data based on the level of background removal to generate second image data from which background has been removed;
A function of calculating a difference between the first image data and the second image data;
a function of outputting the first image data when the difference is equal to or greater than a threshold value, and outputting the second image data when the difference is less than the threshold value;
It is a program characterized by realizing the above.

According to the present disclosure, it is possible to appropriately output either an image that has undergone background removal processing or an image that has not undergone background removal processing.

1 is an external perspective view of an image forming apparatus according to a first embodiment. 1 is a block diagram for explaining a functional configuration of an image forming apparatus according to a first embodiment. 5 is a diagram showing an example of a data configuration of a background removal table in the first embodiment. FIG. FIG. 2 is a flow diagram for explaining a process flow in the first embodiment. FIG. 4 is a diagram illustrating an example of a histogram in the first embodiment. FIG. 4 is a diagram for explaining an operation example in the first embodiment. FIG. 4 is a diagram for explaining an operation example in the first embodiment. FIG. 11 is a flow diagram for explaining a process flow in a second embodiment. FIG. 11 is a diagram for explaining an operation example in the second embodiment. FIG. 13 is a flowchart illustrating a process flow in a third embodiment. FIG. 13 is a diagram for explaining an operation example in the third embodiment. FIG. 13 is a flow diagram for explaining a process flow in a fourth embodiment. FIG. 13 is a diagram for explaining determination items in the fourth embodiment. 13A and 13B are diagrams for explaining types of documents in a fourth embodiment. FIG. 13 is a block diagram for explaining a functional configuration of an image forming apparatus according to a fifth embodiment. FIG. 13 is a diagram for explaining an operation example in the fifth embodiment.

One embodiment for carrying out the present disclosure will be described below with reference to the drawings. Note that the following embodiment is an example for explaining the present disclosure, and the technical scope of the invention described in the claims is not limited to the following description.

[1. First embodiment]
[1.1 Functional configuration]
The functional configuration of an image forming apparatus 10 according to the present embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is an external perspective view of the image forming apparatus 10, and Fig. 2 is a block diagram showing the functional configuration of the image forming apparatus 10.

The image forming device 10 is a digital multifunction device having a copy function, a print function, a scanner function, an email sending function, etc. As shown in FIG. 2, the image forming device 10 is configured with a control unit 100, an image input unit 120, an image forming unit 130, a display unit 140, an operation unit 150, a storage unit 180, and a communication unit 190.

The control unit 100 controls the entire image forming device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 180, and is composed of one or more arithmetic units (e.g., CPUs (Central Processing Units)).

By executing the program, the control unit 100 functions as a region separation unit 102, an image processing unit 104, an edge detection unit 106, an information loss determination unit 108, and an output control unit 110.

The region separation unit 102 separates the image data into predetermined regions and determines that each pixel in the image data is a pixel contained in a predetermined region.

For example, the region separation unit 102 determines whether each pixel in the image data belongs to a background region, a photograph (photographic paper photograph) region, a text region, or a halftone region, based on the density value signals in each of the RGB (R: red, G: green, B: blue) color signals. In this embodiment, region separation processing refers to separating an image into a text region, a photographic paper region (photographic region; continuous tone region), a halftone region, etc. A text region represents characters including letters and symbols. A background region represents the bare part (background) of a document.

For example, the region separation unit 102 determines the pixels contained in the image represented by the image data. Each pixel is determined by separating the image data into one of background candidate pixels, photographic paper pixels (photo pixels), character pixels, and halftone dot pixels. Hereinafter, background candidate pixels, photographic paper pixels, character pixels, and halftone dot pixels are referred to as region pixels. Background candidate pixels are pixels that are extracted as having a high probability of being background pixels in the region pixel determination process.

The algorithm for separating the pixels into regions can use an existing region separation method, such as the method described in Japanese Patent Application Laid-Open No. 2002-232708 below.
(1) The minimum and maximum density values in a pixel block of n×m (for example, 7 pixels×15 pixels) including the pixel of interest are calculated.
(2) The calculated minimum and maximum density values are used to calculate a maximum density difference.
(3) A total density busyness, which is the sum of the absolute values of density differences between pixels adjacent to the pixel of interest (for example, the sum of values calculated in the main scanning direction and the sub-scanning direction), is calculated.
(4) The calculated maximum density difference is compared with a maximum density difference threshold, and the calculated total density busyness is compared with a total density busyness threshold.
(5) When the maximum density difference<the maximum density difference threshold and the total density busyness<the total density busyness threshold, the pixel of interest is determined to belong to the background/photographic paper region.
(6) If the above conditions are not satisfied, the pixel of interest is determined to belong to a character/halftone dot area.
(7) For pixels determined to belong to the background/photographic paper area, if the pixel of interest satisfies the maximum density difference < background/photographic paper determination threshold, it is determined to be a background candidate pixel, and if this condition is not satisfied, it is determined to be a photographic paper pixel.
(8) For pixels determined to belong to a character or halftone dot area, if the pixel of interest satisfies the condition of total density busyness < (maximum density difference multiplied by character/halftone dot determination threshold), it is determined to be a character pixel; if this condition is not satisfied, it is determined to be a halftone dot pixel.

Note that the background region included in image data refers to the region composed of background candidate pixels. Similarly, the region composed of photographic paper pixels (photographic pixels) is called the photographic paper region, the region composed of character pixels is called the character region, and the region composed of halftone pixels is called the halftone region.

The image processing unit 104 performs various image processing operations on the image data. For example, the image processing unit 104 generates image data (second image data) of an image in which the luminance (density) values of each pixel included in the image data (first image data) of the image input by the image input unit 120 are replaced with values defined in a background removal table 184 described below.

The edge detection unit 106 detects edges from image data. For example, the edge detection unit 106 applies an edge detection operator (for example, an operator called Prewitt or Sobel) to an image input by the image input unit 120 or an image processed by the image processing unit 104. In this way, the edge detection unit 106 converts the image for which edges are to be detected into an image in which the edges are accentuated, or detects edges from the image for which edges are to be detected and generates an image (edge image) that shows only the detected edges.

The information loss determination unit 108 compares the two images to determine whether or not there is information loss. Information loss refers to characters, figures, etc. that appear in one image being missing in the other image.

The output control unit 110 controls the output of images. For example, the output control unit 110 controls the output of the image to be output via the image forming unit 130, and controls the transmission of image data of the image to be output to another device via the communication unit 190.

The image input unit 120 inputs (acquires) image data to the image forming apparatus 10. The image input unit 120 is, for example, configured with a scanner device capable of optically reading an image from a document and generating image data. The scanner device reads the reflected light image from the document using an image sensor such as a CCD (Charge Coupled Device) or a CIS (Contact Image Sensor) as image data consisting of analog signals that correspond to RGB (R: red, G: green, B: blue) and represent the reflectance of reflected light from a recording medium such as paper on which the document image is recorded. The image input unit 120 then generates the read image data (digital data) to input the image data. The image input unit 120 may input image data stored in a storage medium such as a USB (Universal Serial Bus) memory or an SD card, or may input image data acquired from another device via the communication unit 190.

The image forming unit 130 forms an image based on image data on a recording medium (e.g., recording paper). The image forming unit 130 is, for example, configured with a laser printer that uses an electrophotographic method, feeds recording paper from the paper feed tray 132 in FIG. 1, forms an image on the surface of the recording paper, and discharges the recording paper from the paper discharge tray 134.

The display unit 140 displays various information. The display unit 140 is configured with a display device such as an LCD (Liquid crystal display), an organic EL (electro-luminescence) panel, or a micro LED (Light Emitting Diode) display.

The operation unit 150 accepts operations from a user who uses the image forming device 10. The operation unit 150 is composed of hard keys (e.g., a numeric keypad), buttons, a sensor that detects touch operations, and the like. The method for detecting the input touch operations may be any common detection method, such as a resistive film method, an infrared method, an electromagnetic induction method, or a capacitive method. The image forming device 10 may also be equipped with a touch panel in which the display unit 140 and the operation unit 150 are integrally formed.

The storage unit 180 is a functional unit that stores various programs and various data necessary for the operation of the image forming device 10. The storage unit 180 is configured, for example, by a storage device such as a solid state drive (SSD) or a hard disk drive (HDD), which is a semiconductor memory.

The storage unit 180 secures an image data storage area 182, which is a storage area for storing image data input by the image input unit 120. The storage unit 180 also stores a background removal table 184 that is used to remove the background from the image data.

The background removal table 184 will be described with reference to FIG. 3. FIG. 3(a) is an example of the background removal table 184. As shown in FIG. 3(a), the background removal table 184 stores the correspondence between input values and output values. The input value is a value indicating the color of the pixel of interest in the image data before conversion, and specifically, is a density value corresponding to each of RGB (a value corresponding to each of the RGB colors).

As shown in FIG. 3(a), multiple levels (tables) are defined for the output values, such as level 1 and level 2, depending on the level of background removal. In this embodiment, there are 16 levels, from 1 to 16.

The value written at the intersection of the input value and the level indicates the density value after conversion for the pixel of interest. In particular, in this embodiment, the density value after conversion is called the output value. In other words, the background removal table 184 is a table that indicates to which density value the density values of the input R, G, and B colors are converted for each color.

In this embodiment, the density value takes any integer value between 0 and 255, and the output value is defined in the background removal table 184 so that the output value is greater than the input value. Therefore, when the density value of image data is converted using the background removal table 184, the converted image becomes brighter than the image before conversion. As a result, the faint background contained in the image is removed by conversion based on the background removal table 184.

In this embodiment, the background removal table 184 is defined so that the image is converted to a brighter image as the level number increases. For example, at level 1, the output value is replaced with the same value as the input value, and the background removal table 184 is defined so that the brightness of pixels with high brightness is replaced with a higher brightness as the level number increases.

Figure 3(b) is a graph showing the relationship between the input values and the output values stored in the background removal table 184. As shown in Figure 3(b), the higher the level number, the more pixels of intermediate brightness are replaced with pixels of higher brightness.

The communication unit 190 allows the image forming apparatus 10 to communicate data with external devices via an external network such as the Internet or a network such as a LAN (Local Area Network). The communication unit 190 is configured, for example, by a communication device or communication module such as a NIC (Network Interface Card) used in a wired/wireless LAN.

[1.2 Processing flow]
The main processes executed by the control unit 100 of the image forming apparatus 10 in this embodiment will be described with reference to Fig. 4. The processes shown in Fig. 4 are executed by the control unit 100 reading out a program stored in the storage unit 180, for example.

First, the control unit 100 acquires image data (step S102). For example, the control unit 100 may acquire image data of a document via the image input unit 120, or may acquire image data stored in a storage medium or image data acquired from another device.

Next, the control unit 100 (area separation unit 102) executes an area separation process based on the image data acquired in step S102 to separate the image represented by the image data into predetermined areas (step S104). For example, the area separation unit 102 determines whether each pixel in the image data belongs to a background area, a photographic paper area, a character area, or a halftone dot area.

Next, the control unit 100 calculates a histogram based on the density values of the pixels included in the background region based on the region separation result in step S104 (step S106). The histogram is generated, for example, by predetermining a luminance range (division) and counting the occurrence frequency of the density values of the pixels included in the background region for each division.

Specifically, the control unit 100 calculates the histogram as follows. First, the control unit 100 compares the average density values of the planes (signals) of the background candidate pixels for the target pixels determined to be background candidate pixels, calculates the minimum value of the average values among the color signals for each target pixel, and generates a background pixel histogram. In this case, the range targeted for the generation of the histogram is the entire background candidate pixels. Therefore, if the density value for each color signal of each background candidate pixel is ( _R1 , _G1 , _B1 ), this background pixel histogram represents the frequency value of the value of MIN ( _R1 , _G1 , _B1 ) for all the background candidate pixels. The number of density sections of this histogram is, for example, 16, with the first density section, the second density section, and the section with the largest density value being the 16th density section.

Figure 5 shows a specific example of a histogram generated in step S106. Figure 5(a) shows a table (histogram) in which 16 sections (density sections) are defined by dividing density values from 0 to 255 into 16 equal parts in units of 16, and the density values (input values) of each pixel included in the image data are counted for each section, showing the frequency of occurrence for each section. In this embodiment, the 16 sections are sections 0 to 15. Figure 5(b) shows the histogram shown in Figure 5(a). In the histogram shown in Figure 5(b), the horizontal axis shows density values indicating the density sections, and the vertical axis shows frequency values which are the frequency of the minimum values belonging to each density section.

Here, FIG. 5 shows a histogram when a background is present. As shown in FIG. 5, the control unit 100 basically determines that an area consisting of pixels of a single color is the background. Since the background only has pixels of a single color density, when a background is present, frequency values exist only in a limited density range group (for example, a set of density ranges consisting of one or two ranges) as shown in FIG. 5(b). If there is no background, for example when the entire document image data is composed of a photographic paper area, even if a histogram is generated for all the pixels extracted as background candidate pixels, there may be density range groups with a wide range of ranges or no density range groups with large frequency values.

Next, the control unit 100 determines whether the input image is an image to be subjected to background removal processing (step S107). The control unit 100 determines whether the image is an image to be subjected to background removal processing, for example, by the following procedure.

(1) Counting Region Pixels The control unit 100 counts the number of pixels determined for each region pixel determined in step S104.

(2) Background Determination The control unit 100 determines whether or not a background exists in the input image data. For example, as shown in FIG. 5B, the control unit 100 determines that a background exists if there is a density division group whose frequency value is greater than a predetermined background determination threshold Sg24 and whose density division width is smaller than a background determination division width threshold (e.g., 3). The background determination threshold Sg24 is set to, for example, 10,000 pixels when the output paper size is assumed to be A4 size and the resolution is 600 dpi (dots per inch). When the resolution is 600 dpi, the number of pixels in the entire output paper of A4 size is approximately 3.5× ¹⁰⁷ pixels.

Next, the control unit 100 determines the smallest density classification value among the density classification groups determined to be background pixels in the histogram for the background candidate pixels as the minimum background pixel density value A. The control unit 100 also determines the sum of the frequency values belonging to the background density classification group as the background frequency value.

When using RGB signals, the closer the pixel value is to "0", the higher the actual density (in this embodiment, this is expressed as "the density value is smaller"), and the closer the pixel value is to "255", the lower the actual density (in this embodiment, this is expressed as "the density value is larger").

(3) Document Type Determination Next, the control unit 100 determines the document type. The document type determination method is to compare the number of area pixels counted for each area pixel with predetermined threshold values corresponding to the background area, the photographic paper area, the halftone dot area, and the character area, respectively, to determine the type of the entire document.

For example, assuming that the detection accuracy is highest for text, followed by halftone dots and then photographic paper, if the ratio of pixels in the text area is 30% or more of the total number of pixels, it is determined to be a text manuscript; if the ratio of pixels in the halftone dot area is 20% or more of the total number of pixels, it is determined to be a halftone dot manuscript (printed photograph manuscript); and if the ratio of pixels in the photographic paper area is 10% or more of the total number of pixels, it is determined to be a photographic paper manuscript. Furthermore, if the ratio of text area and the ratio of halftone dot area are each above a threshold value, it is determined to be a text/halftone dot manuscript (text printed photograph manuscript). Furthermore, if the ratio of text area and the ratio of photographic paper area are each above a threshold value, it is determined to be a text/photo manuscript (text photographic paper manuscript).

(4) Determination of whether the document is a target for background removal Next, the control unit 100 determines whether the determined document type is a document for which background removal should be performed. Documents for which background removal should be performed include text documents, documents that do not include a photographic paper area, text/halftone dot documents (text-printed photograph documents) with a small halftone dot area ratio (for example, a document with a halftone dot area ratio of 40% or less of the entire document), etc.

(5) Determination of whether or not there is a background to be removed When it is determined that the document is one for which background removal is to be performed, the control unit 100 determines whether or not there is a background region for which background removal is to be performed. If the minimum background pixel density value A obtained from the histogram is equal to or greater than a predetermined background density value, it is determined that there is a background region for which background removal is to be performed, and if the minimum background pixel density value A is less than the predetermined background density value, it is determined that the background region is one for which background removal is not to be performed.

By the above-mentioned procedure, the control unit 100 judges whether or not the background area included in the image data is an area where the background should be removed. If the background area included in the image data is an area where the background should be removed, the control unit 100 executes the background removal process (step S107; Yes). On the other hand, if the background area included in the image data is not an area where the background should be removed, the control unit 100 ends the process shown in FIG. 4 (step S107; No).

When performing background removal processing, the control unit 100 calculates a boundary from the histogram generated in step S106 in order to select a background removal table to be used in the background removal processing (step S108). In this embodiment, the boundary refers to a location that divides the histogram into two ranges. Specifically, for example, the boundary is between category 12 and category 13.

The boundary may be found by a predetermined method. For example, if the histogram has two peak divisions, the boundary may be the midpoint between the two divisions, or the boundary may be between the division with the lowest frequency and the division adjacent to that division. The histogram may also be assumed to be the sum of two normal distributions, and the boundary may be calculated based on the parameters of the two normal distributions. The boundary calculated by this process is shown, for example, in FIG. 5(b).

Then, the control unit 100 determines the level of background removal based on the boundary determined in step S108 (step S110). For example, the control unit 100 selects the level with the number corresponding to the number of the section next to the boundary. Specifically, if the boundary is between sections 12 and 13 in step S108, the control unit 100 determines level 13 as the level of background removal. The control unit 100 then performs background removal processing based on the background removal table corresponding to the determined level.

Next, the control unit 100 (image processing unit 104) executes automatic background processing to generate image data (second image data) from the image data (first image data) of the image acquired in step S102 with the background automatically removed (step S112). Automatic background processing refers to a process of converting the initially acquired image into a brighter image by replacing the density value of each pixel included in the image acquired in step S102 based on the background removal table 184. Here, since the level of background removal was determined in step S110, the control unit 100 (image processing unit 104) replaces the density value based on that level. Note that in this embodiment, removing the background by automatic background processing is also referred to as automatic background removal.

In addition to the above-mentioned processing method, the background removal process can use existing removal processes, for example, the method described in JP-A-2000-354167. For example, if it is determined that a background area included in the image data is an area where background removal is required, the control unit 100 selects a background removal table corresponding to the background density value and performs background removal process.

Next, the control unit 100 (edge detection unit 106) performs edge processing (edge detection) on the image that has been subjected to automatic background removal in step S112 (step S114). Note that in parallel with the processing of step S114, the control unit 100 (edge detection unit 106) also performs edge processing (edge detection) on images that have not been subjected to automatic background removal (step S116).

Next, the information loss determination unit 108 compares the image subjected to edge processing in step S114 with the image subjected to edge processing in step S116 (step S118). For example, the information loss determination unit 108 generates a monochrome image (comparison image) by subtracting the density value of a pixel at a corresponding position in the image subjected to edge processing in step S116 from the density value of a pixel included in the image subjected to edge processing in step S114. In this case, the magnitude of the density values of the two images appears as the density value of each pixel in the comparison image. Therefore, in the comparison image, a pixel with a larger density value indicates a pixel with a larger difference between the two images, and a pixel with a smaller density value indicates a pixel with a smaller difference between the two images.

Then, the information loss determination unit 108 determines whether or not there is information loss (step S120). For example, if a difference image is generated in step S118, the information loss determination unit 108 quantitatively calculates the degree of information loss and the size of the area where information loss occurs based on the pixel values included in the difference image.

In this embodiment, the value calculated based on the degree of information loss and the size of the area where information loss has occurred is called the difference score. If the value of the difference score is equal to or greater than a preset threshold, the information loss determination unit 108 determines that there is (will be) information loss when the image acquired (input) in step S102 is converted into a bright image based on the determination pattern determined in step S110.

Specifically, the information loss determination unit 108 determines the difference score and whether or not there is information loss by the following method.
(1) Method based on the number of pixels having a density value equal to or greater than a predetermined density value The information loss determination unit 108 counts the number of pixels having a density value equal to or greater than a predetermined density value for determination (determination density value) from the difference image. For example, the information loss determination unit 108 calculates MIN ( _R1 , _G1 , _B1 ) as a density value for each pixel of the difference image, and determines whether the calculated density value is equal to or greater than the determination density value. If the calculated density value is equal to or greater than the determination density value, the information loss determination unit 108 counts the pixel as a pixel having a density value equal to or greater than a predetermined density value. The information loss determination unit 108 regards the number of pixels counted in this way as the difference score.

The information loss determination unit 108 also sets the threshold value to a pixel count that is a predetermined ratio (determination ratio) to the image size or a pixel count that corresponds to the image size (determination pixel count). The information loss determination unit 108 determines that there is information loss when the difference score is equal to or greater than the pixel count based on the determination ratio or the determination pixel count.

(2) Method of Using the Sum of Density Values as the Difference Score The information loss determination unit 108 uses the sum of all density values of pixels included in the difference image as the difference score. The information loss determination unit 108 also uses a score (determination score) corresponding to the size of the image as the threshold. If the difference score is equal to or greater than the determination score, the information loss determination unit 108 determines that there is information loss.

If it is determined in step S120 that there is no missing information, the control unit 100 (output control unit 110) outputs image data (second image data) that has been subjected to automatic background removal from the image data (first image data) of the image input in step S102 (step S120; No → step S122). For example, the output control unit 110 controls the image generated in step S112 to be formed on recording paper via the image forming unit 130, or controls transmission to another device via the communication unit 190.

On the other hand, if it is determined in step S120 that there is missing information, the control unit 100 (output control unit 110) cancels (skips) the automatic background removal process, and outputs the image data (first image data) of the image input in step S102 (step S120; Yes → step S124).

[1.3 Operation example]
Next, an example of the operation of this embodiment will be described with reference to Fig. 6 and Fig. 7. First, information loss will be described with reference to Fig. 6.

Figure 6(a) is a diagram showing image P100 of a document read based on a user's operation. As shown in Figure 6(a), image P100 has a light background across the entire image. Furthermore, image P100 includes area E100 where dark characters appear, area E102 where light characters appear, area E104 where dark figures appear, and area E106 where light figures appear. Light characters and light figures refer to characters and figures that are expressed with a density (brightness) close to the density (brightness) of the background in the background area. Of these, areas E100 and E104 are likely to be separated as character areas, and areas E102 and E106 are likely to be separated as photographic paper areas or background areas.

Figure 6 (b) shows image P110 when automatic background processing is performed on image P100. By removing the background from image P100, the light background that appeared throughout the image disappears (the background is blown away), as shown in image P110, but the light letters and light figures that appeared in image P100 also disappear. In other words, by brightening image P100, the information of light letters and light figures is lost. As a result, only areas E110 of dark letters and areas E112 of dark figures remain in image P110, and the light letters and light figures that were originally present are no longer visible to the user.

Next, edge processing will be described with reference to FIG. 7. FIG. 7(a) is an image P120 of edges detected by performing edge processing on the image P100 shown in FIG. 6(a). Image P120 includes an area E120 containing edges of dark characters, an area E122 containing edges of light characters, an area E124 containing edges of dark figures, and an area E126 containing edges of light figures.

On the other hand, FIG. 7(b) shows an image P130 of edges detected by performing edge processing on the image P100 shown in FIG. 6(b). Image P130 includes an area E130 that includes edges of dark characters and an area E134 that includes edges of dark figures.

Now, when taking the image difference between image P120 and image P130, the edges contained in region E122 and region E126 in image P120 appear as the difference. Therefore, the edges contained in region E122 and region E126 are shown as information that is missing by brightening image P100.

By appropriately setting the threshold, the difference score based on the edges contained in region E122 and region E126 is determined to be equal to or greater than the threshold, and automatic background removal is canceled. As a result, image P100 as shown in FIG. 6(a) is output from image forming device 10.

In the above embodiment, the determination of whether or not to perform background removal processing based on the density value, the determination of the background removal table, and the processing of converting the input density value to an output value are described, but the values used for the determination and processing may be lightness or luminance values. In this case, the control unit 100 converts the density value based on the input RGB density value signal into a pixel value in the HSV color space or the XYZ color space to obtain the lightness or luminance value. Then, the control unit 100 determines whether or not to perform background removal processing and the background removal table based on the lightness or luminance value. The control unit 100 also converts the lightness or luminance value based on the background removal table, and converts the converted pixel value back into an RGB density signal. In this way, the control unit 100 can perform background removal by converting the density value corresponding to each of RGB based on the lightness or luminance value.

According to this embodiment, when background removal is performed in accordance with the luminance of the background, it is possible to prevent information such as characters and figures with a density close to that of the background from disappearing along with the background, resulting in the loss of necessary information. Therefore, even if information that would be lost by background removal appears in the image, it is possible to automatically reproduce the information without erasing it.

[2. Second embodiment]
Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the method (level) of background removal processing is switched when there is a loss of information. In this embodiment, FIG. 4 of the first embodiment is replaced with FIG. 8. Note that the same functional units and processes are given the same reference numerals, and the description will be omitted.

The main processing of this embodiment will be described with reference to FIG. 8. In this embodiment, if it is determined in step S120 that there is a loss of information, the control unit 100 omits the process of outputting the image (first image data) acquired in step S102. Instead, the control unit 100 changes the level to a number smaller than the level number determined in step S110 (step S120; Yes -> step S202). That is, in this embodiment, the control unit 100 weakens the level (degree) of background removal and repeats the process from step S112. As a result, in the second processing of step S112, the image processing unit 104 generates an image acquired in step S102 that is brighter than the image acquired initially and darker than when S112 was executed the first time.

As a result, the image (third image data) generated in the second automatic background removal is closer to the image (first image data) acquired in step S102 than the image (second image data) generated in the first automatic background removal. Therefore, the density values of the pixels included in the comparison image generated in step S118 for the second time are smaller than the density values included in the comparison image generated in step S118 for the first time, and the difference score is also smaller.

At this time, if the second difference score is less than the threshold value, the output control unit 110 outputs an image (third image data) on which automatic background removal has been performed based on the level changed in step S202 (step S120; No → step S124).

If it is determined in step S120 that there is again a loss of information (step S120; Yes), the control unit 100 may execute step S202 and the processes from step S112 to step S120 again. In step S202 from the second time onwards, the control unit 100 further weakens the level of the base processing by changing the level to a lower number than the level number determined in the immediately preceding step S202.

In this way, by gradually weakening the level of background processing, the control unit 100 can perform background processing at a level where no information is lost, and can output image data where no information is lost. In other words, the control unit 100 can optimize background removal by searching for a level of background removal where the difference score is less than the threshold value, and performing automatic background removal based on the optimal level.

An example of operation in this embodiment will be described with reference to FIG. 9. FIG. 9(a) is a diagram showing an image P200 of a document read by a user operation. As shown in FIG. 9(a), image P200 has a light background across the entire image, and further includes dark and light characters, and dark and light shapes.

Figure 9 (b) shows image P210, which is the result of performing automatic background removal based on the optimal level on image P200. As shown in image P210, automatic background removal based on the optimal level is performed, which makes it possible to remove the background while preventing the loss of faint text and faint figures.

According to this embodiment, if performing background removal causes information other than the background to be lost, the method of background removal processing can be switched to output an image in which information other than the background is not lost.

[3. Third embodiment]
Next, a third embodiment will be described. The third embodiment differs from the first embodiment in that, when there is missing information, a process for emphasizing the contours of an image corresponding to the missing information is executed, and the image corresponding to the missing information is replaced with an image with emphasized contours and output. In this embodiment, FIG. 4 of the first embodiment is replaced with FIG. 10. Note that the same functional units and processes are denoted by the same reference numerals, and description thereof will be omitted.

The main processing of this embodiment will be described with reference to FIG. 10. In this embodiment, if it is determined in step S120 that there is missing information, the control unit 100 (output control unit 110) executes edge enhancement processing (step S120; Yes -> step S302).

The edge enhancement process refers to a process in which the background is removed and the image in the area containing the information lost by the background removal is replaced with an image in which the edges of the image in that area are enhanced. Specifically, the control unit 100 (output control unit 110) executes the following process.

(1) Identifying an area where information is missing due to background removal The control unit 100 (output control unit 110) identifies an area where information is missing by performing background removal. For example, the control unit 100 (output control unit 110) identifies an area including pixels with a density value equal to or greater than a predetermined density value (an area where a predetermined difference appears) in the comparison image generated in step S118 as an area where information is missing.

(2) Generation of an image with enhanced contours The control unit 100 (output control unit 110) extracts an image included in an area where information is missing from the images acquired in step S102, and executes a process of emphasizing contours on the extracted image, thereby generating an image with enhanced contours. For example, the control unit 100 (output control unit 110) generates an image with enhanced contours by causing the image processing unit 104 to execute a process of combining an image included in an area where information is missing with an image obtained by executing edge processing on the extracted image.

(3) Combining an image with an image after automatic background removal The control unit 100 (output control unit 110) replaces the image of an area where information is missing from the image after automatic background removal with an image with emphasized edges, thereby combining the image after automatic background removal with the image with emphasized edges. As a result, faint characters and faint figures that have disappeared due to background removal are replaced with characters and figures with emphasized edges only.

After performing the process of (3) described above, the control unit 100 (output control unit 110) outputs the image data of the processed image via the image forming unit 130 and the communication unit 190.

An example of operation in this embodiment will be described with reference to FIG. 11. FIG. 11(a) is a diagram showing an image P300 of a document read based on a user's operation. As shown in FIG. 11(a), image P300 has a faint background across the entire image, and further includes an area E300 in which faint characters appear, and an area E302 in which faint figures appear.

Fig. 11(b) is a diagram showing an image P310 output by this embodiment. As shown in Fig. 11(b), the faint characters and faint figures included in image P300 are replaced with figures with emphasized edges, so that image P310 includes an area E310 where characters with emphasized edges appear and an area E312 where figures with emphasized edges appear.

This embodiment prevents information loss due to background removal, and furthermore makes it possible to output faint text and figures that would otherwise be lost due to background removal in an image that is easily visible to the user.

[4. Fourth embodiment]
Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that it switches whether or not to perform background removal depending on the characteristics of an acquired image. In this embodiment, FIG. 4 of the first embodiment is replaced with FIG. 12. Note that the same functional units and processes are denoted by the same reference numerals, and descriptions thereof will be omitted.

The main processing of this embodiment will be described with reference to FIG. 12. In this embodiment, the control unit 100 reads a document via the image input unit 120, and then determines the type of the document (step S402).

The determination of the type of document will be described with reference to Figs. 13 and 14. In this embodiment, the control unit 100 first classifies the characteristics of the scanned document into five items (determination items) and then determines the type of document based on the results of each determination item.

First, the judgment items related to the characteristics of the document will be described with reference to FIG. 13. FIG. 13 is a table showing, for each judgment item in this embodiment, the information used for judgment and the judgment result calculated by the judgment. In this embodiment, the control unit 100 makes judgments for the following five judgment items.

(1) Character The control unit 100 judges whether or not the document is a text document. A text document is a document that is mainly composed of text. The information used to judge whether or not the document is a text document is the total number of edge pixels that make up the image of the text. The edge pixels that make up the image of the text are, for example, pixels at edges that have a large difference in density value between the pixel of interest and the surrounding pixels and that show a steep rise.

The control unit 100 selects one pixel included in the image data of the scanned document, and calculates the change in density value in a nearby range (for example, a block of five pixels vertically and five pixels horizontally centered on the target pixel) centered on the target pixel. At this time, if the difference in density value between the target pixel and an adjacent pixel is equal to or greater than a predetermined change amount, and it is deemed that a steep change in density value has occurred, the control unit 100 determines that the target pixel is an edge pixel that shows a steep rise. The control unit 100 shifts the target pixel one by one, and counts the number of edge pixels that show a steep rise for the entire image data. The value counted in this way is calculated as the total number of edge pixels that make up the character image (total number of character edge pixels).

The control unit 100 determines that the document is a text document if the total number of character edge pixels is equal to or greater than a predetermined number of pixels (a text area of a predetermined size exists), and determines that the document is not a text document if the total number of character edge pixels is less than the predetermined number of pixels (a text area of a predetermined size does not exist). Note that in this embodiment, the determination result indicating whether the document is a text document is a text area determination result that indicates 1 if there is a text area of a predetermined size, and 0 if there is no text area of a predetermined size.

(2) Halftone dots The control unit 100 judges whether or not a halftone dot area of a predetermined size exists in the document. The control unit 100 selects one pixel included in the image data of the read document, and calculates the change in density value in a nearby range (for example, a block of 5 pixels vertically and 5 pixels horizontally centered on the pixel of interest) centered on the pixel of interest. At this time, the control unit 100 judges that the pixel of interest is a pixel included in the halftone dot area when the change in density value in the block shows a characteristic considered to be a halftone dot. The control unit 100 shifts the pixel of interest one pixel at a time, and counts the number of pixels considered to be included in the halftone dot area for the entire image data. The control unit 100 also calculates the size of the halftone dot area based on the number of pixels considered to be included in the halftone dot area (halftone dot pixel count).

If the size of the dot area is equal to or larger than a predetermined size, the control unit 100 determines that a dot area of the predetermined size is present, and if the size of the dot area is less than the predetermined size, it determines that a dot area of the predetermined size is not present. Note that in this embodiment, the determination result indicating whether or not a dot area of the predetermined size is included is a dot area determination result indicating 1 if a dot area of the predetermined size is present, and 0 if a dot area of the predetermined size is not present.

(3) Photographic Paper The control unit 100 determines whether or not a photographic paper area of a predetermined size exists in the document. Here, it is assumed that the photographic paper area has the characteristic of being a solid area with little change in density value around it.

The control unit 100 selects one pixel included in the image data of the scanned document, and calculates the change in density value in a nearby range (for example, a block of five pixels vertically and five pixels horizontally centered on the target pixel) centered on the target pixel. At this time, the control unit 100 determines that the target pixel is a pixel included in a solid area if there is little change in density value around the target pixel, such as when the difference in density value between the target pixel and an adjacent pixel is less than a predetermined change amount. The control unit 100 shifts the target pixel one by one, and counts the number of pixels that are considered to be included in a solid area for the entire image data. The control unit 100 also calculates the size of the photographic paper area based on the number of pixels that are considered to be included in a solid area (solid pixel count).

The control unit 100 determines that a photographic paper area of the specified size exists if the size of the photographic paper area is equal to or larger than the specified size, and determines that a photographic paper area of the specified size does not exist if the size of the photographic paper area is less than the specified size. Note that in this embodiment, the determination result indicating whether or not a specified photographic paper area is included is a photographic paper area determination result indicating 1 if a photographic paper area of the specified size exists, and 0 if a photographic paper area of the specified size does not exist.

(4) Blank A blank is an image that does not include images such as characters or symbols (for example, an image of a blank document). The control unit 100 generates a histogram of density values (blank determination histogram) for all pixels included in the image data. At this time, if the frequency of occurrence of density values is extremely concentrated in a specific section, the control unit 100 determines that it is blank. For example, if no characters or figures appear in the image and the entire image is composed of white pixels or pixels with a predetermined density value, the frequency of occurrence of density values will be extremely concentrated in the specific section. Therefore, the control unit 100 determines that it is blank when the number of pixels included in a specific section relative to the number of pixels included in the image data is a predetermined ratio (for example, 90%) or more, or when the variance of the histogram is less than a predetermined value. Here, the ratio value and the range of the variance value used in determining that it is blank may be predetermined or may be set by the user. In this embodiment, the determination result indicating whether it is blank or not is a blank determination result indicating 1 if it is blank and 0 if it is not blank.

(5) Overall histogram (light original)
A thin document is an image in which characters or symbols are represented by pixels with high density values. For example, the control unit 100 generates a histogram of density values (overall determination histogram) targeting all pixels included in the image data. At this time, if the variance of the histogram is equal to or greater than a predetermined value and the average of the histogram is equal to or greater than a predetermined value, the control unit 100 determines that the document is a thin document when the variance of the overall determination histogram is greater than the variance value in determining that the image is a blank image (image of a blank document) and is smaller than a predetermined value, and further, the average density value is high. That is, if the density values of the image are dispersed to some extent but are generally high, the control unit 100 determines that the image is an image of a thin document. The variance value and the average value used when determining that the document is a thin document may be predetermined or may be set by the user. In this embodiment, the determination result indicating whether or not the document is a thin document is an overall histogram result indicating 1 if the document is a thin document and 0 if the document is not a thin document.

In this way, the control unit 100 obtains flag information (judgment result) indicating 0 or 1 for each judgment item.

Next, with reference to FIG. 14, the determination of the document type (mode) based on the determination results and whether or not background removal is performed will be described. T400 in FIG. 14 is a table showing combinations of determination results for each determination item. T410 in FIG. 14 is a table showing the document type (mode) corresponding to the combination of determination results shown in T400. T420 in FIG. 14 is a table showing whether or not background removal is performed corresponding to the document type (mode) shown in T410. Note that the tables shown in T400, T410, and T420 are, for example, stored in advance in storage unit 180.

The judgment items in T400 correspond to the judgment items shown in FIG. 13. The judgment results shown in FIG. 13 correspond to the flag information for each judgment item. Note that the judgment results for places that do not contain a value of 1 or 0 (specifically, places that contain a hyphen) are not taken into consideration.

As shown in T410, in this embodiment, the type of document is one of seven types: blank document, text document, photographic document, document containing printing (halftone dots), document containing faint text, document containing maps, etc., and document containing text, printing (halftone dots), and photographs.

In addition, in this embodiment, the control unit 100 determines whether the judgment results of the judgment items for the manuscript obtained in step S102 match any of the combinations of judgment results for each judgment item corresponding to the manuscript type, in the order of the manuscript types described above.

For example, the control unit 100 first determines whether the document acquired in step S102 is a blank document. Here, as shown in E400 of FIG. 14, the control unit 100 determines that a document in which the character area determination result, the dot area determination result, and the photographic paper area determination result are 0 and the blank determination result is 1 is a blank document, regardless of the determination result of the overall histogram result. If the document acquired in step S102 is not a blank document, the control unit 100 next determines whether the document is a text document.

In this embodiment, particularly when the result of the overall histogram is 1, the document is determined to be a document with faint text or a document containing a map, etc. As shown in E402 of FIG. 14, when the character area determination result and the overall histogram result are 1, and the halftone dot area determination result, photographic paper area determination result, and blank determination result are 0, the control unit 100 determines that the type of document is a document with faint text.

If the type of document is not a document with faint text, the control unit 100 determines whether the overall histogram result is 1, as shown at E404 in FIG. 14. If the overall histogram result is 1, the control unit 100 determines that the type of document is a document that includes a map or the like.

After determining the type of document in this manner, the control unit 100 determines whether the document type is a document containing faint text or a document containing a map, etc. (step S404).

If the type of document is a document containing faint text or a document containing a map, the control unit 100 cancels (skips) the automatic background removal process (step S404; Yes -> step S124). On the other hand, if the type of document is not a document containing faint text or a document containing a map, the control unit 100 executes the processes from step S106 to step S112 and step S122 (step S404; No).

As shown in FIG. 14, in addition to documents containing faint text and documents containing maps, the control unit 100 may also cancel (skip) the automatic background removal process when the document type is a photograph document or a document containing printing (halftone dots).

By performing the above-described process, the control unit 100 can prevent the output of an image with missing information due to background removal for documents that are generally faint, such as documents that contain faint text or maps.

According to this embodiment, background removal can be canceled for images where brightening the image of the document would result in loss of detail, such as documents that contain faint text or maps. This allows the image forming device to appropriately switch between performing and not performing background removal based on the type of document.

[5. Fifth embodiment]
Next, a fifth embodiment will be described. The fifth embodiment is an embodiment in which a user can set a processing method in the case where there is a loss of information and an allowable amount of loss of information (threshold value).

The functional configuration of the image forming device 12 in this embodiment is shown in FIG. 15. As shown in FIG. 15, the image forming device 12 differs from the image forming device 10 shown in FIG. 2 in that the control unit 100 also functions as a setting reception unit 112.

The setting reception unit 112 displays a setting screen on the display unit 140 for setting the processing method in the event of missing information and the amount of missing information to be tolerated. The setting reception unit 112 also receives operations from the user on the setting screen.

The processing method when there is missing information may be, for example, one of the following: canceling automatic background removal as shown in the first embodiment; optimizing background removal as shown in the second embodiment; or executing edge enhancement processing as shown in the third embodiment. The amount of information missing that is permitted (threshold) is set, for example, as a judgment ratio. Note that the number of judgment pixels or judgment score may be set as the threshold. The control unit 100 also executes the main processing based on the settings made via the setting screen.

Figure 16 is an example of a setting screen W500 that the setting reception unit 112 displays on the display unit 140. The setting screen W500 includes an area E500 that displays buttons that allow the user to select a processing method to be used when there is missing information, and an area E504 that displays a slider for setting a judgment ratio (threshold) as the amount of missing information that is acceptable. The setting screen W500 also includes an OK button B500 for confirming the settings, and a cancel button B502 for canceling the settings.

For example, in FIG. 16, if the threshold is set to 1%, this indicates that automatic background removal will not be performed when the ratio of the number of pixels in the difference image having a density value equal to or greater than the judgment density value to the total number of pixels in the image data is 1% or more. Similarly, in FIG. 16, if the threshold is set to 10%, this indicates that automatic background removal will not be performed when the ratio of the number of pixels in the difference image having a density value equal to or greater than the judgment density value to the total number of pixels in the image data is 10% or more. Therefore, the smaller the threshold, the more likely it is that automatic background removal will be canceled (skipped).

As shown in area E502, the setting screen W500 may include a button that allows the user to select canceling background removal for documents that include faint text or documents that include maps, etc. When the user selects to cancel background removal for documents that include faint text or documents that include maps, etc., the control unit 100 may execute the processing described in the fourth embodiment.

Also, the setting screen W500 is an example of a setting screen. Therefore, a screen different from the setting screen shown in the setting screen W500 may be displayed as the setting screen as long as it is a screen that allows the user to set the processing method in the event of missing information or the amount of missing information that is acceptable (threshold value).

According to this embodiment, the user can switch to the desired processing when information is lost due to background removal, and can set a threshold value (the amount of information loss that is acceptable) that serves as a criterion for determining whether or not to perform background removal processing.

[6. Modifications]
The present disclosure is not limited to the above-described embodiments, and various modifications are possible. In other words, the technical scope of the present disclosure also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the gist of the present disclosure.

Although the above-mentioned embodiments are described separately for ease of explanation, they may of course be combined to the extent technically possible.

In the embodiments, the programs that run on each device are programs that control the CPU and other devices (programs that make the computer function) to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and is then stored in various storage devices such as ROMs (Read Only Memory) and HDDs, and is read, modified, and written by the CPU as necessary.

The recording medium for storing the program may be any of semiconductor media (e.g., ROM, non-volatile memory cards, etc.), optical recording media, magneto-optical recording media (e.g., DVD (Digital Versatile Disc), MO (Magneto Optical Disc), MD (Mini Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disk), etc.), and magnetic recording media (e.g., magnetic tape, flexible disk, etc.). In addition, not only are the functions of the above-mentioned embodiments realized by executing the loaded program, but the functions of the present disclosure may also be realized by processing in cooperation with an operating system or other application programs, etc., based on the instructions of the program.

When distributing the program on the market, the program can be stored on a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is of course also included in the present disclosure.

REFERENCE SIGNS LIST 10, 12 Image forming apparatus 100 Control unit 102 Region separation unit 104 Image processing unit 106 Edge detection unit 108 Information loss determination unit 110 Output control unit 112 Setting reception unit 120 Image input unit 130 Image forming unit 140 Display unit 150 Operation unit 180 Storage unit 182 Image data storage area 184 Background removal table 190 Communication unit

Claims (7)

An image processing device including an acquisition unit that acquires first image data and a control unit,
The control unit is
determining a level of background removal based on the input first image data;
performing a background removal process on the first image data based on the level of background removal to generate second image data having been subjected to background removal;
Calculating a difference between the first image data and the second image data;
When the difference is equal to or greater than a threshold value, the first image data is output.
When the difference is less than a threshold value, the second image data is output. The control unit is
The image processing apparatus according to claim 1 , wherein the difference is calculated after edge processing is performed on each of the first image data and the second image data. The control unit is
Dividing the first image data into a background region and a region other than the background region;
3. The image processing apparatus according to claim 1, wherein a level of the background removal is determined based on the background region. The control unit is
when a difference between the first image data and the second image data is equal to or greater than a threshold value when background removal processing is performed based on a first background removal level, instead of outputting the second image data, generate third image data by performing background removal processing based on a second background removal level lower than the first background removal level;
Calculating a difference between the first image data and the third image data;
4. The image processing apparatus according to claim 1, wherein the third image data is output when the difference is less than a threshold value. The image processing device according to any one of claims 1 to 4, characterized in that the control unit controls displaying a screen that accepts an operation to set the threshold value. determining a level of background removal based on the input first image data;
performing a background removal process on the first image data based on the level of background removal to generate second image data that has been background removed;
calculating a difference between the first image data and the second image data;
outputting the first image data when the difference is equal to or greater than a threshold, and outputting the second image data when the difference is less than the threshold;
A control method comprising: On the computer,
A function of determining a level of background removal based on the input first image data;
a function of performing background removal processing on the first image data based on the level of background removal to generate second image data from which background has been removed;
A function of calculating a difference between the first image data and the second image data;
a function of outputting the first image data when the difference is equal to or greater than a threshold value, and outputting the second image data when the difference is less than the threshold value;
A program characterized by realizing the above.

Images