JP5883715B2 - Image processing LSI, image processing system, and image processing method - Google Patents

Description

  The present invention relates to an image processing LSI, an image processing system, and an image processing method, and can be suitably used for an image processing process for extracting a marking region from an image including a character string and a design with a marking.

  Document cameras that capture document images and display them to the public are used as communication tools for teachers and students, for example, in educational settings. In such a scene, there is a usage in which the teacher hides a part of the explanation of the manuscript image as a teaching material and presents it to the student, and makes the student think about the part. In that case, if the teacher paints the character string that he wants to hide on the manuscript image with a marker such as a highlighter pen, when the document camera is used to display it to the student, the marker-painted part is filled and the text cannot be read. It is desirable to be able to perform such processing, and to switch whether or not to perform such processing. Such processing can be realized by using an image processing process for extracting a marking region from an image including a character string with a marking.

  Patent Document 1 discloses a learning support apparatus that can simplify an input operation for creating a mask image. The character recognition unit recognizes the character display area, determines a mask area to be masked based on position designation information input separately, and creates a mask image.

  Patent Document 2 discloses a technique for extracting a marking region based on a difference in color components of image data from a document image in which a necessary portion is marked. A normal image and an image obtained by extracting only the color components used for marking are input, and the image data of the two surfaces are compared to extract the marking region and the character string designated by the marking.

  On the other hand, Patent Document 3 discloses a technique for separating a background layer and a character string from an image including a background and a character string such as a photographic image or gradation. Based on the maximum density difference, which is the difference between the minimum density value and the maximum density value in the m × n pixel block, the background area, the photographic area, and the character area are separated. Patent Document 4 discloses a technique for extracting a photograph area from a color document image in which characters and photographs are mixed.

JP 2011-257453 A JP 2002-189894 A JP 2010-010819 A JP 2009-071625 A

  The learning support device described in Patent Document 1 is configured to create a mask image that masks a character string designated by a teacher so that the student cannot read it, and the mask can be switched on and off. It is necessary to input the position to be masked. In Patent Document 2, a marking region and a character string designated by marking are extracted from a marked document image. If this is combined with the learning support device, designation of characters to be masked becomes easy. However, the technique described in Patent Document 2 does not assume that images other than characters such as photographs, pictures, and designs are mixed. Since a specific color component is extracted as a marking area, if an image other than characters such as a picture is mixed, an area having the same color component as the marking in the picture is extracted as the marking area. End up.

  On the other hand, according to the techniques described in Patent Documents 3 and 4, an image in which characters and photographs are mixed can be separated into a character area and a photograph area. However, since the technique described in Patent Document 3 does not assume that the character area is marked, if the character area is separated based on the maximum density difference, the marked character area can be classified as a photographic area. There is sex. For example, as a pre-processing of a technique for extracting a marking area from a document image described in Patent Document 2, even if the techniques described in Patent Document 3 are combined, if a marked character is erroneously determined to be a photographic area, In the first place, the marking area is excluded from the character area that is the target of extraction, and thus the marking area cannot be appropriately extracted. The technique described in Patent Document 4 can separate a character and a photo area even if there is a color character or a gradation background. However, the entire image is binarized and the background in the character area is ignored. Therefore, the marking in the character area cannot be extracted.

  Means for solving such problems will be described below, but other problems and novel features will become apparent from the description of the present specification and the accompanying drawings.

  According to one embodiment, it is as follows.

  That is, with respect to a document image including a character string and a pattern that are marked so as to cover a part of the character string with a color region different from the color of the character string, the luminance, color component, or A frequency distribution of luminance differences is obtained, and a character region is specified based on the frequency distribution.

  In order to specify the character area, a luminance difference that is the difference between the maximum luminance and the minimum luminance in the image block composed of a plurality of pixels is obtained for each pixel in the document image. The position of a pixel whose luminance difference is within or below a predetermined range is primarily determined as a character area, and based on the result, the pattern area and the character area are separated in units of area larger than the image block. Make a secondary decision.

  A frequency distribution of luminance, color component or luminance difference in the area specified as the character area is further obtained, and a marker area is extracted based on the frequency distribution.

  The effect obtained by the one embodiment will be briefly described as follows.

  That is, even from a document image including a pattern, a marker region can be extracted from a character string image partially marked.

FIG. 1 is a flowchart showing the contents of image processing according to the first embodiment. FIG. 2 is a block diagram showing an overview of the entire system in an example in which the image processing of the first embodiment is applied to a document camera. FIG. 3 is a block diagram illustrating a hardware configuration of the image processing unit according to the first embodiment. FIG. 4 is a schematic diagram for explaining an operation example of image processing in an example in which the image processing of the first embodiment is applied to a document camera. FIG. 5 is an explanatory diagram illustrating an example of the luminance difference Pd calculation process. FIG. 6 is an explanatory diagram of a method for obtaining the threshold value using the frequency distribution of the luminance difference Pd of the document image. FIG. 7 is an explanatory diagram illustrating an image of a determination result when the change amount distinguishing process is performed. FIG. 8 is an explanatory diagram showing the process of image area separation processing. FIG. 9 is an explanatory diagram illustrating a specific example of a dilation process and a dilation process of a binary image. FIG. 10 is an example of the frequency distribution (histogram) of the character area in which the horizontal axis represents luminance and the number of pixels included in the luminance range is counted on the vertical axis. FIG. 11 is an explanatory diagram showing a process of marker area extraction processing in a character area. FIG. 12 is an explanatory diagram showing the process of rectangular area generation processing (step 7) in the marker area. FIG. 13 is an explanatory diagram illustrating an example of the progress of image processing (first half) according to the second embodiment. FIG. 14 is an explanatory diagram illustrating an example of the progress of image processing (second half) according to the second embodiment. FIG. 15 is an explanatory diagram illustrating an example of the progress of image processing (first half) according to the third embodiment. FIG. 16 is an explanatory diagram illustrating an example of the progress of image processing (second half) according to the third embodiment. FIG. 17 is an explanatory diagram for determining the background and character color levels in the character area according to the fourth embodiment. FIG. 18 is an explanatory diagram illustrating a method for specifying the color of characters and markers according to the fourth embodiment. FIG. 19 is a configuration example of a document camera system according to the fifth embodiment.

1. First, an outline of a typical embodiment disclosed in the present application will be described. Reference numerals in the drawings referred to in parentheses in the outline description of the representative embodiments merely exemplify what are included in the concept of the components to which the reference numerals are attached.

[1] <Change amount distinguishing process + character / picture block specifying process (image processing LSI)>
A character string is extracted from an original image (101) including a character string and a pattern (13, 14) that are marked to cover a part of the character string (11) with an area (12) of a color different from the color of the character string. An image processing LSI (200) including an image processing unit (104) for extracting a marked marker region (15) is configured as follows.

  The image processing unit (104) is configured to, for each pixel in the document image with respect to the document image stored in the memory, a difference between a maximum luminance and a minimum luminance in an image block including a plurality of pixels including the pixel. Is obtained as a luminance difference in the pixel (step 2). A change amount distinguishing process in which a pixel position where the luminance difference is in a range from a first threshold value to a second threshold value is a pattern area, and a pixel position where the luminance difference is outside the range is a character area. Perform (Step 3). Based on the result of the primary determination by the change amount distinguishing process, a character / design block specifying process for secondary determination of the design area and the character area in units of areas larger than the image block is performed (step 4). Based on the first frequency distribution of luminance or color intensity or luminance difference in the area determined as the character area by the secondary determination (step 5), the marker area is extracted (step 6).

  As a result, even from a document image including a pattern, a marker region can be extracted from a character string image partially marked.

[2] <Change amount discrimination processing based on histogram of luminance difference of entire document image>
In Item 1, the image processing unit calculates a second frequency distribution of the luminance difference for the entire region of the document image, and a luminance difference of a peak including the luminance difference of 0 in the second frequency distribution. The luminance difference of the skirt on the side where the luminance difference is large is used as the first threshold value, and the luminance difference of the skirt on the side where the luminance difference of the peak where the luminance difference is close to the maximum value is small is obtained as the second threshold value.

  As a result, the character area can be accurately separated from the picture area even from a document image in which the picture and a character string with a part of the marking are mixed.

[3] <Learning of threshold value in change amount discrimination processing>
In Item 1 or 2, the image processing unit gives initial values to the first threshold value and the second threshold value, and then adaptively updates the first threshold value and the second threshold value for each input document image.

  Thereby, in change amount distinction processing, the suitable value of the 1st threshold and the 2nd threshold can be computed in a short time.

[4] <Character / pattern block identification processing by noise removal processing>
In Item 1, in the character / image block specifying process, a binary image is created by obtaining, for each pixel in the document image, a binary value indicating whether or not a result of the primary determination by the change amount distinguishing process is a character area. Then, noise removal processing is performed on the binary image.

  As a result, it is possible to perform the character / picture block specifying process targeting a wider area than the change amount distinguishing process.

[5] <Character / pattern block specifying process including expansion / contraction process>
In Item 4, the noise removal process includes one or more expansion processes for the binary image and the same number of contraction processes after the expansion process.

  As a result, the character / picture block specifying process can be performed by a simple image signal process for a larger area than the change amount distinguishing process.

[6] <Histogram processing of character area performed for each color component>
In Item 1, the first frequency distribution is a frequency distribution of color intensity, and the marker region is extracted based on the first frequency distribution.

  As a result, the marker region can be accurately extracted even when the characters are marked with various darkness and colors, and markers of a plurality of types of colors.

[7] <Histogram processing for each color component + continuous number of pixels of the same color>
In Item 6, the marker region is extracted based on the number of pixels in which the pixels in the same color intensity section of the first frequency distribution are continuous in the document image.

  Thereby, in a character area analysis process, a character and a marker can be distinguished correctly.

[8] <Character area analysis processing including noise removal processing>
In Item 1, 6, or 7, the image processing unit is determined to be a marker region for each pixel based on the luminance, the luminance difference, or the color intensity range occupied by a peak in the first frequency distribution. A binary image is created depending on whether or not it is, and a noise removal process is performed on the binary image to extract the marker region.

  Thereby, the marker area in the character area can be extracted more accurately.

[9] <Generate rectangular area covering marker area>
In one of Items 1 to 8, a rectangular region surrounding the outer periphery of the marker region is obtained, and among the image data of the original image, the luminance or color or luminance and color of the image data of the rectangular region is changed. A display image (16) is output.

  As a result, a rectangular region covering the extracted marker region can be generated and processed so that the marked characters cannot be read, or conversely highlighted.

[10] <Marker color registration area>
In one of Items 1 to 9, a marker registration area (90) is provided in the document image, and a third frequency distribution of luminance or luminance difference or color intensity in the marker registration area is calculated, Based on the third frequency distribution, a range of luminance or luminance difference or color intensity to be determined as a marker region is obtained, and the marker region is extracted from the document image.

  Thereby, the marker area can be extracted more easily and reliably, the color of the specific color can be extracted from the original document mixed with complex colors, and the versatility becomes higher.

[11] <Following processing of marker area to document movement>
In one of Items 1 to 10, each frame constituting a plurality of frames taken in time series is defined as the original image. The image processing unit extracts a motion vector by comparing a document image of a frame from which the marker region has been extracted with a document image of a subsequent frame that has been captured, and a marker region in which position information is corrected by the motion vector. The marker area in the document image of the subsequent frame is used.

  Thereby, the extraction result of one marker area can be used over a plurality of frames without extracting a marker area for every frame.

[12] <Change amount distinguishing process + character / pattern block specifying process (image processing system)>
A character string is extracted from an original image (101) including a character string and a pattern (13, 14) that are marked to cover a part of the character string (11) with an area (12) of a color different from the color of the character string. An image processing system for extracting a marked marker region (15), which is configured as follows.

  For the original image, for each pixel in the original image, a difference between the maximum luminance and the minimum luminance in an image block composed of a plurality of pixels including the pixel is obtained as a luminance difference in the pixels (step 2). A change amount distinguishing process in which a pixel position where the luminance difference is in a range from a first threshold value to a second threshold value is a pattern area, and a pixel position where the luminance difference is outside the range is a character area. Perform (Step 3). Based on the result of the primary determination by the change amount distinguishing process, a character / pattern block specifying process is performed in which the pattern area and the character area are secondarily determined in units of areas larger than the image block (step 4). Based on the first frequency distribution of luminance or color intensity or luminance difference in the area determined as the character area by the secondary determination (step 5), the marker area is extracted (step 6).

  As a result, even from a document image including a pattern, a marker region can be extracted from a character string image partially marked.

[13] <Change amount discrimination process based on histogram of luminance difference of entire document image>
In Item 12, the second frequency distribution of the luminance difference for the entire region of the original image is calculated, and in the second frequency distribution, the peak of the peak including the luminance difference of 0 is larger. The luminance difference is set as the first threshold value, and the luminance difference at the bottom of the peak where the luminance difference is close to the maximum value is determined as the second threshold value.

  As a result, the character area can be accurately separated from the picture area even from a document image in which the picture and a character string with a part of the marking are mixed.

[14] <Character / pattern block identification processing by noise removal processing>
In the item 12, the character / image block specifying process creates a binary image by obtaining a binary value indicating whether or not the result of the primary determination by the change amount distinguishing process is a character area for each pixel in the document image. , Including noise removal processing on the binary image.

  As a result, it is possible to perform the character / picture block specifying process targeting a wider area than the change amount distinguishing process.

[15] <Character / pattern block specifying process including expansion / contraction process>
In item 14, the noise removal processing includes at least one expansion processing on the binary image and the same number of contraction processing after the expansion processing.

  As a result, the character / picture block specifying process can be performed by a simple image signal process for a larger area than the change amount distinguishing process.

[16] <Character area analysis processing including noise removal processing>
In item 12, whether or not each pixel is determined to be a marker region based on the luminance or the luminance difference or the color intensity range occupied by a peak in the first frequency distribution of luminance or color intensity or luminance difference A binary image is created, and noise removal processing is performed on the binary image to extract the marker region.

  Thereby, the marker area in the character area can be extracted more accurately.

[17] <Document camera>
In one of of claim 12 to claim 16, wherein the image processing system further comprises a display device camera and a memory (203) (105, 106). A document is imaged and input by the camera, stored in the memory, color image processing (206) is performed on the captured image data to form the document image, and the extracted image data of the document image is extracted. A display image in which the brightness or color of the image data in the marker area or the brightness and color is changed is displayed on the display device.

  Thus, even from a document image including a pattern, a marker region is extracted from a character string image partially marked, and desired signal processing is performed on the image data of the extracted marker region. A document camera that can be displayed can be provided.

[18] <Teacher display and student display>
In Item 17, the display device is a first display device (106), and the image processing system further includes a second display device (105), and displays the original image on the second display device.

  Thereby, it is possible to display different images for the teacher and the student.

[19] <Change amount distinguishing process + character / pattern block specifying process (image processing method)>
A character string is extracted from an original image (101) including a character string and a pattern (13, 14) that are marked to cover a part of the character string (11) with an area (12) of a color different from the color of the character string. An image processing method for extracting a marked marker region (15), which is configured as follows.

  For the original image, for each pixel in the original image, a difference between the maximum luminance and the minimum luminance in an image block composed of a plurality of pixels including the pixel is obtained as a luminance difference in the pixels (step 2). A change amount distinguishing process in which a pixel position where the luminance difference is in a range from a first threshold value to a second threshold value is a pattern area, and a pixel position where the luminance difference is outside the range is a character area. Perform (Step 3). Based on the result of the primary determination by the change amount distinguishing process, a character / pattern block specifying process is performed in which a pattern area and a character area are secondarily determined for each area larger than the image block (step 4). Based on the frequency distribution of luminance or color intensity or luminance difference in the character region (step 5), the marker region is extracted (step 6).

  As a result, even from a document image including a pattern, a marker region can be extracted from a character string image partially marked.

[20] <Change amount discrimination processing based on histogram of luminance difference of entire document image>
In Item 19, a second frequency distribution of the luminance difference for the entire region of the original image is calculated, and in the second frequency distribution, the peak of the peak including the luminance difference of 0 on the side where the luminance difference is large is calculated. The luminance difference is set as the first threshold value, and the luminance difference at the bottom of the peak where the luminance difference is close to the maximum value is determined as the second threshold value.

  As a result, the character area can be accurately separated from the picture area even from a document image in which the picture and a character string with a part of the marking are mixed.

[21] <Character / pattern block identification processing by noise removal processing>
In item 19, the character picture block specifying process creates a binary image by obtaining a binary value indicating whether or not the result of the primary determination by the change amount distinguishing process is a character area for each pixel in the document image. Then, noise removal processing is performed on the binary image.

  As a result, it is possible to perform the character / picture block specifying process targeting a wider area than the change amount distinguishing process.

[22] <Character / pattern block specifying process including expansion / contraction process>
In Item 21, the noise removal processing includes at least one expansion processing on the binary image and the same number of contraction processing after the expansion processing.

  As a result, the character / picture block specifying process can be performed by a simple image signal process for a larger area than the change amount distinguishing process.

[23] <Histogram processing of character area performed for each color component>
In Item 19, the first frequency distribution is a frequency distribution of color intensity, and the marker region is extracted based on the first frequency distribution.

  As a result, the marker region can be accurately extracted even when the characters are marked with various darkness and colors, and markers of a plurality of types of colors.

[24] <Histogram processing for each color component + continuous number of pixels of the same color>
In item 23, the marker region is extracted based on the number of pixels in which the pixels in the same color intensity section of the first frequency distribution are continuous in the document image.

  Thereby, in a character area analysis process, a character and a marker can be distinguished correctly.

[25] <Character area analysis processing including noise removal processing>
In item 19, whether or not each pixel is determined to be a marker region based on the luminance, the luminance difference, or the color intensity range occupied by a peak in the first frequency distribution of luminance, color intensity, or luminance difference A binary image is created, and noise removal processing is performed on the binary image to extract the marker region.

  Thereby, the marker area in the character area can be extracted more accurately.

[26] <Image area separation by histogram of whole document image + marker area extraction by histogram of character area (image processing system)>
A character string is extracted from an original image (101) including a character string and a pattern (13, 14) that are marked to cover a part of the character string (11) with an area (12) of a color different from the color of the character string. An image processing system for extracting a marked marker region (15), which is configured as follows.

  A second frequency distribution of luminance or color intensity or luminance difference in the entire area of the original image is obtained for the original image, and a character area in the original image is specified based on the second frequency distribution. (Steps 2, 3, 4). The marker area is extracted based on the first frequency distribution of luminance or color intensity or luminance difference in the area identified as the character area (steps 5 and 6).

  As a result, even from a document image including a pattern, a marker region can be extracted from a character string image partially marked.

[27] <Change amount distinction process + image area separation by character / pattern block identification process>
Judgment of the change amount distinguishing process for a binary image based on a result of determination by the change amount distinguishing process and a change result distinguishing process for judging whether the region is a character area or a picture area in units of pixels in the entire area of the original image The character area is specified by character / pattern block specifying processing for determining whether the area is wider than the unit or not.

  As a result, even if the brightness and color in the marker area is the same as the brightness and color in the pattern area, the character area and the pattern area are accurately separated to identify the character area, and the marker area is extracted accurately. Can do.

[28] <Change amount discrimination processing based on histogram of luminance difference of entire document image>
In the item 26 or the item 27, in the second frequency distribution, the luminance difference at the skirt on the side where the luminance difference of the peak including the luminance difference is 0 is set as the first threshold value, and the luminance difference near the maximum value is set. The luminance difference at the skirt on the side where the luminance difference of the peak is small is obtained as the second threshold value.

  As a result, the character area can be accurately separated from the picture area even from a document image in which the picture and a character string with a part of the marking are mixed.

2. Details of Embodiments Embodiments will be further described in detail.

Embodiment 1
The first embodiment will be described. FIG. 1 is a flowchart showing the contents of image processing according to the first embodiment, FIG. 2 is a block diagram showing an overview of the entire system in an example in which the image processing of the first embodiment is applied to a document camera, and FIG. FIG. 4 is a block diagram illustrating a hardware configuration of a processing unit. FIG. 4 is a schematic diagram illustrating an example of image processing operation in an example in which the image processing according to the first exemplary embodiment is applied to a document camera.

  In FIG. 2, a document 101 that is a subject is imaged through an optical system 102 such as a lens, converted into an electrical signal by an image sensor 103, and subjected to necessary signal processing by an image processing unit 104. B106 is displayed. As the image sensor 103, for example, a CCD or a CMOS sensor is generally used. The image processing unit 104 performs various image corrections (clamping, shading correction, defective pixel correction, color correction, γ correction, and the like) based on the electrical signal from the image sensor 103. The corrected video is output to the teacher monitor A105. On the student monitor B106, an image subjected to the image processing according to the first embodiment is displayed after correction similar to that for the teacher.

  FIG. 3 is a block diagram illustrating an example of a hardware configuration of the image processing unit 104 in the system illustrated in FIG. A CPU 201, a nonvolatile memory 202, a memory 203 such as a DRAM, an imaging system drive control unit 204, a sensor image input unit 205, a color signal processing unit 206, an image recognition processing unit 207, and a display image processing unit 208 are connected to the bus 209. Connected through. If the portion surrounded by the broken line is constituted by one-chip LSI 200, it is not necessary to employ a complicated manufacturing process in which a nonvolatile memory, a DRAM, or the like is mixed. On the other hand, when the non-volatile memory 202 and the memory 203 such as a DRAM are mixedly mounted on the same chip or the same module, the mounting area and power consumption can be suppressed.

The CPU 201 controls the overall flow of the LSI 200 and performs calculations such as calculating parameters for the next process based on the calculation results of the signal processing units 204 to 208 in the LSI 200. The non-volatile memory 202 is a memory for storing a program of the CPU 201, initial parameters, a constant data table, and the like. The memory 203 is mainly used as an image memory. The imaging system drive control unit 204 is a control unit that performs lens zoom control and focus control, an image sensor operation mode, and other timing control. Sensor image input unit 205 is a sensor image input unit for capturing pixel data from the image sensor 103, an image signal input from the image sensor 103 arranged into image data for each pixel is stored in memory 203. The color signal processing control unit 206 reads the data stored in the memory 203 by the sensor image input unit 205 and performs various corrections such as color (clamping, shading correction, defective pixel correction, color correction, γ correction) on the video signal. Etc.).

The image recognition processing unit 207 is a unit that performs processing in accordance with various uses based on an input image, and mainly performs image recognition. Since high programmability is required for image recognition, the image recognition processing unit 207 is preferably realized by a high-performance CPU or a faster programmable device to maintain real-time processing. . In particular, due to the high parallelism of the processes, having an architecture such as SIMD (Single Instruction Multiple Dataflow), it may be employed parallel processor. The image recognition processing unit 207 can measure a specific color, object shape, motion vector, and the like in the image.

  The display image processing unit 208 adjusts the image resolution to the image size that can be displayed on the monitor, and further appropriately performs processing such as overlaying the character image and outputs the result.

  Each signal processing unit 204 to 208 reads out the image data stored in the memory 203, processes it, and stores the result in the memory 203. Since sensor image input, color signal processing, image recognition processing, and display image processing are sequentially performed on input images, dedicated processing is performed between the processing units in order to reduce traffic on the bus 209. You may connect with the bus. Each of the signal processing units 204 to 208 performs arithmetic processing with dedicated hardware, but the CPU 201 is changed to one with high arithmetic performance or replaced with another processor, and part or all of those processing is realized by software. You may do it. For example, part or all of processing such as sensor image input, color signal processing, image recognition processing, and display image processing other than image recognition may be executed by a programmable unit constituting the image recognition processing unit 207.

  The operation of the image processing unit 104 shown in FIG. 3 will be described in more detail.

  After the power is turned on, the CPU 201 is booted and operable by a program stored in the nonvolatile memory 202. When the CPU 201 completes the initial setting of all the modules, the CPU 201 proceeds to a sequence that enables video input. First, the imaging system drive control unit 204 is instructed to start driving. Based on this instruction, the imaging system drive control unit 204 performs initial setting of the mode for the image sensor 103, then issues various timing signals such as a horizontal synchronization signal and a vertical synchronization signal, and video input synchronized with the timing signal. Get a signal.

  The image input signal is captured by the sensor image input unit 205 in the format transmitted from the sensor, converted into a pixel order (pixel arrangement), format, and timing that can be handled by this system, and temporarily stored in the memory 203. Store. The video input signal may be temporarily stored in the memory 203 in the format, the data may be read out again, and converted into the pixel order (pixel arrangement), format, and timing by the sensor image input unit 205. Although the traffic that is the amount of data transferred on the bus 209 increases, the time restriction on the processing of the sensor image input unit 205 is relaxed.

  Thereafter, the color signal processing unit 206 performs various conversions on the image signal converted into a predetermined format. For example, clamp processing to fix a predetermined part of the waveform of the input image signal to a constant voltage, shading correction to correct the amount of light around the lens, defective pixel correction to extract and correct defective pixels, Color correction for adjusting the hue, γ correction for correcting the saturation and brightness of the image displayed on the display, and the like. For example, the color signal processing unit 206 includes hardware that performs real-time processing of a video signal including these signal processing. These signal processing may be executed by software. The video signal processed by the color signal processing unit 206 is temporarily stored in the image memory 203.

  The stored image is then input to the image recognition processing unit 207 to perform various recognition processes. The image recognition processing unit 207 can output an image whose target output is corrected, or a parameter representing a specific amount, and the data is stored in the memory 203 again.

  The display image processing unit 208 performs control to output the image stored in the image memory 203 to the monitor as it is, or to superimpose a character for painting a specific area or character string, and outputs a video. The video output here can be composed of a plurality of video outputs in order to output a video according to the purpose.

  FIG. 4 is a schematic diagram for explaining an operation example of image processing in an example in which the image processing of Embodiment 1 is applied to a document camera. The document 101 includes an explanatory note 11, a photograph 13, a natural image 14, and the like as shown in FIG. Here, the explanatory note 11 includes a marker region 12 partially marked with a highlighter pen, for example. In the image processing according to the first embodiment, the original image 101 is read, the marker area 15 is extracted as shown in (b), and the rectangular area 16 in which the extracted marker area is painted black is generated, and (c) As shown in FIG. 4, the image is superimposed on the original image 101 and displayed for students. In order to encourage students to think, the teacher displays the important items in a hidden state at first, and displays them later to make a strong impression on the students. The image processing according to the first embodiment is, for example, image processing that can be applied to such image processing, and includes a character string partially marked and a document image including a picture such as a photograph or a natural image From 101, a marker area marking a character string is extracted.

  FIG. 1 is a flowchart showing the contents of image processing according to the first embodiment. For example, in order to realize the function of the document camera shown in FIG. 4, the image processing unit 104 in FIG. It is executed by the recognition processing unit 207. When a document image is read (step 1), a luminance difference Pd for each pixel is calculated (step 2), and a change amount distinguishing process (step 3) is executed based on this. Here, the original image refers not to an image itself input from the sensor but to an image that has been appropriately converted by the sensor image input unit 205 and is subjected to appropriate correction processing by the color signal processing unit 207. It may be an image. As will be described later, the luminance difference is a parameter representing a spatial (planar direction) change in luminance in the image, and based on this, each pixel is located in the character area or located in the picture area. It is determined whether or not. Since this is a microscopic determination for each pixel, erroneous determination is likely to occur, and is regarded as primary determination. Next, a character design block specifying process (step 4) is performed. This is a secondary determination in which the primary determination result is reviewed macroscopically. While the change amount distinguishing process (step 3) is determined for each pixel, it is determined in units of larger areas (wide areas) and separated into character areas and picture areas. Next, a character area analysis process (step 5) is performed. In the area determined to be a character area, a marked character is distinguished from an unmarked character, and a marker area is extracted (step 6). Thereafter, an image of a rectangular area covering the marker area is generated (step 7), and is displayed so as to be superimposed on the original image (step 8). As shown in FIG. A hidden image can be generated.

  If the marker region can be appropriately extracted, it can be applied to other processes using the marker region thereafter. For example, the character color is converted to the same color as the marker color to hide the character string, the marker area is blinked and highlighted instead of hiding the character, and character recognition is performed on the marked character string. It can also be handled as character or semantic data.

  Of the image area separation process for separating the document image into the character area and the picture area, the luminance difference Pd calculation process (step 2) will be described in more detail. An image block composed of a plurality of pixels is determined for the document image, and a brightness difference that is the difference between the maximum brightness and the minimum brightness in the image block is determined for each pixel in the document image. For example, the maximum luminance Pmax and the minimum luminance Pmin in an m × n pixel rectangular image block are obtained using an m × n pixel maximum / minimum value filter, and the difference is calculated as a luminance difference Pd = Pmax−Pmin. And the luminance difference at the center pixel of the m × n image block.

FIG. 5 is an explanatory diagram illustrating an example of the luminance difference Pd calculation process. (A) is an example of a 3 × 3 matrix. The central pixel P4 is the pixel of interest, and for nine pixels including the surrounding pixels P0 to P3 and P5 to P8, the maximum luminance is Pmax (P0: P8) and the minimum luminance is Pmin (from the luminance information of each pixel (nine)). P0: P8) is calculated. Use a comparator for comparison. The brightness difference Pd = Pmax (P0: P8) −Pmin (P0: P8) is calculated from the calculated Pmax (P0: P8) and Pmin (P0: P8). As indicated by arrows in (b), a raster scan is performed in which the target pixel is shifted in the horizontal direction for each pixel for the entire original image, and the luminance difference for each pixel is calculated in the same manner as described above. . (B) is schematically drawn by superimposing a part of a character to be ideally displayed and a grid-like pixel, but image data has luminance (Y) and color (Cr, Cb or I, Q for each pixel. Or the intensity component information of the three primary colors of RGB. In general, the luminance difference calculation process targets luminance information (Y). However, depending on the character color, background color, marker color, etc. in the character area, intensity component information for each color may be used. it can. Further, when the peripheral portion of the original image and the pixel of interest, but also a part of pixels among the peripheral pixels m × n image block can not be obtained luminance information has become out of the range of the original image, Compensation may be performed by a known end process or the maximum value / minimum value may be obtained in consideration of only pixels for which luminance information is obtained.

For the image area separation processing, next, change amount discrimination processing (step 3) is performed on the result of the luminance difference Pd calculation processing (step 2). In general, character areas have the following characteristics. That is,
Feature A: The luminance between adjacent pixels on the character is almost the same, and the luminance difference Pd is 0 or very small.
Feature B: The luminance between adjacent pixels in the background of the character is almost the same, and the luminance difference Pd is 0 or very small.
Feature C: There is a large luminance difference at the boundary between the character and the background.

  On the other hand, in the picture area, the luminance of adjacent pixels tends to change gently, and the luminance difference is not large.

Based on the above characteristics, two threshold levels TH1 and TH2 for the luminance difference are defined, and the change amount distinguishing process is performed according to the following evaluation conditions. TH1 is a threshold level for determining a certain color (corresponding to either character or background), and TH2 is a threshold level for distinguishing between a character and the background.
Condition A: When Pd ≦ TH1, the character area is determined.
Condition B: When TH1 <Pd ≦ TH2, the pattern area is determined.
Condition C: If TH2 <Pd, it is determined as a character area.

  The condition A is either the background of the character area or the area of the character, and the condition C is a boundary area between the character and the background of the character area.

  FIG. 6 is an explanatory diagram of a method for obtaining the threshold value using the frequency distribution of the luminance difference Pd of the document image. A frequency distribution (histogram) of the luminance difference Pd is obtained for the entire document image. The abscissa represents the difference in luminance for each appropriate range, and the ordinate represents the number of pixels included in the range. Although it becomes a strip-like graph, the feature of an image can be extracted by observing this envelope. If the character string is drawn in black on a white background, black is the minimum luminance value and white is the maximum luminance value, so the peak near Pd1 with the smallest luminance difference is the white background between characters or the background area. It is considered that there is a luminance difference between them, and the peak in the vicinity of Pd8 having the largest luminance difference is considered to be a boundary region between the character and the background. A gentle peak centered on the intermediate luminance difference Pd4 is considered to be a pattern region. This is because in natural images and photographs, the frequency of extremely large or very small luminance differences is small, and the luminance difference between pixels is generally widely distributed in a relatively intermediate value. In a manuscript image composed of a character area consisting only of a character string without marking and a pattern area of a natural image, a frequency distribution such as an envelope 81 including two peaks in the vicinity of the luminance differences Pd1 and Pd8 is obtained.

  When the character area includes a marked character string, when the brightness and color of the marker are determined, a peak appears in the brightness difference between the marker and the character and the brightness difference between the marker and the background determined accordingly. Since the brightness and color of the marker are constant, the brightness difference between the marker and the character and the brightness difference between the marker and the background take specific values and are not widely distributed. Since the brightness difference between the marker area and the character is smaller than the brightness difference between the white background and the character, a new peak appears on the side lower than Pd8. When the marker color is close to the black character, a peak 82 appears near Pd2 where the luminance difference is small, and when the marker color is light and close to a white background, a peak 83 appears near Pd6, for example.

  TH1 is a threshold level for determining a certain color (equivalent to either character or background), and TH2 is a threshold level for distinguishing between a character and the background, so the range from Pd2 to Pd3 is the center. In particular, TH1 should be determined in the vicinity of Pd3, and TH2 should be determined in the vicinity of Pd5 around the range from Pd5 to Pd6. In the image recognition process, the threshold values TH1 and TH2 can be obtained by obtaining the frequency distribution after the luminance difference Pd calculation process (step 2) or simultaneously with this process and analyzing the characteristics of the frequency distribution. For example, TH1 can be determined from the bottom of the envelope in the expected range of the threshold, the range from Pd2 to the vicinity of Pd3, and TH2 can be determined from the bottom of the envelope in the range from Pd5 to the vicinity of Pd7.

  FIG. 7 is an explanatory diagram illustrating an image of a determination result when the change amount distinguishing process is performed.

  (A) is the figure which expanded the character area 11. FIG. The character without a marker is shown on the upper side, and the character 12 marked on the lower side is shown. One grid represents one pixel, and a character having a boundary between the character and the background in the pixel actually has intermediate luminance, but in FIG. 7, the character itself is displayed in a pseudo manner. (B) shows the pixels determined to be the character region 20 corresponding to the above conditions A and C as a result of performing the above-described change amount distinguishing process on the image of (a), with diagonal grid-like hatching. The pixels determined to be the pattern area 21 corresponding to the condition B are represented by diagonally upward hatching. (C) is an enlarged view of the pattern area. Like (b), (d) corresponds to the above condition A and condition C as a result of performing the change amount distinguishing process on the image of (c). Pixels 20 determined to be character regions are represented by diagonal grid-like hatching, and pixels 21 determined to be a pattern region corresponding to the condition B are represented by diagonal hatching rising to the right.

  In the character area, there are many pixels 20 that meet the conditions A and C and are determined to be character areas, but particularly in the character area that has been marked, there are many pixels 21 that are determined to be a picture area. In the marked character region, the luminance difference between the background and the character is smaller than the luminance difference in the character region where the background is not marked and the background is white. On the other hand, even in the pattern area, it is determined as the character area 20 in an area where the change in shading is abrupt. Although not illustrated, in an area where the luminance difference is smaller and less than TH1, the pattern area is also determined as a character area.

  The erroneous determination as described above occurs in the change amount distinguishing process (step 3) in order to determine whether the local area of each pixel is a character or a picture. This is regarded as primary determination. A character / pattern block specifying process (step 4) is performed on the primary determination result. This is a secondary determination for determining whether the area is a character area or a picture area in units of an area (wide area) larger than the change amount distinguishing process performed for the local area for each pixel.

  FIG. 8 is an explanatory diagram showing the process of image area separation processing. (A) is the original image 101 before the change amount distinction process (step 3) is executed. (B) is a determination result of the change amount distinguishing process (step 3), and (c) is a determination result of further executing the character / picture block specifying process (step 4). In the change amount distinguishing process (step 3), in (b), the region 20 determined as the character region is included in the region 20 determined as the character region, and the region 21 determined as the pattern region is included. Also included is a minute area 20 determined as a character area. In (c), each large area is separated into a character area and a pattern area.

  Specifically, the character / pattern block specifying process (step 4) can perform, for example, an expansion process and a contraction process of a binary image. Noise can be removed by performing expansion processing a plurality of times (for a number of pixels) and then performing a contraction process for the same number of times (the same number of pixels). FIG. 9 is an explanatory diagram illustrating a specific example of a dilation process and a dilation process of a binary image. Since the primary determination result by the change amount distinguishing process (step 3) is represented by binary values of a character area or a picture area for each pixel, it can be handled as a binary image. As shown in (a), an example in which a minute region 23 due to noise indicated by a 3 × 3 pixel grid-like hatching is included in a region 22 that has been subjected to right-down hatching will be described. If the area 22 with the right-down hatching is expanded by one pixel, it expands by one pixel in each direction and expands by one pixel inside, so that the minute area 23 indicated by the grid-like hatching is eroded. Thus, a region 24 of 1 × 1 pixel is obtained. Further, when the two-pixel expansion process is performed from one pixel and (a), the minute area indicated by the grid-like hatching is eroded and disappears (25). From here, even if one pixel reduction processing is performed, the minute area indicated by the lattice-like hatching that has once disappeared is not reproduced, and as shown in (d), the area is reduced by one pixel from the periphery, Further, when one pixel reduction processing is performed, as shown in (e), it is possible to generate an image in which the inner 3 × 3 pixel minute region, that is, the noise region has disappeared in the region of the same size as (a). When performing binary image expansion processing and contraction processing in the character / picture block specifying process (step 4), the character area is contracted after expansion processing, and the pattern region is contracted after expansion processing. Expansion / contraction processing is performed for each of the pattern areas. Similar effects can be obtained by majority processing or filter processing instead of expansion / contraction processing.

The character area analysis process (step 5) will be described. In a region determined as a character region by the character / pattern block specifying process (step 4), a frequency distribution (histogram) of luminance, color intensity, or luminance difference is obtained.
FIG. 10 shows an example of a histogram in which the horizontal axis represents luminance, and the number of pixels included in the luminance range is counted on the vertical axis. At this time, three luminance values of background, character, and marker are assumed. In general, in a histogram of pixel values in a character area, the number of pixels corresponding to a background color is the largest, and the number of pixels corresponding to a character color and a marker appears as a peak. (A) is an example of a histogram when there is no marker, and a peak of the number of pixels of the character color appears in the luminance value Y2, and a peak of the number of pixels of the background color appears in the luminance value Y6. (B) is an example of a histogram when there is a marker. In addition to the tendency of (a), a peak of the number of pixels appears in the luminance value Y4 of the marker.

  If the original image before the marker entry is taken, the histogram (a) of the image before the marker entry and the histogram (b) where the marker is entered are calculated, and the histograms of (a) and (b) are compared. By performing the above, the brightness of the marker can be extracted. Thereby, the brightness | luminance of a marker can be extracted correctly.

  On the other hand, when only the histogram (b) in which the marker has already been entered is obtained, the marker region can be extracted by extracting three peaks. That is, it is possible to determine a pixel in the range of the luminance values Y3 to Y5 as a pixel in the marker area. If the area to be analyzed contains a pattern, there may be many pixels in the range of luminance values Y3 to Y5. In this step, the character area after image area separation is analyzed. Therefore, the pixels included in the range of the luminance values Y3 to Y5 can be determined as pixels in the marker area. Even if the image area separation is insufficient and a pattern is included as an error, the number of pixels is significantly smaller than that before the image area separation, so there is almost no influence on the determination of the marker area.

  When the feature of the marker area cannot be extracted efficiently with the histogram of luminance, the marker area can be extracted with the histogram of color components. This is because the marker is often a characteristic color. The color component histogram can be determined by obtaining a histogram of Cr and Cb, which are color difference components other than Y. Furthermore, it is possible to extract with higher accuracy by performing analysis with each color, for example, analysis with an R (red) component, analysis with a G (green) component, and analysis with a B (blue) component. Since there is a linear relationship between YCrCb and RGB, it can be easily converted into each other by a 3-by-3 matrix operation, so if it cannot be extracted with luminance Y and color difference components Cr and Cb, it will be converted to RGB Then, a histogram can be obtained.

  In addition, a histogram for the luminance difference Pd calculated in step 2 can be obtained to identify a characteristic luminance difference in the marker region, and the marker region can be extracted based on the specified luminance difference. Furthermore, even if it is possible to identify a luminance difference characteristic for the marker region even from the histogram of the luminance difference of the entire document image obtained to determine the threshold value of the change amount distinction process (step 3), it is necessary to again. The marker area can be extracted without obtaining a histogram of only the character area. In general, the brightness difference between the marker and the character, and the marker and the background is an intermediate value, making it difficult to distinguish the brightness difference from the pattern area. Since the influence is small, the histogram calculation process only in the character area can be omitted.

  The marker area extraction process (step 6) will be described. The conversion process is performed again with the brightness level in the character area as Yin, the left end (minimum) brightness level of the marker brightness as ML, and the right end (maximum) brightness level as MR. It is determined whether or not ML ≦ Yin ≦ MR is satisfied for the pixels in the entire character area, and a binary image representing the result in binary is generated. FIG. 11 is an explanatory diagram showing a process of marker area extraction processing in a character area. In the example of FIG. 10B, when it is determined that the peak centered on the luminance value Y4 is the marker color, the above processing is performed with ML = Y3 and MR = Y5.

  FIG. 11A shows the result of extracting the marker portion in the character area by the above processing. A region 26 determined to be a marker region is represented by thick hatching at the lower right, and a region 27 determined as a background having no other characters or markers is represented by thin hatching at the upper right. Since it is determined for each pixel whether or not the above-described determination formula is satisfied, a noise component for each pixel is included in the generated binary image. This is because there may be a noise level that happens to be a marker in addition to the original marker. As a countermeasure, noise countermeasures are taken for the binary image (a) through a noise removal filter. For example, in a 3 × 3 matrix, a median filter that changes the value from 9 pixels to an intermediate value, a filter that takes a majority vote from 9 pixels and replaces the central pixel with a larger number can be considered. Thereby, it is possible to extract the marker area from which noise is removed as shown in FIG.

  Further, by performing the above processing on the entire document image, a plurality of marker regions can be extracted, and the coordinates of each marker region in the document image can be calculated.

  A rectangular area generation process (step 7) in the marker area will be described. FIG. 12 is an explanatory diagram showing the process of rectangular area generation processing (step 7) in the marker area. (A) is an image corresponding to the marker region extracted from the coordinates specified in the marker region extraction process (step 6). In the specified marker area, a rectangular area for covering up the internal characters is generated. In (b), in the pixel to be the marker area 15, the upper, lower, left, and right sides are obtained from the upper, lower, left, and right vertices, and the rectangular areas surrounded by the upper, lower, left, and right sides are all set to the same color. The hue at this time is the same as the character color included. This character color can be specified by analyzing the histogram of FIG. In order to obtain more accuracy, confirmation may be performed by creating a histogram again in the extracted marker region. (C) is the one in which the entire marker area is filled with the specified color.

  These descriptions have been made for one marker area, but a plurality of marker areas may be executed simultaneously. Usually, since the number of markers is limited, a limited number can be realized by hardware. When processing by software, a plurality of marker areas are sequentially processed. Processing time is increased, but no dedicated hardware is required.

  The rectangular area overlapping process (step 8) will be described. After the rectangular area is determined, this rectangular data is displayed superimposed on the normal output screen. The data of the rectangular area generated by the rectangular area generating process (step 7) in the marker area is stored in the image memory 203 as a character image. On the other hand, a normal screen on which an original image is projected as it is is also stored in the image memory 203, and data of the normal screen and the rectangular area screen is sent to the display image processing unit 208. The display image processing unit 208 overwrites the rectangular area screen (character image) on the normal screen image data, and outputs the video to the monitor. In the two image superimposing examples, the respective images are temporarily stored in the image memory 203 and superimposed by the display image processing unit 208. However, the rectangular area screen (character image) is stored in the image memory 203. Instead of storing the data, the data indicating the coordinates of the vertex or the start point and the end point may be transferred to the display processing unit 208, and the display image processing unit 208 may replace the luminance of the rectangular area on the normal screen (original image).

  The process of hiding text by changing the rectangular area from the marker color to the text color has been explained. However, the text color is changed to the marker color without changing the marker color so that the text cannot be read. May be replaced.

  With the processing flow as described above, it is possible to automatically recognize the brightness and color of the marker area painted with a marker such as a highlighter pen and follow it along with the movement of the screen, thereby performing display in which a specific area is filled. As a result, in a manuscript with mixed colors or a manuscript with mixed patterns and characters, a means for specifying the character area is used as a pre-process, and marker recognition is automatically performed on the character string marked with the specific character string. It is possible to fill and display.

[Embodiment 2]
Using a predetermined area of the screen, the user registers the initial specific color as a marker color, determines a color area corresponding to the registered color, and replaces it with black. As a result, it is possible to extract the marker region more easily and reliably and to perform a process of obscuring the marked character string on the document image.

  13 and 14 are explanatory diagrams illustrating an example of the progress of image processing according to the second embodiment.

  (A) is an image of the monitor A. The original image is displayed as it is on the image of the monitor A 105, and a frame indicating a registration area for allowing the user to register the initial specific color as the marker color is displayed in an overlapping manner. At that time, the user is prompted to register the marker color in the registration area 90. The marker writer writes the marker color in the registration area 90 on the sheet. The hardware side extracts an image of the registration area 90 and registers it as an initial image 91 in the image memory 203. (B) is an image of the monitor A 105 after the marker is entered in the registration area 90. An image in the registration area 90 is extracted and registered in the image memory 203 as a marker registration image 92. Next, the initial image 91 is read from the image memory 203, and a histogram for each hue is taken in the image recognition processing unit 207 as shown in (c). The histogram information is stored in the image memory 203 as histogram information of the initial image 91. Further, the marker registration image 92 is read from the image memory 203, and a histogram for each hue is taken in the image recognition processing unit 207 as shown in (d). The histogram information is stored in the image memory 203 as histogram information of the marker registration image 92. The difference between the histogram information of the initial image 91 and the histogram information of the marker registration image 92 is taken. FIG. 14E shows an example of the result of the difference. + Represents the increase in the hue increased from the initial image 91 in the marker registration image 92, and − represents the decrease. The hue with the largest increase is recognized as the marker hue.

  Next, a document image is captured. As shown in (f), the original image has a color marking 12 registered using a registration area 90 on a part of the character string 11. In order to detect a pixel corresponding to the hue m from the captured original image and further specify a character color from the detected area, a marker area where pixels matching the hue m are concentrated as shown in FIG. To detect. The image recognition processing unit 207 extracts a histogram of the input image as shown in (h). As a result of the histogram extraction, a marker color (hue m) and a hue value of a marked word (character string) each have a peak. The other hue p is a hue representing the color of the character.

  The character image 16 in which the pixel of hue m in the marker area is replaced with the hue p is generated, and this is overlaid with the original image in the display image processing unit 208, whereby marking is performed as shown in (i). You can obscure the character string.

  Thereafter, in the second embodiment, after the video input from the sensor is stored in the image memory 203 through the processing of the color signal processing unit 206 (original image), the hue m is extracted by the image recognition processing unit 207. A character replaced with the hue P is created and stored in the image memory 203 (character image). A display image processing unit 208 captures the original image, performs a process of superimposing the character images, outputs a video, and displays the image.

  Accordingly, since the marker area is limited to the registration area 90, the marker area can be extracted more easily and reliably, the color of a specific color can be extracted from a document in which complicated colors are mixed, and more versatile. Increases nature.

[Embodiment 3]
An embodiment will be described in which the overlay display process is performed by tracking the marker area at high speed even when the document image moves.

  15 and 16 are explanatory diagrams illustrating an example of the progress of the image processing according to the third embodiment.

  An initial document image is captured, stored as a reference image in the image memory 203, a character string of a part of the document image is marked, the image is captured, and stored in the image memory 203 as a marked image.

The image recognition processing unit 207 reads the reference image and the marked image from the image memory 203. In order to correlate the images, the images are divided. As shown in (a), a partial area of the reference image is a reference block 94, and a corresponding area of the marked image is a search range 95. As shown in (b), in the search range 95, images of a plurality of target blocks 96 assuming the directions and sizes of various motion vectors 97 are defined, and for each, an absolute difference from the reference block 94 is defined. The sum of values (SAD : Sum of Absolute Difference ) is taken. The direction in which the calculated value of SAD is the minimum represents the motion vector 97. It is preferable to divide the entire original image into several blocks, determine a plurality of reference blocks, execute SAD calculation at least at two or more locations, and select a vector of the entire motion direction from the plurality of results by majority determination.

  As shown in (c), the amount of deviation between the absolute coordinates of the reference image and the marked image is calculated from the calculated vector, and the coordinates of the reference image are corrected based on this amount of deviation (step 98). Thereafter, the reference image is subtracted from the marked image (step 99). Thereby, only the marker part is extracted.

  A tracking process is performed for the extracted marker portion. As shown in FIG. 16D, the movement vector is calculated based on the difference between the image one frame before and the current image. A specific calculation method is the same as the motion vector extraction described with reference to FIGS. 15A and 15B. As shown in (e), the character image 16 with the marker portion filled is moved in accordance with the calculated movement vector 97, and the latest document image is tracked. Thereby, the extraction result of one marker area can be used over a plurality of frames without extracting a marker area for every frame. On the other hand, this tracking cannot cope with a case where the document image is replaced with another document. Therefore, it is possible to improve the misrecognition by updating the extraction of the marker area at a rate of once every several frames.

  In the third embodiment, by taking a difference based on an original document image without a marker and painting only by paying attention to the changed portion, erroneous recognition of colors can be suppressed even in an original document mixed with complicated colors. It is possible to prevent the filling of the part that was not assumed. In addition, it is possible to specify the portion to be painted with higher accuracy than the first embodiment and the second embodiment described above with respect to fluctuations in ambient lighting conditions. Further, the tracking process can be performed at a higher speed than the image recognition process of the entire first embodiment.

[Embodiment 4]
In the first embodiment, the background in the character area and the color level of the character are determined only by the luminance Y, but in the fourth embodiment, the determination is made based on R, G, and B which are the three primary color components. is there. Compared with the luminance Y alone, there are an increase in processing circuits and an increase in calculation time. On the other hand, analysis with R, G, and B components has the effect of being recognized more accurately because of more information. FIG. 17 is an explanatory diagram for determining the background and the color level of a character in the character area according to the fourth embodiment, and FIG. 18 is an explanatory diagram illustrating a method for specifying the character and marker colors.

  For example, assuming that the background is white, the characters are black, and the markers are two colors of cyan and yellow, when the RGB three primary color components are represented in a three-dimensional coordinate space, the distribution is as shown in (a). Further, assuming that each of R, G, and B components in each pixel is 8 bits, a color can be expressed with 3 colors in a total of 24 bits. The correspondence between 24 bits and each color is as shown in (b). Each notation of the R, G, and B components in (b) is 8-bit data expressed in hexadecimal. If these 24 bits are regarded as a one-dimensional axis and a histogram of each pixel in the character area is taken, the result is as shown in (c).

  Next, the color of the marker is specified from the four colors selected by the histogram shown in (c). First, since there are generally the most background colors in the character area, the background with the most distribution in (c) is the background. Next, the process moves to character specification. Since characters and markers generally have the same number of pixels, they cannot be distinguished from the histogram in (c). On the other hand, with respect to the spatial pixel distribution, the marker tends to be a series of adjacent pixels, whereas a character does not necessarily have a long continuous adjacent pixel. Therefore, for each of the R, G, and B colors, the number of pixels in which pixels in the same color section of the histogram continue is counted, for example, in the horizontal direction. For the character in FIG. 18 (a), the portion where black continues is counted, and this is designated as H_A. Similarly, H_B is counted for the marker (b), and H_C is counted for the marker (c). If the histogram of each count value is taken, it will become like (d), (e), and (f), respectively. Characters tend to be less frequent and more frequent in the horizontal direction. On the other hand, the marker is continuous, but the frequency is low. By utilizing this tendency, it is possible to specify the colors of characters and markers based on a histogram of count values of consecutive pixels in the horizontal direction.

  Thereby, even in an image in which a plurality of character colors and marker colors having the same luminance Y are mixed, a marker region can be extracted with high accuracy.

[Embodiment 5]
FIG. 19 is a configuration example of a system in which the technique of the embodiment illustrated in the first to fourth embodiments is applied to a document camera. (A) is a document camera system in which two display data output units 208_1 and 208_2 are mounted on the image processing unit 104, and can display different images on the monitor A105 for teachers and the monitor B106 for students. . For example, the original image is displayed as it is on the teacher's monitor A105 in a state where characters can be read through the marker, and the image processing of the embodiments illustrated in the first to fourth embodiments is performed on the student's monitor B106. It is possible to display the character string marked in the original image by covering the area and covering it.

  (B) includes one display data output unit 208. As an output signal format, it can be temporarily output to the PC 107 and displayed on the monitor or projector 106 via the PC 107.

  Although the invention made by the present inventor has been specifically described based on the embodiments, it is needless to say that the present invention is not limited thereto and can be variously modified without departing from the gist thereof.

  For example, the process of obscuring the character string marked by the document camera with the generated rectangular area has been described, but the present invention can be similarly applied to other image processing systems. In addition, the marker area can be used for emphasizing the document image and for character recognition instead of using it to cover the contents of the document image.

DESCRIPTION OF SYMBOLS 1 Document image input 2 Brightness difference Pd calculation process 3 Change amount distinction process 4 Character pattern block specification process 5 Character area analysis process 6 Marker area extraction process 7 Rectangular area generation process in marker area 8 Rectangular area superposition process 11 Marking partly 12 Character string with marking 13, 14 Picture area 15 Marker area 16 Character image 20 Pixel determined as character area or its area 21 Pixel determined as picture area or its area 26 Marker area Pixel or its area determined as 27 or a pixel or its area determined as a background having no character or marker other than the marker 90 Marker color registration area 94 Reference block 95 Search range 96 Target block 97 Motion vector 101 Original image 104 Image processing unit 105, 106 Monitor 200 Image processing LSI
203 Memory 207 Image Recognition Processing Unit 208 Display Image Processing Unit

Claims (16)

An image processing unit is provided for extracting a marker region marking a character string from a document image including a character string and a pattern subjected to marking that covers a part of the character string with a color region different from the color of the character string. An image processing LSI,
The image processing unit
For each pixel in the original image with respect to the original image, a difference between the maximum luminance and the minimum luminance in an image block including a plurality of pixels including the pixel is obtained as a luminance difference in the pixels, and the luminance difference is a first threshold value. A change amount distinguishing process for primarily determining the position of a pixel in the range from the first threshold to the second threshold and the position of the pixel having the luminance difference outside the range as a character area,
Based on the result of the primary determination by the change amount distinguishing process, a character pattern block specifying process that secondarily determines a pattern area and a character area in a larger area unit than the image block,
An image processing LSI that extracts the marker area based on a first frequency distribution of luminance, color intensity, or luminance difference in an area determined as a character area by the secondary determination. 2. The character / image block specifying process according to claim 1, wherein a binary image is created by obtaining, for each pixel in the document image, a binary value indicating whether or not a result of the primary determination by the change amount distinguishing process is a character area. An image processing LSI that performs noise removal processing on the binary image . 3. The image processing LSI according to claim 2, wherein the noise removal processing includes at least one expansion processing on the binary image and the same number of contraction processing after the expansion processing . 2. The image processing LSI according to claim 1, wherein the first frequency distribution is a color intensity frequency distribution, and the marker region is extracted based on the first frequency distribution . The image processing LSI according to claim 4, wherein pixels in the same color intensity section of the first frequency distribution extract the marker area based on the number of continuous pixels in the document image . 2. The image processing unit according to claim 1, wherein the image processing unit depends on whether or not each pixel is determined as a marker region based on the luminance, the luminance difference, or the color intensity range occupied by a peak in the first frequency distribution. An image processing LSI that creates a binary image, performs noise removal processing on the binary image, and extracts the marker region . In Claim 1, a rectangular area surrounding the outer periphery of the marker area is obtained, and a display image in which the luminance or color of the image data of the rectangular area or the luminance and color is changed among the image data of the original image is output. Image processing LSI. 2. The marker area according to claim 1, wherein a marker registration area is provided in the document image, a third frequency distribution of luminance or luminance difference or color intensity is calculated in the marker registration area, and the marker area is based on the third frequency distribution. An image processing LSI for obtaining a range of luminance or luminance difference or color intensity to be determined and extracting the marker region from the document image . In claim 1, each frame constituting a plurality of frames imaged in time series is set as the original image,
The image processing unit extracts a motion vector by comparing a document image of a frame from which the marker region has been extracted with a document image of a subsequent frame that has been captured, and a marker region in which position information is corrected by the motion vector. An image processing LSI serving as a marker area in the document image of the subsequent frame .   An image processing system for extracting a marker region marking a character string from a document image including a character string and a pattern that are marked to cover a part of the character string with a color region different from the color of the character string. And
  For the original image, for each pixel in the original image, a difference between a maximum luminance and a minimum luminance in an image block including a plurality of pixels including the pixels is obtained as a luminance difference in the pixels, and the luminance difference is a first luminance difference. A change amount distinguishing process for primarily determining the position of a pixel in a range from a threshold value to a second threshold value as a picture area and the position of a pixel in which the luminance difference is outside the range as a character area,
  Based on the result of the primary determination by the change amount distinguishing process, a character pattern block specifying process that secondarily determines a pattern area and a character area in a larger area unit than the image block,
  An image processing system that extracts the marker area based on a first frequency distribution of luminance or color intensity or luminance difference in an area determined as a character area by the secondary determination.   11. The character / image block specifying process according to claim 10, wherein a binary image is created by obtaining, for each pixel in the document image, a binary value indicating whether or not a result of the primary determination by the change amount distinguishing process is a character area. An image processing system including a noise removal process for the binary image. 12. The image processing system according to claim 11, wherein the noise removal processing includes at least one expansion processing on the binary image and the same number of contraction processing after the expansion processing . 11. The pixel region according to claim 10, wherein each pixel is determined to be a marker region based on the luminance or the luminance difference or the color intensity range occupied by a peak in the first frequency distribution of luminance or color intensity or luminance difference. An image processing system that creates a binary image based on whether or not, performs noise removal processing on the binary image, and extracts the marker region . 11. The image processing system according to claim 10, further comprising a camera, a memory, and a display device, wherein a document is imaged and input by the camera, stored in the memory, and color signal processing is performed on the captured image data. And an image processing system for displaying on the display device a display image in which the brightness or color of the extracted image data of the marker region or the brightness and color is changed among the image data of the document image . 15. The image processing system according to claim 14, wherein the display device is a first display device, the image processing system further includes a second display device, and the original image is displayed on the second display device .   This is an image processing method for extracting a marker region marking a character string from a document image including a character string and a pattern subjected to marking that covers a part of the character string with a color region different from the color of the character string. And
  For the original image, for each pixel in the original image, a difference between a maximum luminance and a minimum luminance in an image block including a plurality of pixels including the pixels is obtained as a luminance difference in the pixels, and the luminance difference is a first luminance difference. A change amount distinguishing process for primarily determining the position of a pixel in a range from a threshold value to a second threshold value as a picture area and the position of a pixel in which the luminance difference is outside the range as a character area,
  Based on the result of the primary determination by the change amount distinguishing process, a character pattern block specifying process that secondarily determines the pattern area and the character area in units of areas larger than the image block,
  An image processing method for extracting the marker area based on a frequency distribution of luminance or color intensity or luminance difference in the character area.

Images