JP7240163B2 - Image processing device, image management method, program - Google Patents

Description

The present invention relates to an image processing device, an image management method, and a program.

Various application procedures by using image data obtained by photographing various certificates such as driver's licenses with terminal devices such as mobile phones and smartphones, and character information obtained by character recognition of the image data are increasingly being done electronically. For example, there is a mobile terminal device described in Japanese Patent Application Laid-Open No. 2002-200313 as a device that performs character recognition by photographing with a camera of a terminal device.

In the technique disclosed in Patent Document 1, for example, character recognition is performed on a plurality of document images obtained while a document to be read is continuously photographed, and character information such as characters and values contained in the document images is detected. do. Furthermore, in the technique disclosed in Patent Document 1, character information indicating a predetermined item name is detected from among the detected character information, and recorded corresponding to the item name, for example, Character information such as names and addresses that differ for each individual is read and acquired.

Japanese Patent No. 6364051

By the way, in the case of a form larger than a driver's license, it may not be possible to photograph the entire form at once with a camera such as a smartphone in order to obtain an image with the resolution required for recognizing characters. For such a form, if the technology disclosed in Japanese Patent Laid-Open No. 2002-200010 is used, all the character information recorded in the form can be acquired by dividing the form into a plurality of areas and continuously photographing the form. becomes possible. However, when frame images are captured by dividing each area in this way, one sheet of form is restored using a plurality of obtained frame images.
However, it is difficult to restore the entire form because it is not known to which position on the form each of the captured frame images corresponds.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and its object is to detect the position of each of a plurality of photographed frame images on a form and restore the image of the entire form. An object of the present invention is to provide an image processing device, an image management method, and a program that facilitate the processing.

In order to solve the above-described problems, the image processing apparatus of the present invention includes a frame image acquisition unit that acquires frame images obtained by partially continuously photographing one form, and character recognition of character images in the frame images. Character recognition unit to perform character recognition, feature information of superimposition target image which is the frame image from which character information is obtained by character recognition, and feature information of reference image which is a frame image obtained by photographing a form prior to superimposition target image, an image processing unit that repeats a process of superimposing the image to be superimposed on the reference image to generate an image in which at least a part of the image of the form is restored , wherein the image processing unit includes the reference image and the When there is no overlapping area in the image to be superimposed, another frame image, which is photographed between the reference image and the frame image and for which character information was not obtained in the character recognition, is placed behind the reference image as a background image. It is characterized by overlapping as

In the image processing apparatus according to the aspect of the invention, when the superimposition target image is superimposed on the reference image, if the character information included in the reference image and the superimposition target image overlap, the image processing unit performs the following: It is characterized by selecting which of the frame images is to be superimposed on the top .

Further, the image processing apparatus of the present invention comprises: a frame image acquisition unit that acquires frame images obtained by partially continuously photographing one form; a character recognition unit that performs character recognition of character images in the frame images; Based on the feature information of the superimposition target image, which is the frame image for which the character information is obtained by recognition, and the feature information of the reference image, which is the frame image of the form photographed prior to the superimposition target image, the superimposition target image is used as the reference. and an image processing unit that repeats the process of superimposing the image on the image to generate an image in which at least a part of the image of the form is restored, and when the image processing unit superimposes the superimposition target image on the reference image, When the character information included in the reference image and the image to be superimposed overlap, it is characterized in that it is selected which of the reference image and the frame image should be superimposed.

The image management method of the present invention includes a frame image acquisition process in which a frame image acquisition unit acquires frame images obtained by partially continuously photographing one form; A character recognition process for performing recognition, characteristic information of an image to be superimposed, which is the frame image from which character information is obtained by the image processing unit through character recognition, and a reference, which is a frame image obtained by photographing a form prior to the image to be superimposed. an image processing step of repeating a process of superimposing the superimposition target image on the reference image according to image feature information, and generating an image in a state in which at least a part of the image of the form is restored ; In the process, if there is no overlap region between the reference image and the superimposition target image, another frame image shot between the reference image and the frame image from which character information was not obtained in the character recognition is processed. It is characterized by including superimposing as a background image behind the reference image .
Further, the image management method of the present invention includes a frame image acquisition process in which a frame image acquisition unit acquires frame images obtained by partially continuously photographing one form, and a character recognition unit in which a character image in the frame image character recognition process for performing character recognition, the image processing unit extracts the feature information of the superimposition target image, which is the frame image from which the character information is obtained by character recognition, and the frame image of the form photographed prior to the superimposition target image. an image processing step of repeating a process of superimposing the image to be superimposed on the reference image according to feature information of a certain reference image to obtain an image in a state in which at least a part of the image of the form is restored; In the processing step, when the superimposition target image is superimposed on the reference image, if the character information included in the reference image and the superimposition target image overlap, the reference image or the frame image is superimposed with whichever is upper. and selecting whether the

The program of the present invention comprises a computer, frame image acquisition means for acquiring frame images obtained by partially continuously photographing one form, character recognition means for performing character recognition of character images in the frame images, and character recognition by character recognition. The image to be superimposed is superimposed on the reference image based on the feature information of the image to be superimposed, which is the frame image from which the information is obtained, and the feature information of the reference image, which is the frame image of the form captured prior to the image to be superimposed. and image processing unit means for repeating the processing to restore at least a part of the image of the form, wherein the image processing unit means performs the reference image and the superimposition target image. If there is no overlapping area in step 2, another frame image, which is photographed between the reference image and the frame image and for which character information was not obtained in the character recognition, is superimposed as a background image on the back of the reference image. including.
Further, the program of the present invention comprises: frame image obtaining means for obtaining frame images obtained by partially continuously photographing one form; character recognition means for performing character recognition of character images in the frame images; and the feature information of the reference image, which is the frame image of the form photographed prior to the superimposition target image, to convert the superimposition target image into the reference image image processing section means for repeating the process of superimposing the document image on the reference image to generate an image in which at least a part of the image of the form is restored. When the target images are superimposed, if the character information included in the reference image and the superimposition target image overlap, selecting which of the reference image and the frame image is to be superimposed on the top.

According to the present invention, there is provided an image processing apparatus, an image management method, and a program that detect which position of a form was photographed by each of a plurality of photographed frame images, and facilitate restoration of the image of the entire form. be able to.

1 is a schematic block diagram showing the configuration of a form data management system of one embodiment; FIG. It is a figure which shows an example of the template of the form in the same embodiment. It is a figure which shows an example of the form in the same embodiment. It is an example (part 1) of a frame image captured in the same embodiment. FIG. 10 is an example (part 2) of a frame image captured in the same embodiment; FIG. It is a figure which shows the form definition table in the same embodiment. FIG. 4 is a diagram (part 1) showing a form information table in the same embodiment; It is a figure which shows the superimposition state table in the same embodiment. It is a figure which shows the form information total table in the same embodiment. It is a flowchart which shows the flow of the imaging|photography process of the form by the terminal device in the same embodiment. FIG. 11 is a diagram (part 2) showing a form information table in the same embodiment; 3 is an example (part 3) of a frame image captured in the same embodiment; FIG. 13 is a diagram (part 3) showing a form information table in the same embodiment; 10 is a flow chart showing a processing flow for selecting a form information table to be saved by the terminal device according to the same embodiment. 7 is a flow chart showing the flow of processing for superimposing frame images to generate a superimposed image by the terminal device according to the same embodiment. It is a flow chart which shows a flow of detection processing of a background image by a terminal device in the embodiment. It is a figure which shows an example of the process by the matching process part in the same embodiment. FIG. 11 is a diagram (Part 1) showing the configuration of a superimposed image in the same embodiment; 4 is an example (part 4) of a frame image captured in the same embodiment; FIG. 10 is a diagram (part 2) showing the configuration of a superimposed image in the same embodiment; It is an example of the frame image image|photographed in the other structural example of the same embodiment. FIG. 11 is a diagram showing a form information table in another configuration example of the same embodiment; FIG. 10 is a diagram showing a state in which two frame images are superimposed in another configuration example of the same embodiment; It is a figure which shows the superimposition state table in the other structural example of the same embodiment.

A form data management system S including a terminal device 1 according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a form data management system S. As shown in FIG. The form data management system S includes a terminal device 1 , a form data management device 3 , and a communication network 4 connecting the terminal device 1 and the form data management device 3 .

The terminal device 1 is, for example, a device such as a smartphone with a built-in camera, a mobile phone, a personal digital assistant (PDA (Personal Digital Assistant)), or a tablet PC (Personal Computer). The terminal device 1 executes a predetermined application program (hereinafter referred to as AP) downloaded from the form data management device 3 or another server device through the communication network 4, and functions, for example, as a communication device or as a form It functions as a device for photography.

The terminal device 1 includes an operation unit 10, an imaging unit 11, a display unit 12, a frame image acquisition unit 13, a storage unit 14, a character recognition unit 15, a selection unit 16, a recognition data management unit 17, a communication unit 18, and an image processing unit. 20.

In one embodiment, the form to be photographed by the terminal device 1 includes, for example, a title (medical checkup personal form) as shown in FIG. A template consisting of ruled lines is printed in advance, and when information of characters and numbers is recorded in the template shown in FIG. 2, for example, a form shown in FIG. 3 is obtained. It is assumed that the size of the form is such that the entire document cannot be photographed in one shot when photographed at a resolution required for character recognition processing by the character recognition unit 15 .

The imaging unit 11 is, for example, a CCD (Charge Coupled Device) camera, and when instruction information to start imaging is supplied from the operation unit 10, it continuously performs imaging at a frame rate of 10 to 30 frames per second. , frame image data is acquired for each period of the frame rate. Further, the photographing unit 11 finishes photographing when instruction information for finishing photographing is supplied from the operation unit 10 . For example, FIGS. 4 and 5 are examples of frame images 61 and 62 obtained by continuously photographing the form shown in FIG. 3 from top to bottom.

The display unit 12 is, for example, a touch panel in which an input device such as a touch pad is arranged on the screen, and displays frame images captured by the imaging unit 11 or image data stored in the storage unit 14 on the screen. do.
The communication unit 18 communicates with a device external to the terminal device 1 , such as the form data management device 3 , through the communication network 4 .
The storage unit 14 includes a frame image storage unit 141 , an application storage unit 142 , a form definition information storage unit 144 , a form information storage unit 145 and a superimposed image storage unit 147 .

The frame image storage unit 141 stores frame image data acquired by the frame image acquisition unit 13 from the imaging unit 11 together with frame identification information associated with each frame image. The application storage unit 142 stores a predetermined AP downloaded from the form data management device 3 or another server device through the communication network 4 .

The form definition information storage unit 144 stores, for example, a form definition table 1441 shown in FIG. 6 in advance. The form definition table 1441 has items of "form identification information", "item name", "item number", and "definition information". The "form identification information" item contains form identification information assigned in advance to each template type, which is information that can uniquely identify template types, such as templates with different column formats and item names. written. Information on the item name included in the form corresponding to the form identification information is written in the item of "item name".

An item number assigned in advance to each item name is written in the item “item number”. When a plurality of pieces of information are recorded in one item, an item number corresponding to the number is written. For example, in FIG. 6, the item "previous history" has two upper and lower lines to record information, so two item numbers "item 008" and "item 009" are written.

In the "definition information" item, information indicating whether the item name and the information recorded corresponding to the item name (hereinafter referred to as item record information) is written horizontally or vertically, Names and information indicating the positional relationship of item record information are written. Here, the information indicating the positional relationship between the item name and the item record information is, for example, information in which the item name and the item record information are positioned in one horizontal line or one vertical line as shown in FIG. For example, the item record information is located in any row or column on the top, bottom, left, or right.

The form definition information storage unit 144 stores a form definition table 1441 corresponding to a form type template designated in advance by the user. When photographing a form using another type of template, ie, a template with different form identification information, it is necessary to write the form definition table 1441 corresponding to the other template in the form definition information storage unit 144 .

The form information storage unit 145 stores a form information table having the same data format generated for each frame image. For example, the form information table 1451-1 shown in FIG. 7 is a table generated corresponding to the frame image 61 shown in FIG. The form information table 1451-1 has items of "frame identification information", "item number", and "acquired information". Frame identification information associated with the frame image corresponding to the table is written in “frame identification information”. An item number corresponding to the item name information is written in the item "item number". Item record information is written in the item of "acquired information".

The item of "acquired information" corresponding to the item number of the item name for which the character recognition unit 15 has acquired the item name but has not acquired the item record information corresponding to the item name is blank. Further, in the following description, any one table stored in the form information storage unit 145 is indicated as the form information table 1451 without a branch number.

The superimposed image storage unit 147 stores superimposed image data and a superimposed state table 1471 shown in FIG. Here, the superimposed image is defined as an initial image of one frame image, and the next frame image converted by the image processing unit 20 so as to match the size of the previous frame image and the positions of items and ruled lines. This is an image generated by repeatedly superimposing a frame image on the previous frame image.

As shown in FIG. 8, the superimposition state table 1471 has items of "frame identification information" and "arrangement state". In the "frame identification information" item, the frame identification information of the frame images superimposed on the superimposed image is written in the order in which they are superimposed. In the example shown in FIG. 8, the frame image with the frame identification information "1" is the frame image selected as the initial image of the superimposed image. 3” has been superimposed.

In the item "arrangement state", information indicating which frame image is arranged above or below is written in relation to the frame images before and after when frame images are overlapped with an overlapping area. . For example, in the example of FIG. 8, the frame image whose frame identification information is "1" is first superimposed on the template image, and there is no preceding frame image. A "-" is written to indicate that it does not exist. In the present embodiment, since the frame image to be overlaid later is overlaid, the frame image with the frame identification information "1" is positioned at the bottom in relation to the frame image with the frame identification information "2" that is overlaid next. . Therefore, "bottom" is written in the lower column of the "arrangement state" item of the frame identification information "1", and "upper" is written into the upper column of the "arrangement state" item of the frame identification information "2". is written. Note that FIG. 8 shows an example in which three frame images are superimposed, and a record (row) is generated according to the number of superimposed frame images.

On the other hand, the application storage unit 142 of the storage unit 14 needs to be held in a storage area even when the power of the terminal device 1 is turned off. Preallocated to the region.
Further, the information stored in the form definition information storage unit 144 of the storage unit 14 may be retained in the storage area even after the application for photographing the form ends or the power of the terminal device 1 is turned off. , is preconfigured in a non-volatile storage area. It should be noted that the form definition table 1441 may be re-registered each time the AP for photographing the form is activated.

The operation unit 10 is connected to, for example, a touch pad provided on the screen of the display unit 12, and when the user touches the screen with a finger, the touch pad outputs the touched position on the screen. Take action according to the information. For example, when the document photographing AP is activated, it displays a photographing start button for instructing the start of photographing processing and a photographing end button for instructing the end of photographing processing on the screen of the display unit 12 .

The operation unit 10 initializes the frame image storage unit 141, the form information storage unit 145, and the superimposed image storage unit 147 when the position information output by the touch pad is the position of the shooting start button. After performing the initialization, the operation unit 10 outputs to the image capturing unit 11 instruction information for starting image capturing, which instructs the image capturing unit 11 to start the process of continuously capturing the documents. Further, when the position information output from the touch pad is the position of the shooting end button, the operation unit 10 provides the shooting end button to the shooting unit 11, the character recognition unit 15, and the selection unit 16, indicating that the shooting process is to be completed. output the instruction information.

The frame image acquisition unit 13 acquires a frame image generated and output by the imaging unit 11 . When acquiring a frame image, the frame image acquisition unit 13 generates new frame identification information, associates the acquired frame image with the generated frame identification information, and writes and stores them in the frame image storage unit 141 . For example, the frame identification information is generated by the frame image acquisition unit 13 by incrementing the value from the smallest number, for example, "1", so that the chronological order of the frame images can be known.

The character recognition unit 15 extracts and cuts out images of characters and numbers (character images) from the frame image by OCR (Optical Character Recognition) processing, performs character recognition using a pattern dictionary, a font dictionary, etc., and recognizes characters and numbers. Get character information indicating Here, the information obtained by character recognition is, for example, character information indicating characters and numbers, and may be, for example, text data. Further, the character recognition unit 15 acquires the coordinates of the circumscribing rectangle of each character or number during character recognition.

Further, when outputting the character information and the coordinate information, the character recognition unit 15 associates the frame identification information associated with the frame image subjected to the character recognition processing with the character information and the coordinate information to be output. When character recognition processing is performed, the character recognition unit 15 detects all combinations of character information and coordinate information included in the frame image at once, but cannot acquire combinations of character information and coordinate information from the frame image. In this case, the character recognition processing is regarded as unsuccessful, and the frame identification information corresponding to the frame image is not provided.

The character recognition unit 15 takes in frame identification information, character information, and coordinate information. Based on the information of the form definition table 1441 pre-stored in the form definition information storage part 144 and the fetched coordinate information, the character recognition unit 15 selects characters included in a specific coordinate range from the fetched character information. Detect character information contained in

For example, in the case of horizontal writing, the specific coordinate range is a range specified by the coordinate information of each character included in a character string in which the characters are arranged in one line and the intervals between the characters are constant. The character recognition unit 15 identifies the character or character string of the item name from the character information included in the detected specific coordinate range. The character recognition unit 15 separates the character information included in the specified coordinate range including the character or character string of the specified item name into the item name information and the corresponding item record information and acquires them.

For example, in the case of the form shown in FIG. 3, the character recognition unit 15 acquires the information of the item name of "name" and the item record information of "Taro Tokkyo" for the item name of "name". Note that the character recognition unit 15 does not always acquire both the item name information and the corresponding item record information, and may acquire only the item name information. For example, in the case of the frame image 61 shown in FIG. 4, the frame image 61 does not include the item recording information corresponding to the item "age". Therefore, the character recognition unit 15 acquires only item name information. For example, when the item recording information is not included in the object to be photographed, when it is not written on the form from the beginning, or when it is unclear and the characters cannot be recognized. Therefore, even if the form information table 1451 is generated, the item record information may not be recorded. On the other hand, the character recognition unit 15 cannot separate the item record information without the item name information, and cannot acquire only the item record information. Therefore, even if the form information table 1451 is generated, there are cases where no item record information is recorded.

The character recognition unit 15 reads the item numbers corresponding to all the acquired item names from the form definition table 1441, and associates the read item numbers with the acquired item recording information. When the character recognition unit 15 completes acquiring the item name information and the item recording information for one frame image, it creates a new form information table 1451 in the form information storage unit 145, and stores the item in the generated form information table 1451. A number, item recording information corresponding to the item number, and frame identification information are written and stored.

As described above, when character information and coordinate information cannot be obtained from a frame image, the character recognition unit 15 does not obtain frame identification information corresponding to the frame image. Therefore, the form information table 1451 corresponding to the frame image is not generated in the form information storage unit 145 .

Upon receiving instruction information to end imaging from the operation unit 10 , the selection unit 16 selects a form information table 1451 to be saved from among the form information tables 1451 stored in the form information storage unit 145 .
How to select the form information table 1451 to be saved can be determined arbitrarily. Suppose that it is decided to select the storage target so as to store it. In this case, for example, the selection unit 16 performs the following processing.

The selection unit 16 deletes from the form information storage unit 145 the form information table 1451 in which the item record information is not recorded. Further, the selection unit 16 selects, for example, any one of the form information tables 1451, and records all item record information included in the selected form information table 1451 in any of the other form information tables 1451. If so, all the form information tables 1451 are deleted as redundant information. By performing these deletion processes, duplication of item record information among the form information tables 1451 can be reduced, and item record information can be stored without omission.

The selection unit 16 finally selects the form information table 1451 remaining in the form information storage unit 145 as a storage target. After completing the selection of the storage target, the selection unit 16 outputs instruction information for starting image processing to the image processing unit 20 .

The image processing unit 20 reads out the form information table 1451 written in the form information storage unit 145 first among the form information tables 1451 stored in the form information storage unit 145 . The image processing unit 20 reads from the frame image storage unit 141 the frame image corresponding to the frame identification information indicated in the item "frame identification information" of the read form information table 1451, and uses the read frame image as the first reference. be an image.

Further, based on the frame identification information of the reference image, the image processing unit 20 retrieves and reads out the form information table 1451 that follows the reference image when arranged in time series from the form information storage unit 145 . The image processing unit 20 reads from the frame image storage unit 141 the frame image corresponding to the frame identification information indicated in the item of "frame identification information" of the read form information table 1451, and uses it as an image to be superimposed.

Further, the image processing unit 20 calculates feature information indicating features of feature points of each of the reference image and the superimposition target image. A feature point is information indicating a characteristic point such as a corner in an image of interest, and a feature amount indicates what kind of feature the feature point of the image of interest has. Information. As an algorithm for calculating feature information, for example, KAZE (an example of an image feature extraction algorithm) is applied.

Further, in the background image detection process, the image processing unit 20 selects a frame image that follows the reference image when arranged in time series based on the frame identification information of the reference image from the frame image storage unit 141. search for. The image processing unit 20 reads the searched frame image and the frame identification information corresponding to the frame image from the frame image storage unit 141, and uses the read frame image as a background image.

Here, the background image detection process is a frame image for which the form information table 1451 does not exist in the form information storage unit 145 when there is no overlapping area between the reference image and the superimposition target image, that is, a frame image that is not to be saved. This is processing for detecting a frame image serving as a background that fills the space between the reference image and the image to be superimposed.
The image processing unit 20 also calculates feature information indicating features of each of the reference image and the background image.

Further, if the unprocessed form information table 1451 exists in the form information storage unit 145 after the process of superimposing the superimposition target image in front of the reference image is completed, the image processing unit 20 transfers the converted superimposition target image to the following: is the reference image of In addition, in the background image detection process, the image processing unit 20 uses the converted background image as the next reference image after the process of superimposing the background image on the back of the reference image is completed. When the form information table 1451 to be processed runs out in the form information storage unit 145 , the image processing unit 20 outputs instruction information to start transmission processing to the recognition data management unit 17 .

The image processing unit 20 performs a matching process of detecting pairs of portions having similar shapes based on the feature information of the reference image and the feature information of the superimposition target image.
In addition, when the image processing unit 20 cannot detect a predetermined number or more of pairs in the reference image and the superimposition target image, the image processing unit 20 assumes that there is no overlapping region between the reference image and the superimposition target image. A background image including frame identification information and feature information is detected. Here, it is assumed that the predetermined number is, for example, 2 or more.

Further, in the background image detection process, the image processing unit 20 performs a matching process of detecting a pair of portions having similar shapes based on the feature information of the reference image and the feature information of the background image.
Based on the matching result, the image processing unit 20 performs a process of transforming the image to be superimposed or the background image so as to match the size of the reference image and the positions of the items and ruled lines. The image processing unit 20 calculates the transformed superimposition target image and its corresponding frame identification information, or the transformed background image and its corresponding frame identification information. Here, the conversion processing is, for example, to calculate a homography matrix from the matching result as follows, and perform perspective conversion to convert the superimposition target image or the background image using the calculated homography matrix.

In this embodiment, the projective transformation matrix of the frame image is a matrix calculated based on the coordinate information of the points paired by the matching process, and is a matrix corresponding to the parameters for perspective transformation.
Therefore, in the perspective transformation, coordinate transformation is performed on a frame image that has been distorted, enlarged, or reduced at the time of photographing, so that it corresponds to the coordinates of the template image that can be seen from the front. Therefore, the coordinate system of the frame image after perspective transformation and the coordinate system of the template image match.

Further, the image processing unit 20 receives the data of the first reference image, writes the data of the first reference image to the superimposed image storage unit 147 as the initial image data of the superimposed image, and generates the superimposed state table 1471 . When the image processing unit 20 receives a perspective-transformed superimposition target image and frame identification information corresponding to the superimposition target image, or a perspective-transformed background image and frame identification information corresponding to the background image, the image processing unit 20 stores the superimposed image. A superimposed image is read out from the unit 147 .

In the case of the superimposition target image, the image processing unit 20 generates a new superimposed image by superimposing the perspective-transformed superimposition target image on the front surface of the superimposed image so that the coordinates match the read superimposed image. The superimposed image storage unit 147 is overwritten with the image. In the case of a background image, the image processing unit 20 generates a new superimposed image by superimposing the perspective-transformed background image on the back of the superimposed image so that the coordinates match the readout superimposed image, and superimposes this superimposed image. The image storage unit 147 is overwritten.

In addition, the image processing unit 20 writes the frame identification information corresponding to the superimposed target image or the background image to the superimposed state table 1471 stored in the superimposed image storage unit 147 in the item of "frame identification information", In the item of "arrangement state", information of either "-", "above", or "below" is written based on the positional relationship between the newly added frame image and the previous frame image.

When the recognition data management unit 17 receives instruction information for starting transmission processing from the image processing unit 20, the recognition data management unit 17 identifies the form indicated in the item of “form identification information” in the form definition table 1441 stored in the form definition information storage unit 144. The identification information and the superimposed image and superimposed state table 1471 stored in the superimposed image storage unit 147 are read. Further, the recognition data management unit 17 reads out all the form information tables 1451 stored in the form information storage unit 145 . The recognition data management unit 17 reads from the frame image storage unit 141 the frame image corresponding to the frame identification information indicated in the item “frame identification information” of the read superimposition state table 1471 .

In addition, the recognition data management unit 17 associates all the read information with the user identification information assigned to the terminal device 1 in advance, and transmits the information to the form data management device 3 through the communication unit 18 . Here, the user identification information is information that individually identifies a user. 3, the user ID (IDentification) at the time of logging in may be used.

The form data management device 3 is connected to the communication network 4 as, for example, a web server device, and includes a recording processing section 31 and a storage section 32 . The recording processing unit 31 receives information transmitted by the recognition data management unit 17 of the terminal device 1 through the communication network 4 and writes the received information into the storage unit 32 .

The storage unit 32 stores a form information total table 321 shown in FIG. The form information total table 321 includes "user identification information", "form identification information", "superimposed image information", "superimposed image configuration information (frame identification information)", "pre-conversion frame image information", "item number", It has items of "acquired information" and "display state".

The user identification information, the form identification information, and the superimposed image data received from the recognition data management unit 17 are written in the items of "user identification information", "form identification information", and "superimposed image information" by the recording processing unit 31. . In the item of "superimposed image configuration information (frame identification information)", the frame identification information of the item of "frame identification information" of the superimposed state table 1471 received from the recognition data management unit 17 is displayed in the order recorded in the item. Therefore, it is written by the recording processing unit 31 . In the item of "frame image information before conversion", the data of the frame image before perspective conversion corresponding to the frame identification information recorded in the "superimposed image configuration information (frame identification information)" is written by the recording processing unit 31. .

The “item number” and “acquisition information” items include the “item number” item and the “acquisition information” item of the form information table 1451 corresponding to the frame identification information recorded in the “superimposed image configuration information (frame identification information)”. Information” item is written by the recording processing unit 31 . In the "display status" item, the item record information displayed in the superimposed image among the item record information recorded in the "acquired information" item is indicated by "○", and the item record information not displayed is indicated by "○". is written. Thereby, for example, the user can later confirm which item record information is to be saved while referring to the superimposed image.

In the case of a background image, since the corresponding form information table 1451 does not exist, the items of "item number", "acquired information", and "display state" are blank.

(Image management processing by terminal device)
Next, referring to FIGS. 10 to 18, frame images obtained by photographing a form and character information obtained from the frame images are stored in the storage unit 32 of the form data management device 3 while arranging the frame images in chronological order. The image management processing to be stored in the .

The application storage unit 142 of the terminal device 1 stores a pre-downloaded AP for photographing a form. Further, the form definition information storage unit 144 stores in advance a form definition table 1441 corresponding to the template type of the form to be photographed from now on. In response to the user's instruction to activate the AP for photographing a document, the terminal device 1 activates the AP for photographing a document stored in the application storage unit 142 .

The activated AP for photographing a form activates the photographing unit 11 and the display unit 12 and displays a photographing start button and a photographing end button on the screen of the display unit 12 . The display unit 12 starts preview display for displaying an image formed on the imaging device of the imaging unit 11 on the screen.

FIG. 10 is a flow chart showing the flow of the document photographing process by the terminal device 1 in the image management process. While referring to the preview display on the display unit 12, the user points the lens of the photographing unit 11 toward the top area of the form shown in FIG. 3, and selects the photographing start button (step Sa1).

When the shooting start button is selected, the operation unit 10 initializes the frame image storage unit 141, the form information storage unit 145, and the superimposed image storage unit 147, and outputs shooting start instruction information to the shooting unit 11. do. Upon receiving instruction information to start shooting from the operation unit 10, the shooting unit 11 starts a process of continuously shooting at a frame rate of 10 to 30 frames per second, and generates frame images for each shooting. Output to the image acquisition unit 13 .

While referring to the preview display on the display unit 12 of the terminal device 1, the user moves the terminal device 1 from the beginning to the end of the form, with a part of the form as the shooting target. Thereby, the photographing part 11 photographs a frame image of a part of the form from the head to the end of the form at 10 to 30 frames per second without the user performing an operation such as pressing the shutter button. In this case, since the user does not need to perform a special operation such as pressing the shutter button, the user takes a picture as if he were shooting a moving image. For example, the photographing unit 11 first outputs the frame image 61 shown in FIG.

When acquiring the frame image output by the photographing unit 11, the frame image acquisition unit 13 generates new frame identification information, associates the generated frame identification information with the acquired frame image, and writes the generated frame identification information to the frame image storage unit 141. store (step Sa2). Here, it is assumed that "1" is generated as the frame identification information for the frame image 61 and associated with it.

When a new frame image is written in the frame image storage unit 141, the character recognition unit 15 performs character recognition processing on the written frame image, and extracts character information from the frame image and coordinate information corresponding to the character information. is detected (step Sa3).

The character recognizing unit 15 determines whether the OCR processing is successful or unsuccessful depending on whether or not a combination of one or more character information and coordinate information is detected from one frame image (step Sa4). If the character recognition unit 15 does not detect one frame image or a combination of one or more character information and coordinate information, it determines that the OCR process is not successful (step Sa4, No), advances the process to step Sa2, The frame image acquisition unit 13 acquires the next frame image.

On the other hand, when the character recognition unit 15 detects a combination of one or more character information and coordinate information from one frame image, the character recognition unit 15 determines that the OCR processing has succeeded (step Sa4, Yes), and all the detected character information and the coordinate information are associated with the frame identification information of the frame image.

The character recognition unit 15 recognizes item names and item record information from the character information based on all the character information and coordinate information corresponding to the frame identification information and the form definition table 1441 stored in the form definition information storage unit 144. Detect and retrieve. Based on the form definition table 1441, the character recognition unit 15 associates the acquired item record information with the item number (step Sa5).

The character recognition unit 15 generates a new form information table 1451 in the form information storage unit 145, and writes the frame identification information to the “frame identification information” item of the generated form information table 1451. FIG. The character recognition unit 15 writes the acquired item number in the item of "item number" and writes the item recording information corresponding to the item number in the item of "acquired information" for storage (step Sa6). For example, in the case of the frame image 61 shown in FIG. 4, the character recognition section 15 generates a form information table 1451-1 shown in FIG. In FIG. 7, the item number "007" is the item "Age", and as shown in FIG. Only item numbers are written to table 1451-1.

The character recognizing unit 15 determines whether or not instruction information for ending photographing has been supplied from the operation unit 10 (step Sa7). When the character recognizing unit 15 determines that the instruction information for ending the photographing is supplied (step Sa7, Yes), the processing ends. On the other hand, if the character recognizing unit 15 determines that the instruction information to end shooting has not been supplied (step Sa7, No), the process proceeds to step Sa2, and the frame image acquiring unit 13 acquires the next frame image. conduct.

For example, next, when the frame image 62 shown in FIG. 5 is acquired by the frame image acquisition unit 13, the form information table 1451-2 shown in FIG. 12 is obtained by the frame image obtaining unit 13, a form information table 1451-3 shown in FIG. 13 is generated in the form information storage unit 145. FIG.

FIG. 14 is a flowchart showing a flow of processing for selecting a form information table to be stored by the terminal device 1 in the image management processing. When receiving instruction information to end shooting from the operation unit 10, the selection unit 16 refers to the form information storage unit 145 and deletes all the form information tables 1451 that do not contain item record information (step Sb1).

Next, the selection unit 16 reads out the form information table 1451 lastly written in the form information storage unit 145 among the unprocessed form information tables 1451 that have not yet been processed (step Sb2). That is, the selection unit 16 reads the form information table 1451 written at the latest time when the unprocessed form information tables 1451 are arranged in the order written in the form information storage unit 145 .

The selection unit 16 stores each of the item record information recorded in the item of “acquisition information” of the read form information table 1451 in one of the other form information tables 1451 stored in the form information storage unit 145 . By determining whether or not it is recorded, it is determined whether or not it is to be saved (step Sb3). Since the item record information is character information that has been character-recognized by OCR processing, it may not be possible to read it accurately. Therefore, the determination in step Sb3 is not to determine whether or not the character information of the item record information matches. It is a process of determining whether or not any item record information is recorded in the name.

The selection unit 16 records each of the item record information recorded in the “acquired information” item of the form information table 1451 in one of the other form information tables 1451 stored in the form information storage unit 145 . If so, it is determined that it is not to be saved (step Sb3, No), and the read form information table 1451 is deleted from the form information storage unit 145. FIG.

On the other hand, the selection unit 16 selects which of the other form information tables 1451 stored in the form information storage unit 145 each of the item record information recorded in the item “acquired information” of the form information table 1451 is. is not recorded, it is determined to be a storage target (step Sb3, Yes). This is because there is item record information that is lost when the form information table 1451 is deleted. The selection unit 16 determines whether or not the unprocessed form information table 1451 exists in the form information storage unit 145 (step Sb5).

If the selection unit 16 determines that the unprocessed form information table 1451 exists in the form information storage unit 145 (step Sb5, Yes), the process proceeds to step Sb2. On the other hand, when the selection unit 16 determines that the unprocessed form information table 1451 does not exist in the form information storage unit 145 (step Sb5, No), the selection unit 16 outputs instruction information for starting image processing to the image processing unit 20 ( Step Sb6), the process ends.

The determination in step Sb3 may be made to determine whether or not the character information of the item record information is the same. However, since the character information is character-recognized by OCR processing, it may not be read accurately. . Therefore, as the determination in step Sb3, it is not determined whether or not the character information of the item record information matches. It may be a process of determining whether or not any item record information is recorded in the name.

In addition, in the process of step Sb2 shown in FIG. 14, the selection unit 16 reads out the form information table 1451 last written in the form information storage unit 145 among the unprocessed form information tables 1451. For example, The form information table 1451 written in the form information storage unit 145 first may be read. As shown in FIG. 14, when the form information table 1451 that was last written to the form information storage unit 145 is processed in order, the older form information table 1451 remains to be saved when arranged in chronological order. Conversely, if the form information table 1451 written first to the form information storage unit 145 is processed in order, the newer form information table 1451 remains to be stored when arranged in chronological order. Alternatively, the selection unit 16 may randomly read out the form information table 1451 from the unprocessed form information table 1451 .

In addition, in the process of step Sb2 shown in FIG. 14, the selection unit 16 may sequentially read out the form information table 1451 in which the number of recorded item record information items is small. By doing so, the form information table 1451 containing more item record information can finally be saved.

In the process shown in FIG. 14, the user may arbitrarily select the form information table 1451 to be saved after the process of step Sb1. In this case, the selection unit 16 determines "Yes" as the determination result of step Sb3 for the form information table 1451 selected by the user to be saved. By doing so, the form information table 1451 selected by the user to be saved can be left.

Further, in the processing of step Sb1 shown in FIG. 14, the form information table 1451 that does not include the item record information of the predetermined number or more may be deleted. By doing so, it is possible to select the form information table 1451 containing item record information of a certain number or more as a storage target. Further, in the process shown in FIG. 14, only the processes of steps Sb1 and Sb6 may be performed without performing the processes of steps Sb2 to Sb5.

FIG. 15 is a flow chart showing the flow of processing for overlapping frame images to generate a superimposed image by the terminal device 1 in the image management processing.

When the image processing unit 20 receives the instruction information to start image processing from the selection unit 16, the image processing unit 20 selects the form information table 1451 written in the form information storage unit 145 first among the form information tables 1451 stored in the form information storage unit 145. read out. The form information table 1451 written to the form information storage unit 145 first is written to the form information storage unit 145 at the earliest time when the form information tables 1451 stored in the form information storage unit 145 are arranged in chronological order. The frame image corresponding to this form information table 1451 includes the leading portion of the form (step Sc1).

The image processing unit 20 reads from the frame image storage unit 141 the frame image corresponding to the frame identification information indicated in the item "frame identification information" of the read form information table 1451, and uses the read frame image as the first reference. An image is obtained (step Sc2). The image processing unit 20 writes the data of the first reference image to the superimposed image storage unit 147 as the initial image data of the superimposed image.

Further, the image processing unit 20 generates a superimposed state table 1471 in the superimposed image storage unit 147, writes the frame identification information of the first reference image in the item of “frame identification information” of the generated superimposed state table 1471, Write "-" in both the upper and lower columns of the "Position" item (step Sc3).

Based on the frame identification information of the reference image, the image processing unit 20 retrieves and reads out the form information table 1451 that follows the reference image when arranged in time series from the form information storage unit 145 . The image processing unit 20 reads from the frame image storage unit 141 the frame image corresponding to the frame identification information indicated in the item "frame identification information" of the read form information table 1451, and uses the read frame image as the superimposition target image. (Step Sc4).

The image processing unit 20 calculates feature information indicating features of each of the reference image and the superimposition target image. (Step Sc5).

The image processing unit 20 acquires a reference image, frame identification information and feature information corresponding to the reference image, a superimposition target image, and frame identification information and feature information corresponding to the superimposition target image. The image processing unit 20 performs a matching process of detecting pairs of portions having similar shapes based on the feature information of the reference image and the feature information of the superimposition target image (step Sc6).

The image processing unit 20 determines whether or not a predetermined number or more of corresponding pairs have been detected (step Sc7). Proceed to step Sc8.

As shown in FIG. 17, the image processing unit 20, for example, coordinates the character "age" indicated by reference numeral 600 in the frame image 61 that is the reference image and "age" indicated by reference numeral 700 in the frame image 62 that is the superimposition target image. , and detect the coordinates of the characters in pairs. Further, the image processing unit 20 detects a pair of coordinates of a corner 601 of the frame image 61 that is the reference image and coordinates of a corner 701 of the frame image 62 that is the superimposition target image. .

The image processing unit 20 calculates a projective transformation matrix corresponding to the superimposition target image based on the matching result. The image processing unit 20 perspectively transforms the superimposition target image based on the calculated projective transformation matrix, and acquires the superimposition target image subjected to perspective transformation and the frame identification information of the superimposition target image (step Sc8).

The image processing unit 20 reads a superimposed image from the superimposed image storage unit 147 corresponding to the superimposition target image that has undergone perspective transformation. The image processing unit 20 generates a new superimposed image by superimposing the superimposition target image in front of the superimposed image so that the coordinates of the read superimposed image match, and overwrites the superimposed image storage unit 147 with the generated superimposed image. . The image processing unit 20 writes the frame identification information corresponding to the image to be superimposed on the superimposed state table 1471 stored in the superimposed image storage unit 147 in the item “frame identification information” and updates the item “arrangement state”. (Step Sc10).

For example, when the image to be superimposed is the frame image 62 next to the frame image 61, the image processing unit 20 writes "lower ” is written in the upper column of the “arrangement state” item of the frame identification information “2” corresponding to the frame image 62, and “−” is written in the lower column.

After finishing the process of step Sc10, the image processing unit 20 determines whether or not the unprocessed form information table 1451 exists in the form information storage unit 145 (step Sc11). When the image processing unit 20 determines that the unprocessed form information table 1451 exists in the form information storage unit 145 (step Sc11, Yes), the image processing unit 20 uses the perspective-transformed superimposition target image as the next reference image, and performs the processing from step Sc4. Processing is performed (step Sc13).

In the processing of step Sc7 shown in FIG. 15, when the image processing unit 20 determines that the number of corresponding pairs or more cannot be detected (step Sc7, No), there is no overlap region between the reference image and the superimposition target image. , the process of detecting a background image including a reference image and frame identification information and feature information corresponding to the reference image is started (step Sc9).

(Detection processing of background image)
FIG. 16 is a flowchart showing the flow of background image detection processing. The image processing unit 20 detects a background image including a reference image and frame identification information and feature information corresponding to the reference image.

The image processing unit 20 refers to the frame identification information corresponding to the reference image, and searches the frame image storage unit 141 for the frame image that follows the reference image when arranged in chronological order. The image processing unit 20 reads the retrieved frame image and the frame identification information corresponding to the frame image from the frame image storage unit 141, and uses the read frame image as a background image (step Sd1).

The image processing unit 20 calculates feature information indicating features of the background image (step Sd2).

The image processing unit 20 pairs parts having similar shapes based on the reference image, the frame identification information and the feature information corresponding to the reference image, the background image, and the frame identification information and the feature information corresponding to the background image. (step Sd3).

The image processing unit 20 calculates a projective transformation matrix corresponding to the background image based on the matching result. The image processing unit 20 perspectively transforms the background image based on the calculated projective transformation matrix (step Sd4).

The image processing unit 20 perspectively transforms the background image and reads out the superimposed image from the superimposed image storage unit 147 . The image processing unit 20 generates a new superimposed image by superimposing the background image on the back of the superimposed image so that the readout superimposed image and the coordinates match, and overwrites the superimposed image storage unit 147 with the generated superimposed image. The image processing unit 20 writes the frame identification information corresponding to the superimposed background image to the item of “frame identification information” in the superimposed state table 1471 stored in the superimposed image storage unit 147, and updates the item of “arrangement state”. (Step Sd5).

For example, the image processing unit 20 writes "lower" in the upper column of the "arrangement state" item corresponding to the background image, and writes "-" in the lower column. In addition, the image processing unit 20 writes "above" in the lower column of the "arrangement state" item corresponding to the frame image immediately before the background image.

When the process of step Sd5 is completed, the image processing unit 20 sets the perspective-transformed background image as the next reference image (step Sd6), exits the background image detection subroutine, and performs the processes from step Sc4 of FIG. conduct.

In step Sc11 of FIG. 15, when the image processing unit 20 determines that the unprocessed form information table 1451 does not exist in the form information storage unit 145 (step Sc11, No), the image processing unit 20 performs transmission processing to the recognition data management unit 17. Output the instruction information to start.

For example, when the frame images 61, 62, and 63 are captured, the recognition data management unit 17, upon receiving instruction information for starting transmission processing from the image processing unit 20, stores the form definition table stored in the form definition information storage unit 144. The form identification information "0001" shown in the item 1441 "form identification information" is read. The recognition data management unit 17 also reads out the superimposed image 40 and the superimposed state table 1471 stored in the superimposed image storage unit 147 . The recognition data management unit 17 also reads out the form information tables 1451-1, 1451-2 and 1451-3 stored in the form information storage unit 145. FIG. The recognition data management unit 17 retrieves the frame images 61, 62, and 63 corresponding to the frame identification information "1", "2", and "3" shown in the item "frame identification information" of the superimposition state table 1471 read out. is read out from the frame image storage unit 141 . Further, the recognition data management unit 17 associates all of the read information with the user identification information assigned in advance to the terminal device 1, and transmits the information to the form data management device 3 through the communication unit 18 (step Sc12). , terminate the process.

When the frame images 61, 62, and 63 are captured, the superimposed image storage unit 147 stores the superimposed image 40 shown in FIG. ing. In the superimposed image 40, the frame image 61, which is the first reference image, exists in the rearmost position, and the frame images 62-r and 63-r are the perspective-transformed frame images 62 and 63, respectively.

First, the frame image 62-r is superimposed in front of the frame image 61, which is the first reference image, and then the frame image 62-r is used as the reference image, and the frame image 63-r is superimposed in front of the frame image 62-r. are superimposed. As described above, in this embodiment, since the frame images to be superimposed later are superimposed, the frame image 61 and part of the frame image 62-r are not visible. Also, a superimposed state table 1471 shown in FIG. 8 is finally generated in the superimposed image storage unit 147 .

On the other hand, it is assumed that instead of the frame image 62, the photographing unit 11 photographs a frame image 62w shown in FIG. 19, for example. The frame image 62w is blurred near the item name and at the right end. As the item name, only "sex" is captured. The item record information corresponding to the item names other than "sex" is captured, but the item names are not captured, so the character recognition unit 15 acquires the item record information corresponding to the item names other than "sex". Can not. In addition, since the item record information corresponding to "sex" is also unclear, the character recognition unit 15 cannot acquire the item record information corresponding to "sex" either. Therefore, the selection unit 16 deletes the form information table 1451-2w of the frame image 62w in the process of step Sb1 shown in FIG.

Since there is no overlapping area between the frame images 61 and 63, the image processing unit 20 determines "No" in step Sc7 of FIG. 15, and the background image detection process shown in FIG. 16 is performed. In step Sd1 of the background image detection process, the image processing unit 20 selects the next frame image 62w from the frame image storage unit 141 as the background image based on the frame identification information of the frame image 61, which is the first reference image. Detect as

The perspective-transformed frame images 62w-r are superimposed on the back of the frame image 61 as shown in FIG. The image processing unit 20 uses the perspective-transformed frame image 62w-r as a reference image, and again reads the frame image 63 as an image to be superimposed in step Sc4 of FIG. The image processing unit 20 overlays the perspective-transformed image to be superimposed, that is, the frame image 63-r on the front surface of the frame image 62w-r, so that the superimposed image 40a shown in FIG. 20 is finally obtained.

When the recording processing unit 31 of the form data management device 3 receives the information transmitted by the recognition data management unit 17 of the terminal device 1 through the communication network 4, it writes the received information into each item of the form information total table 321 and stores it. .

The recording processing unit 31 refers to the “arrangement state” information of the superimposition state table 1471 received from the recognition data management unit 17 when writing information in the “display state” item of the form information total table 321 . The recording processing unit 31 sets the display state of the frame image whose item number is the same as that of the frame image before and after the frame image whose item number is the same as the display state of the frame image whose “arrangement state” information is “top”. The display state corresponding to the frame image that is "lower" is recorded as "x".

For example, from the frame image 61 shown in FIG. 4, as shown in the form information table 1451-1 in FIG. information has been obtained. Also, from the frame image 62 shown in FIG. 5, as shown in the form information table 1451-2 in FIG. Record information is obtained. Therefore, in the frame image 61 and the frame image 62, the item number "005" "sex", the item number "006" "medical checkup date", and the item number "007" "age" are duplicated.

As shown in the superimposed image 40 in FIG. 18, the frame image 62-r is superimposed in front of the frame image 61-r. Therefore, as shown in FIG. The column below the frame identification information "1" corresponding to the frame image 61-r is "bottom", and the column above the frame identification information "2" corresponding to the frame image 62-r is "up". It's becoming Therefore, as shown in FIG. 9, the recording processing unit 31 sets the item numbers "005", "006", and "007" corresponding to the frame identification information "1" in the "display state" item of the form information total table 321. , and writes "o" for the item numbers "005", "006", and "007" corresponding to the frame identification information "2".

Note that when a frame image 62w is captured instead of the frame image 62, the frame image 62w is a background image, and therefore corresponds to the frame identification information of the frame image 62w in the form information total table 321 of FIG. The items of "item number", "acquired information", and "display state" are blank. It should be noted that characters such as "background" indicating a background image may be written in the "display state" item so that the user can easily recognize that the image is a background image.

In the above-described embodiment, the image processing unit 20 reads the form information table 1451 first written in the form information storage unit 145 among the form information tables 1451 in the process of step Sc1 shown in FIG. However, for example, the form information table 1451 last written in the form information storage unit 145 may be read. As shown in FIG. 15, when the form information table 1451 first written in the form information storage unit 145 is processed in order, as shown in FIG. A new frame image 62-r is superimposed on the front. Conversely, if the processing is performed in order from the form information table 1451 that was last written into the form information storage unit 145, for example, when arranged in chronological order, the old frame image 62-r is superimposed on the front of the new frame image 63. .

Also, in the above-described embodiment, the terminal device 1 is moved from the top of the form to the end to take the picture, but the terminal device 1 may be moved from the end of the form to the top.

14 is performed after the process shown in FIG. 10 is completed, and the process shown in FIG. 15 is performed after the process shown in FIG. 14 is completed. When a form is photographed from the top toward the end, or vice versa, the frame images obtained are in time series, so item recording between frame images is difficult. Duplication of information can be confirmed within a certain range of frame images. Therefore, when frame images are obtained in time series, the process shown in FIG. 14 is started after the process shown in FIG. , the processing shown in FIG. 15 may be started and the processing shown in FIGS. 10, 14, and 15 may be performed in parallel.

With the above-described configuration of the present embodiment, in the terminal device 1, the frame image acquisition unit 13 acquires the frame image acquisition unit 13 each time the image capturing unit 11 continuously captures an arbitrarily continuously determined partial area of the form as an image target. Get the frame image that will be displayed. The character recognition processing unit 15 performs character recognition processing for each frame image. The image processing unit 20 selects a frame image that is the first superimposed image and the first reference image from among the frame images from which character information has been detected by the character recognition process. In addition, the image processing unit 20 selects the frame image next to the reference image in chronological order from among the frame images in which the character information is detected as an image to be superimposed, and determines the characteristics of the reference image and the image to be superimposed. Based on the indicated feature information, the image to be superimposed is transformed, the transformed image to be superimposed is superimposed on the superimposed image, and the transformed image to be superimposed is used as the next reference image, and the image to be superimposed is superimposed on the superimposed image repeatedly. As a result, the frame images for which the character information has been acquired among the frame images obtained by photographing a part of the form can be associated in chronological order, so that the user can determine which position of the form the frame image corresponds to. It becomes easy to understand. In addition, if there is an omission of photography, the association is interrupted in the middle, so it is possible to easily find the omission of photography.

In addition, since association is performed in chronological order using only captured frame images, it is possible to use the terminal device 1 for an arbitrary form without preparing an image of a form template or the like in advance. When the ruled lines of the first basic image are slanted, for example, the ruled lines of the superimposed image 40 are slanted as shown in FIG. The superimposed image 40 in which the inclination of the ruled lines is eliminated can be obtained by the image processing part 20 performing image conversion such that the ruled lines are arranged in the horizontal direction and the vertical ruled lines are arranged in the vertical direction.

In addition, in the configuration of the present embodiment described above, when there is no overlap region between the reference image and the superimposition target image, the image processing unit 20 converts another frame image captured between the reference image and the superimposition target image into the reference image. overlaid as a background image on the back of the . As a result, even if there is no continuity when arranging frame images in which text information is detected in chronological order, frame images in which text information is not detected can be associated as background images. It becomes possible to relate frame images.

(Another configuration example of one embodiment)
In the present embodiment described above, when superimposing the frame image of the superimposition target image in the process of step Sc10 shown in FIG. When performing the process of step Sc10, the image processing unit 20 may change the order in which the frame images are superimposed based on the recording status of the items "item number" and "acquired information" in the form information table 1451. .

For example, assume that the imaging unit 11 has captured a frame image 62a shown in FIG. 21 instead of the frame image 62 shown in FIG. In the frame image 62a, the item recorded information of "gender" and the item recorded information of "medical examination date" are blurred. Therefore, the character recognition unit 15 cannot acquire the item record information for these items, and acquires only the character information of the item names "sex" and "medical examination date".

When the processing of the image processing unit 20 described in the present embodiment is applied to the frame image 62a, images obtained by perspectively transforming the frame image 62a (hereinafter referred to as frame images 62a to 62r) are superimposed on the front surface of the frame image 61. The information of “sex”/“male” and “date of medical examination”/“June 10, 2017” displayed in 61 disappears in the superimposed image 40 .

FIG. 22 shows a form information table 1451a-2 generated in the form information storage unit 145 corresponding to the frame image 62a. The image processing unit 20 selects the "acquired information" corresponding to the common item numbers "005", "006", and "007" of the form information table 1451-1 and the form information table 1451a-2 shown in FIG. Browse items. Regarding the item numbers “005” and “006”, “Male” and “June 10, 2017” are recorded as item record information in the form information table 1451-1. Item record information is not recorded in the information table 1451a-2.

In such a case, the image processing unit 20 determines that if the frame images 62a-r are superimposed on the front of the frame image 61, the item record information corresponding to the item numbers "005" and "006" will be hidden. The image processing unit 20 superimposes the frame images 62a to 62a on the back of the frame image 61 as shown in FIG. 23 according to the determination result. In addition, as shown in FIG. 24, the image processing unit 20 writes "above" in the column below the item "arrangement state" of the frame identification information "1" in the superimposition state table 1471, and sets the frame identification information "2". Write "bottom" in the upper column of the "placement state" item.

Note that if the previous frame image is a background image, the form information table 1451 does not exist in the background image. Therefore, since there is no common item number between the image to be superimposed and the previous background image, the image processing unit 20 superimposes the perspective-transformed superimposition target image in front of the previous background image. .

In the other configuration example of the present embodiment described above, when the superimposition target image is superimposed on the reference image, if the character information included in the reference image and the superimposition target image overlap, the image processing unit 20 may which of the reference image and the image to be superimposed is to be superimposed on the top is selected based on the character information included in the . As a result, character information is less likely to be hidden by other frame images in the finally generated superimposed image, and more character information can be displayed. In addition, it is possible to display a large amount of item record information in the superimposed image 40, thereby facilitating confirmation of the item record information by the user who confirms it later.

Further, when a sufficient storage area of the storage unit 14 can be secured, the frame image acquisition unit 13 stores all frame images generated by the photographing unit 11 in the frame image storage unit 141 so that the character recognition unit 15 A frame image to be processed may be selected from frame images stored in the frame image storage unit 141 .

In the above-described embodiment, if the image processing unit 20 performs perspective transformation based on an erroneous matching result, the transformed image may become unreadable. For example, the smaller the value of the distance value of the matching result, the higher the accuracy. may By adding such processing, the accuracy of matching can be improved. Also, if the size of the frame image obtained as a result of the perspective transformation by the image processing unit 20 is too large, it may be excluded as a failure of the transformation.

Further, in the configuration of the above-described embodiment, if another form or an arbitrary object is caught in the image when the form is photographed, the processing by the image processing unit 20 will detect the other form or an arbitrary object as a feature. may be calculated. Therefore, when the image processing unit 20 reads out the frame image from the frame image storage unit 141, the line indicating the outer circumference of the frame image may be detected, and the characteristic information of the frame image may be calculated after trimming the outer circumference. .

Further, in the present embodiment, the terminal device 1 includes the image processing unit 20 and the superimposed image storage unit 147, and functions as one "image processing device" that performs image processing for generating a superimposed image. The image processing unit 20 and the superimposed image storage unit 147 may be separated, and an external device (for example, an image processing server device, etc.) may include the image processing unit 20 and the superimposed image storage unit 147 . In this case, the terminal device 1 outputs the information stored by the frame image storage unit 141 and the information stored by the form information storage unit 145 to an external device. The image processing unit 20 of the external device generates a superimposed image based on the information output by the terminal device 1 and outputs the finally obtained superimposed image to the terminal device 1 . In such a case, the terminal device 1 and the external device are integrated to form an "image processing device" that generates a superimposed image.

In addition, in this embodiment, the superimposition state table 1471 stores the information of “arrangement state” indicating which frame image is on top. The superimposition state table 1471 may not be provided if predetermined processing such as superimposing or lowering is performed. In this case, since the recording processing unit 31 cannot refer to the superimposition state table 1471, it is not necessary to provide the "display state" item of the form information total table 32 shown in FIG.

The terminal device 1 in one embodiment described above may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in this recording medium may be read into a computer system and executed. It should be noted that the “computer system” here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" refers to a volatile memory inside a computer system that acts as a server or client when sending a program via a network such as the Internet or a communication line such as a telephone line. It may also contain a program holder. Further, the above program may be for realizing part of the functions described above, or may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system, such as FPGA (Field Programmable Gate Array). ), etc., may be implemented using a programmable logic device.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design and the like are included within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Terminal device 3... Form data management apparatus 4... Communication network 10 Operation part 11... Photographing part 12... Display part 13... Frame image acquisition part 14... Storage part 15... Character recognition part 17... Recognition data management part 18... Communication part 20... Image processing unit 31... Recording processing unit 32... Storage unit

Claims (7)

a frame image obtaining unit that obtains frame images obtained by partially continuously photographing one form;
a character recognition unit that performs character recognition of the character image in the frame image;
The image to be superimposed is determined based on the feature information of the image to be superimposed, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form photographed prior to the image to be superimposed. an image processing unit that repeats the process of superimposing the document image on the reference image to obtain an image in which at least a part of the image of the form is restored;
with
The image processing unit
When there is no overlap region between the reference image and the superimposition target image, another frame image shot between the reference image and the frame image from which character information was not obtained in the character recognition is used as the reference image. Overlay as a background image behind the
An image processing apparatus characterized by : The image processing unit
When the image to be superimposed is superimposed on the reference image, if the character information included in the reference image and the image to be superimposed overlap, selecting which of the reference image and the frame image is to be superimposed on the top. 2. The image processing apparatus according to claim 1 , wherein: a frame image obtaining unit that obtains frame images obtained by partially continuously photographing one form;
a character recognition unit that performs character recognition of the character image in the frame image;
The image to be superimposed is determined based on the feature information of the image to be superimposed, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form photographed prior to the image to be superimposed. an image processing unit that repeats the process of superimposing the document image on the reference image to obtain an image in which at least a part of the image of the form is restored;
with
The image processing unit
When the image to be superimposed is superimposed on the reference image, if the character information included in the reference image and the image to be superimposed overlap, selecting which of the reference image and the frame image is to be superimposed on the top.
An image processing apparatus characterized by: a frame image obtaining process in which the frame image obtaining unit obtains frame images obtained by partially continuously photographing one form;
a character recognition process in which the character recognition unit performs character recognition of the character image in the frame image;
The image processing unit uses the feature information of the superimposition target image, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form captured prior to the superimposition target image, to determine the an image processing step of repeating a process of superimposing an image to be superimposed on the reference image to obtain an image in which at least a part of the image of the form is restored;
including
The image processing process includes:
When there is no overlap region between the reference image and the superimposition target image, another frame image shot between the reference image and the frame image from which character information was not obtained in the character recognition is used as the reference image. including overlaying it as a background image behind the
An image management method characterized by: a frame image obtaining process in which the frame image obtaining unit obtains frame images obtained by partially continuously photographing one form;
a character recognition process in which the character recognition unit performs character recognition of the character image in the frame image;
The image processing unit uses the feature information of the superimposition target image, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form captured prior to the superimposition target image, to determine the an image processing step of repeating a process of superimposing an image to be superimposed on the reference image to obtain an image in which at least a part of the image of the form is restored;
including
The image processing process includes:
When the image to be superimposed is superimposed on the reference image, if the character information included in the reference image and the image to be superimposed overlap, selecting which of the reference image and the frame image is to be superimposed on the top. including
An image management method characterized by: the computer,
frame image acquisition means for acquiring frame images obtained by partially continuously photographing one form;
character recognition means for performing character recognition of the character image in the frame image;
The image to be superimposed is determined based on the feature information of the image to be superimposed, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form photographed prior to the image to be superimposed. An image processing unit that repeats the process of superimposing the image on the reference image to obtain an image in which at least a part of the image of the form is restored;
It is a program for functioning as
In the image processing section means,
When there is no overlap region between the reference image and the superimposition target image, another frame image shot between the reference image and the frame image from which character information was not obtained in the character recognition is used as the reference image. including overlaying it as a background image behind the
program. the computer,
frame image acquisition means for acquiring frame images obtained by partially continuously photographing one form;
character recognition means for performing character recognition of the character image in the frame image;
The image to be superimposed is determined based on the feature information of the image to be superimposed, which is the frame image from which the character information is obtained by character recognition, and the feature information of the reference image, which is the frame image of the form photographed prior to the image to be superimposed. An image processing unit that repeats the process of superimposing the image on the reference image to obtain an image in which at least a part of the image of the form is restored;
It is a program for functioning as
In the image processing section means,
When the image to be superimposed is superimposed on the reference image, if the character information included in the reference image and the image to be superimposed overlap, selecting which of the reference image and the frame image is to be superimposed on the top. including
program.

Images