AU2016238852B2

AU2016238852B2 - Method for analyzing contents of at least one image of a deformed structured document

Info

Publication number: AU2016238852B2
Application number: AU2016238852A
Authority: AU
Inventors: Jeremy Nicolas Laurent Dura; Laurent Patrice Rostaing; Alain Rouh
Original assignee: IDEMIA Public Security SAS
Current assignee: IDEMIA Public Security SAS
Priority date: 2015-10-05
Filing date: 2016-10-05
Publication date: 2022-03-03
Anticipated expiration: 2036-10-05
Also published as: EP3153991A1; CA2944500A1; FR3042056B1; AU2016238852A1; CA2944500C; EP3153991C0; FR3042056A1; EP3153991B1

Abstract

The present invention relates to a method for analyzing an image of a deformed structured document, applied with an analysis device 5 connected to a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and to at least one region to be analyzed, comprising steps for: - acquiring (El) said image of said document; 10 - selecting (E2) initial reference document templates from stored templates; - for each stored region of interest of each of said initial reference document templates: extracting (E5) points of interest in the acquired image, matching (E6) the points of interest with the stored points of 15 interest, - selecting (E7) a final reference document template from among said initial reference document templates and estimating (E8) a fine geometrical between the acquired image and a reference image associated with the final reference template from said matches, 20 - determining and analyzing (E9, E10) at least one region to be analyze of said acquired image depending on said estimated fine geometrical transformation. 1/7 6 7 7bis -dievileTorFtaRing images~ _ _ _ _ _ \_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _from a distance_ _ _ _ _ System for analyzing contents | | 5 Storage|| device | Analysis system I 2-- 3 Data processing Image device capture | | device computer ------------------- I I| | -_ - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 L - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 1 FIG. 1 <RA DNA T N <KE I «< << << 05130 S Q3 C)4210R02 «<U<4 c2

Description

1/7 6 7 7bis

-dievileTorFtaRing images~ _ _ _ _ _ \_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _from a distance_ _ _ _ _ System for analyzing contents | |

Storage|| device |

Analysis system I

2-- 3 Data processing Image device capture | | device computer ------------------- I I|

| -_- - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 L- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1

1

FIG. 1

<RA DNA T N <KE I «< << << 05130 S Q3 C)4210R02 «<U<4 c2

Method for analyzing contents of at least one image of a deformed structured document

GENERAL TECHNICAL FIELD The present invention relates to the field of identification and analysis of documents. More specifically, it relates to a method and a device for analyzing contents of at least one image which may be remotely acquired from a possibly deformed structured document such as an identity document, a game ticket, a proof of address, an invoice, a form, etc...

BACKGROUND Methods, such as the one described in patent FR 2 952 218, propose to take into account deformations in the image of a document submitted via a software route, in order to determine an image of a virtual document identical with the real document but which would not be deformed. For this purpose, methods have been proposed determining pairings of points of interest of a two-dimensional grid projected onto the document with the corresponding points in the creased document and calculating a deformation of the document compatible with these pairings. Also, the proposed method in patent application W02011/058418 proposes correcting a perspective transformation of the image of a document to be analyzed. In order to correct such a transformation, points of the image of the document are matched with those of the image of a document template; a homography of the image of the document compatible with these pairings is then calculated. The thereby proposed method gives the possibility of correcting the effects of a perspective projection from one plane into another plane, which may be considered as a unique and global transformation for the whole image of the relevant document. However, this method does not give the possibility of correcting the effects related to possible physical deformations of the document in particular deformations having the consequence that the document is not planar once it is deformed. For example, the method proposed in W02011/058418 is not capable of correcting a deformation of the document comprising to at least one creasing and/or at least one curving and/or at least one folding of the document. Moreover, such methods require significant computing power in order to compute a complex deformation of the document compatible with the pairings of a large number of points of interest. Such methods may also require a more complex system than a simple image shooting device, for example in order to project onto the creased document patterns required for computing smoothing out by software.

Therefore there exists a need for a method allowing identification of the symbols on a creased document, by means of a simple imaging device not comprising any mechanical smoothing, or projection system, without being sensitive to light variations and to local deformations undergone by the document, and without requiring expensive computations such as those relating to the determination of a specific global template for the deformation of a document.

SUMMARY

The present invention according to a first aspect proposes a method for analyzing contents of at least one image of a structured document which may be deformed, said method being applied by an analysis device configured so as to be connected to a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document template having a fixed location in any structured document according to said given reference document template, each reference document template being further associated with a reference image of a document according to said template, and characterized in that it comprises steps of: - acquiring said at least one digital image of said document, - selecting at least one initial reference document template from among reference document templates stored in the storage device, - for each region of interest stored of each of said selected initial reference document templates: " extracting points of interest in the acquired image, " matching the points of interest extracted with the points of interest stored of said region of interest of the initial reference document template, - selecting a final reference document template from among said initial reference document templates selected depending on said matches, - estimating a fine geometrical transformation between the acquired image and the reference image associated with the final reference template, taking into account the local deformations of the document, from said matches, wherein the fine geometrical transformation is estimated by means of a reverse distance weighting algorithm or by interpolation with splines, - determining at least one region to be analyzed of said acquired image by projecting said at least one region to be analyzed of the final reference document template in said acquired image depending on said estimated fine geometrical transformation, and - analyzing the contents of said at least one region to be analyzed of the determined acquired image. The contents of the document may thus be read efficiently, without requiring mechanical smoothing. The fact of achieving pairing of the points of interest of the image for each region of interest of the template independently of each other, instead of attempting to achieve a pairing of the points of interest of the whole document of a single go, gives the possibility of taking into account the local deformations of the document to be analyzed by means of elementary transformations such as affine functions or homographies. This thus gives the possibility of not having to carry out an expensive computation of the pairings and of the deformation of the whole of the document directly from the set of points of interest of the document and to limit the risk of a pairing error. This finally gives the possibility of making use of the knowledge of the structure of the document template, and of its different regions of interest, in order to evaluate the local deformations of the document before analyzing the regions to be analyzed of the document. Such an acquisition step may be carried out remotely.

A region to be analyzed of the final reference document template comprising a heading, the step for analyzing the contents of the region to be analyzed of the acquired image determined by projecting said region to be analyzed of the template may further comprise an association of the analyzed contents with said heading. This allows automatic characterization of the analyzed contents by associating a label describing its contents.

Said method according to the first aspect may comprise: - a step for determining at least one global geometrical transformation between the acquired image of said document and at least one reference image associated with a selected initial reference document template, and - for each stored region of interest of each of said selected initial reference document templates, a step for determining a region of interest of the corresponding acquired image, by projection in the acquired image of said stored region of interest depending on the determined global geometrical transformation, and wherein the step for extracting points of interest may comprise the extraction of said points of interest in the region of interest of the determined acquired image. This gives the possibility of only searching for points of interest of the acquired image and only associating them with those of the template in the small extension areas. Such searches and associations are thereby carried out more rapidly and more reliably. The taking into account of an overall geometrical transformation gives the possibility of limiting the risk of an error during these steps in the case of deformation of the acquired image relatively to that of the template, for example because of a perspective effect.

In a first alternative embodiment, said at least one initial reference document template is selected from among the reference document templates stored in the storage device depending on a machine readable zone (MRZ) of the acquired image. The reading of the MRZ of the document to be analyzed thus allows determination of the type of the document and of preselecting the applicable document template, or at least a restricted set of document templates among which is found the document template which may be applied to the document to be analyzed.

In a second alternative embodiment, the selection of said at least one initial reference document template from among the reference document templates stored in the storage device comprises the determination of points of interest in the acquired image, the matching of the determined points of interest with the points of interest of at least one reference document template stored in the storage device and the determination of said at least one initial reference document template depending on said matching.

In the absence of MRZ, a restricted set of document templates among which is found the document template which may be applied to the document to be analyzed may thus all the same be selected.

Said global geometrical transformation may be determined depending on the machine readable zones (MRZ) of the acquired image and of an initial reference document template. The comparison of the geometrical patterns of the MRZ thus gives the possibility of inferring therefrom a first geometrical transformation between the image of the document to be analyzed and that associated with the document template and of taking into account this transformation for determining the regions of interest in the image of the document to be analyzed.

Said step for selecting the final reference document template may comprise the computation for each initial reference document template of a matching score depending on the matches carried out for said initial reference document template, and the selection as a final reference document template of the initial reference document template having the maximum matching score. The reference document template used for determining the regions of the document to be analyzed is thus the template which matches best the document to be analyzed. As an example, said document may be from among an identity document, an invoice, a documentary proof, a form.

The fine geometrical transformation may be estimated by means of a weighting algorithm by reverse distance or interpolation with splines.

The analysis method according to the first aspect may further comprise the computation of a corrected acquired image from the acquired image of the deformed document and from said estimated fine geometrical transformation. A corrected image may thus be nevertheless provided by the method, for example in order to display it on a document or a screen, in order to make visible to the eyes of a human operator, the result of the taking into account of the deformations of the document to be analyzed.

Said at least one region to be analyzed of the acquired image comprising a string of characters may be read by an optical character recognition (OCR) method. For example this allows the analysis of strings of characters of customized fields of the document to be analyzed and recording them in a form which may be used by a word processing.

According to a second aspect, the invention relates to a computer program product comprising code instructions for executing an analysis method according to the first aspect when this program is executed by a processor.

According to a further aspect, the invention relates to a device for analyzing contents of at least one image of a structured document which may be deformed, the device comprising a data processing device configured so as to be connected to a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document template having a fixed location in any structured document according to said given reference document template, each reference document template further being associated with a reference image of a document according to said template, and a device for capturing said at least one digital image of said document, the data processing device configured for performing steps of: - selecting at least one initial reference document template from among reference document templates stored in the storage device; - for each stored region of interest of each of said selected initial reference document templates: " extracting points of interest in the acquired image, " matching the extracted points of interest with the stored points of interest of said region of interest of the initial reference document template, - selecting a final reference document template from among the initial reference document templates selected depending on said matches, - estimating a fine geometrical transformation between the acquired image and the reference image associated with the final reference template taking into account the local deformations of the document, from said matches, wherein the fine geometrical transformation is estimated by means of a reverse distance weighting algorithm or by interpolation with splines, - determining at least one region to be analyzed of said acquired image by projecting said at least one region to be analyzed of the final reference document template in said acquired image depending on said estimated fine geometrical transformation, and - analyzing the contents of said at least one region to be analyzed of the determined acquired image.

According to a fourth aspect, the invention relates to a system for analyzing contents of at least one image of a structured document which may be deformed comprising: - a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document template having a fixed location in any document structured according to said given reference document template, - the analysis device according to the third aspect configured so as to be connected to said storage device.

According to a fifth aspect, the invention relates to a device for taking images from a distance comprising an analysis system according to the fourth aspect of the invention.

According to a sixth aspect, the invention relates to a mobile terminal comprising a device for capturing images at a distance according to the fifth aspect of the invention.

Such computer program product, analysis device and system, remote image shooting device, and mobile terminal have the same advantages as those mentioned for the method according to the first aspect.

As used herein, except where the context requires otherwise the term 'comprise' and variations of the term, such as 'comprising', 'comprises' and 'comprised', are not intended to exclude other additives, components, integers or steps.

Reference to any prior art in the specification is not an acknowledgment or suggestion that this prior art forms part of the common general knowledge in any jurisdiction or that this prior art could reasonably be expected to be understood, regarded as relevant, and/or combined with other pieces of prior art by a skilled person in the art.

PRESENTATION OF THE FIGURES

Other features and advantages of the present invention will become apparent upon reading the description which follows of a preferential embodiment. This description will be given with reference to the appended drawings wherein: - Fig. 1 illustrates a system for analyzing contents according to an embodiment of the invention;

- Fig. 2 illustrates an example of an identity document for which the method according to the invention is applied;

- Figs. 3a, 3b and 3c illustrate different types of physical deformations which a document may undergo for which the method according to the invention is applied;

- Fig. 4 is a diagram schematizing an application of an analysis method according to the invention;

- Fig. 5 illustrates the result of applying a step for matching points of interest of a document template with the equivalent points of interest of the acquired image belonging to a few particular regions of interest, of the method according to the invention.

DETAILED DESCRIPTION

The present invention relates to an application of a method for analyzing contents of at least one image of a structured document to be analyzed, which may be deformed 1, by a data processing device 2 and an image capture device 3 comprised in an analysis device 4 configured so as to be connected to a storage device 5. The analysis device 4 and the storage device 5 are comprised in a system for analyzing the contents 6 as illustrated in Fig. 1. Such an analysis device or such an analysis system may be comprised in a device for shooting images at a distance 7a which may itself be comprised in a mobile terminal 7.

By structured document is meant a document in which at least one portion of the information of interest to be analyzed in the document are written into one or several predetermined zones of known location and fixed for all the documents of a same type, as opposed to a document on no particular form. The information borne by the document may both be in the form of handwriting or of printing characters or further be graphic elements like images or logos. Such documents are for example identity documents such as passports, identity cards or driving licenses, forms, invoices, proof, examination MCQs or game tickets. For each of these types of documents at least one document template may be defined. The document to be analyzed which may be deformed in which the method proposes to analyze contents is supposed to be compliant with one of the predefined document templates, subsequently called reference document template. Such a reference document template defines a geometrical layout of the signs making it up, i.e. the theoretical position of the signs, characters or blocks making it up when the document is not physically deformed. With a reference document template, may thus be associated regions of interest 8 illustrated in Fig. 2. A region of interest may be a zone of the document with a small extent comprising a graphical element, such as an image, a logo or a certain string of characters, generic string, i.e. not customized and present in all the documents according to the same template, and with a fixed location in all the documents depending on the document template. As an example, in the case of a passport, as illustrated in Fig. 2, such a region of interest may be a rectangular area comprising a heading such as the string of characters "surname". Such a string of characters is written on all the passports issued according to the same template, every time at the same location. Each document template is also associated with points of interest 9 comprised in said regions of interest and having a fixed location in any structured document according to the reference document template. The points of interest of a region of interest are zones with a very small extension having particular properties which make them easily recognizable as compared with other points in this region, for example high gradient points, angles, ends of lines, SIFT ("scale invariant feature transform") points. Such points also bear a significant amount of information (cf "Lowe D. G., 1999, Object recognition from local scale invariant features, Proceedings of the International Conference on Computer Vision, Vol. 2, p. 1150-1157"). In the example given above, the ends of the "s" character may thus be points of interest. In a document template, each region of interest may be characterized by a shape and a localization in the document, and each point of interest of a region of interest may be characterized in the form of a localization and of a descriptor corresponding to a local signature, such as for example histograms of the orientations for the SIFT points. With each reference document template may also be associated at least one region to be analyzed 10 containing a non-generic piece of information to be analyzed, i.e. potentially different for each document to be analyzed. The contents of such a region to be analyzed may be a character such as an alphanumerical character like a letter, a figure or a special character such as a hyphen, or else a string of such characters, or else further a graphic element like a specific pattern of the organism/country issuing the document. Such regions to be analyzed may be zones of the document comprising customized fields for example the field "name" of a passport, a photograph of the owner of the document or a security element. As an example in the case of the passport illustrated in Fig. 2, the regions to be analyzed are the framed regions comprising the strings of characters "DONATIEN", "UTOPIA", etc... In a document template, each region to be analyzed may be characterized by a form and a localization in the document as well as a heading characterizing the type of information contained in the region to be analyzed.

Such structured documents as well as the associated reference document templates may also comprise a machine readable zone (MRZ) 11. Such a machine readable zone (MRZ) for example appears in the lower area of the passport as illustrated in Fig. 2. Such a readable area notably contains pieces of information relating to the name, the passport number, the nationality, the date of birth, the gender, the expiry date of the passport, the personal identity number, the country issuing the passport. It may be read by an optical character recognition (OCR) method. Such a machine readable zone (MRZ) of a structured document may give the possibility of identifying a reduced number of reference document templates potentially associated with the structured document from among the stored reference document templates.

Such a document may have physical deformations of several types as illustrated in Figs. 3a to 3c resulting from the handling of the document by the user. Such deformations may for example be creases, as illustrated in Fig. 3a, folds as illustrated in Fig. 3b or curved parts as illustrated in Fig. 3c.

Fig. 1 illustrates an analysis device configured for analyzing the contents of at least one image of such a structured document by applying an analysis method described with reference to Fig. 4. This method proposes pairing, for each region of interest of at least one document template, points of interest with corresponding points in the document to be analyzed and of comparing the positioning of the paired points in order to determine a deformation of the document, and then using the knowledge of this deformation for determining the position in the document to be analyzed of regions to be analyzed corresponding to those of the document template. By processing the zones of interest of the document in an independent way of each other, local deformations of the document may be determined, reliably modeling the complex deformation of the document, without however requiring as much computing power as during the determination of an overall deformation of the document by means of a complex transformation taking into account the whole of the points of interest of the document. The method thus gives the possibility of properly analyzing a creased document without requiring any smoothing, from a pre-recorded reference document template corresponding to the creased document.

This method is applied by the analysis device 4 comprising the image capture device 3 and the data processing device 2. Such an image capture device 3 is configured for acquiring at least one digital image of said document to be analyzed during an acquisition step El. Such an image capture device may comprise a camera for shooting images at a distance such as a camera, a video camera or else a scanner. This image capture device is configured for acquiring a digital image of the document, for example as a matrix of pixels, reproducing the possible physical geometrical deformations of the document. Such an image may also comprise geometrical deformations produced by the perspective projection made by the image capture device. Such deformations are all the more pronounced since the image shooting is carried out at a distance and that the layout of the document relatively to the device is away from a front-parallel position. Such an acquisition exclusively requires exhibiting the document to be processed in the image shooting field of the image capture device, and does not require that the document be put into contact with the analysis device, for example by being inserted into the device or laid flat on an imaging surface. Such a data processing device 2 may comprise a computer and a communication interface. Such an onboard computer may consist in a processor or microprocessor, of the x-86 or RISC or ARM type for example, a controller or micro-controller, a DSP, an integrated circuit such as an ASIC or a programmable circuit such as an FPGA, a combination of such elements or any other combination of components giving the possibility of applying the computing steps of the method described below. Such a communication interface may be any interface, either analogue or digital, allowing the computer to exchange information with the other elements of the analysis system 6. Such an interface may for example be a series interface RS232, a USB interface, a Firewire, HDMI interface or a wired network interface of the Ethernet type or a wireless interface of the Wifi type or further any type of interface.

The data processing device 2 is configured so as to be connected via this communication interface with the storage device 5 and the image capture device 3. Such a storage device 5 may comprise a rewritable nonvolatile memory (flash memory or EEPROM memory) which may store a database. The storage device stores reference document templates as those described above, each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document template having a fixed location in any document structured according to said given reference document template. Each reference document template is further associated with a reference image of a document according to said template.

Alternatively, the storage device may consist in a RAM memory of the data processing device 2.

In order to be able to compare the acquired image of the document to be analyzed with a reference document template for identifying the deformations undergone by the document to be analyzed and the location of the regions to be analyzed in this document, it is necessary to identify, from among the document templates stored in the storage device, the document template corresponding to the document to be analyzed.

For this, the computer of the data processing device may be configured for applying a step E2 for selecting at least one initial reference document template from among the reference document templates stored in the storage device. If the document to be analyzed contains one or several pieces of information identifying without any ambiguity the applicable document template, the selection step E2 selects a single document template exactly corresponding to the document to be analyzed. Failing this, this step allows selection of a set of reference templates close to each other and among which is found the reference template exactly corresponding to the document to be analyzed. In a first embodiment alternative, said at least one initial reference document template is selected from among the reference document templates stored in the storage device depending on the machine readable zone (MRZ) of the acquired image. To do this, the machine readable zone (MRZ) may be read by an optical character recognition (OCR) method. Next, depending on the read information, for example depending on the country issuing the document or on the number of the document comprised in the machine readable zone (MRZ), one or several initial reference document templates are selected from the stored reference document templates.

In a second alternative embodiment, the selection of said at least one initial reference document template from among the reference document templates stored in the storage device comprises the determination of points of interest in the acquired image, the matching of the determined points of interest with the points of interest of at least one reference document template stored in the storage device and the determination of said initial reference document template depending on said matching. In the absence of MRZ, this alternative embodiment nevertheless gives the possibility of selecting one or several document templates for which the points of interest correspond quite well to corresponding points of the image of the document to be analyzed. In the case when neither the reading of the MRZ, nor the comparison of points of interest gives the possibility of identifying one or several reference document templates corresponding to the structured document, this step may not lead to the selection of any initial reference document template, or else on the contrary to the selection of all the reference document templates stored as initial reference document templates.

The computer of the data processing device is then configured for applying for each stored region of interest each of said at least one selected initial reference document template: * a step E5 for extracting points of interest in the determined acquired image, * a step E6 for matching the extracted points of interest with the stored points of interest of said region of interest of the initial reference document template. For each selected initial reference document template, points of interest are thus extracted from the whole of the acquired image; and then for each region of interest the extracted points of interest are paired with those of the template. Such matching is achieved depending on the similarity of the descriptors of the points of interest. Such matching also depends on their localization in order not to pair two points of interest having very similar descriptors but located at localizations, the distance of which is for example incompatible with the matchings of the other points of interest of the same region of interest. Such a matching is illustrated in Fig. 5 wherein a large number of points of the letters of the word passport, forming points of interest of the document template, are matched with the equivalent points of the acquired image. During this step, a local geometrical transformation, making the points of interest of the reference document template match those of the corresponding acquired image, may be determined for each region of interest of the reference document template. Such a transformation may for example be obtained from: • an affine template determining an affine application connecting the points of interest of the acquired image and the points of interest of the reference document template, such as a translation, a rotation a homothety or scaling, or a combination of these transformations. Such a template gives the possibility of preserving the alignments of points and the ratios of distances between points of the document, * a homographic template determining a homographic application connecting the points of interest of the acquired image and the points of interest of the reference document template. Such a template gives the possibility of having a plane of a planar surface as seen by a camera correspond with the plane of the same surface in another image, an interpolation template determined by means of a reverse distance weighting algorithm, as proposed in document "Franke R., 1982, Scattered data interpolation: tests of some methods, mathematical of computation, 38(157), 181-200", and/or interpolation with splines. Such a transformation, defined by a given region of interest, thus gives the possibility of characterizing the locally undergone deformation by the structured document to be analyzed at this region of interest. The matching E6 of the points of interest for each region of interest represent much less intensive computations as a matching of the whole of the points of interest of a document and this for two reasons. First, the use of points of interests in regions of interest limits the extension of the zone where points of interests are sought, and therefore reduces by as much their number. Secondly, the matching by region drastically limits the number of possible matching combinations. Both of these effects allow much less intensive computation on the one hand, but also produce more reliable and more accurate results by limiting the probability of false pairing which occurs in the matching and the estimation of the geometrical transformation: there are less false points of interests, for example related to dirt on the document, and which would then be outside the regions of interest, and there is also less risk of wrong matching of point of interests. The matching E6 may be applied by means of one of the methods described in the following documents: • "Random sample consensus : a paradigm for model fitting with applications to image analysis and automated Cartography", 1981, by M. A. Fischler and R. C. Bolles; • "Multiple View Geometry in Computer Vision", 2000, by R. Hartley and A. Zisserman;

* "A robust Technique for Matching Two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry", 1995, by Z. Zhang, R. Deriche, 0. Faugeras and Q. T. Luong. Next, the computer of the data processing device may apply the selecting E7 of a final reference document template from among said initial reference document templates selected depending on said matches. For this purpose, a computation may be applied for each initial reference document template with a matching score depending on the matchings achieved for said initial reference document template, and the initial reference document template having the maximum matching score may be selected as final reference document template. As an example, such a score may be computed depending on the number of points of interest which are matched or further depending on the similarity level of the descriptors of the points of interest. The selection of the reference template to be taken into account for the subsequent steps of the method is thereby made from among the initial document reference templates selected previously so as to select the reference template for which the document to be analyzed is the closest. When a single reference document template was selected during the selection step E2, this step for selecting a final reference document template may be used for validating the initial selection of the template by making sure that the number of matched points of interest is sufficient, for example greater than a predetermined threshold. It is thus possible to make sure that the document to be analyzed is not very far from the selected template and that the initial selection is not erroneous. In the case when no reference document template has been considered as particularly relevant during the selection step E2, this step gives the possibility of determining the closest reference document template to the structured document, and of deciding to consider it as valid or not for an analysis of the contents of the structured document.

The matches, determined for each region of interest of the final reference document template, are representative for each of these regions of interest of the local deformations of the document to be analyzed with respect to the final reference document template. The computer of the data processing device may then estimate E8, a so-called fine geometrical transformation between the acquired image and the reference image associated with the final reference template taking into account the local deformations of the document from said matches carried out during step E6 at the level of the zones of interest taken by one by one, and, if necessary transformations resulting from step E6. Such a fine geometrical transformation may be estimated as a complex global template from matched points and/or simple local transformations computed for each region of interest during step 6. As an example, such a fine transformation may be determined by means of a reverse distance weighting algorithm or by interpolation with splines. In other words, the so-called fine geometrical transformation estimated during step E8 is thereby obtained by making use of: (i) all or a portion of the points of interest matched during step E6; and (ii) optionally their associated geometrical transformations, zone of interest per zone of interest. The use of such deformation templates gives the possibility of not limiting the method to the taking into account of simple deformations such as deformations due to the perspective induced by poor positioning of the processed document, for example in a tilted plane with respect to the image shooting plane, as this is the case of the method described in document W02011/058418. Thus, the method described gives the possibility of taking into account both such simple deformations but also much more varied deformations and much more complex such as local deformations due to creasing of the document. Such a fine geometrical transformation may be determined by successively applying at least two algorithms and/or applications from among the examples mentioned above. Different algorithms may thus be tested in order to determine the most suitable or the most suitable combination for describing the local deformations of the document. Different algorithms may be adapted and recorded so that at least one algorithm corresponds to each type of deformation, such as creasing, folding, curving etc...

And then the computer of the data processing device determines E9 at least one region to be analyzed of said acquired image by projecting said at least one region to be analyzed of the final reference document template in said acquired image depending on said estimated fine geometrical transformation. The location in the acquired image of the region(s) to be analyzed in the document to be analyzed may thus be determined by taking into account the local deformations of the document, for the moderate computation cost of a projection.

Finally, the analysis device analyzes E10 the contents of said at least one region to be analyzed of the determined acquired image. The region(s) to be analyzed of the acquired image may thus be read by an optical character recognition method (OCR). Such a readout is particularly suitable if the regions to be analyzed comprise strings of characters. In order to analyze images comprised in these regions to be analyzed, the regions to be analyzed of the acquired image may be processed by an analysis method adapted to the type of relevant image, such as a face recognition method in the case of a portrait.

When the regions to be analyzed of the final reference document template are associated with a heading, the step for analyzing the contents of a region to be analyzed of the determined acquired image by projection of a region to be analyzed of the template may comprise association of the analyzed contents with the heading associated with this region to be analyzed of the template. Thus, the analysis of the document to be analyzed does not require correction of its local deformations in the acquired image. It is directly achieved in the acquired image by using the fine geometrical transformation estimated in step E8. This allows a gain in time and in computing power, and avoids the errors committed during a correction thus allowing a gain in accuracy of analysis.

In an embodiment, the computer of the data processing device is further configured for applying an optional step E3 for determining a global geometrical transformation between the acquired image of said document 1 and at least one reference image associated with an initial reference document template. Such a transformation being a global geometrical transformation will then poorly take into account the local deformations of the document and will not allow alone efficient determination of the position in the acquired image of the regions to be analyzed from those of a reference template. It is then not substituted for the estimated fine geometrical transformation during the estimation step E8. On the other hand, such a global transformation gives the possibility of taking into account the layout of the document acquired at a distance and may therefore be used for improving the pairing of the points of interest of a region of interest of a document template and those of the acquired image, as described below. Moreover, such a transformation remains inexpensive to compute because of its simplicity.

In a first embodiment, such a step is applied before any matching of the points of interest of the acquired image and of a reference template. Such a determination of a global transformation may then be applied by comparing geometrical elements of the acquired image with equivalent elements of at least one reference image of a document template. When several initial reference document templates have been selected during the selection step E2, such a transformation may be determined only once with respect to a single reference template from among the selected initial reference document templates. Alternatively, such a determination may be applied independently with respect to each selected initial reference document template. In a first alternative embodiment, the global geometrical transformation of the document is determined depending on machine readable zones (MRZ) of the acquired image and of the reference image of the initial reference document template(s). This gives the possibility of limiting the required computing time. To do this, the machine readable zones of the acquired image and of the reference image of the initial reference document template may be read by a method for optical recognition of geometrical shapes, and then a global geometrical transformation between the acquired image of said document and the reference image associated with the initial reference document template may be determined. In a second alternative embodiment, in the absence of MRZ, such a determination of a global transformation may be applied by comparing geometrical elements of the acquired image and of a reference image of at least one document template such as lines of a logo or of an emblem. As an example, such a global geometrical transformation may be determined from: . an affine model determining an affine application connecting the positions of the characters of the machine readable zone (MRZ) of the acquired image and the positions of the characters of the machine readable zone (MRZ) of the reference image of the initial reference document template, such as a translation, a rotation, a homothety, a scaling, or a combination of these transformations. Such a template gives the possibility of preserving the alignments of points and the ratios of distances between points of the document, " a homographic template determining a homographic application connecting the positions of the characters of the machine readable zone (MRZ) of the acquired image and the positions of the characters of the machine readable zone (MRZ) of the reference image of the initial reference document template. Such a template gives the possibility of having a plane of a planar surface as seen by a camera matched with the plane of the same surface in another image, * an interpolation template determined by means of a reverse distance weighting algorithm and/or by interpolation with splines.

In a second embodiment, the analysis method is carried out in an iterative way. A global geometrical transformation as described above may then be either determined or not during a first iteration of the method and then such a global geometrical transformation may be determined during another iteration from the fine geometrical transformation of the document obtained in the estimation step E8 during a preceding iteration. In both of these embodiments, the computer of the data processing device is then configured for applying for each region of interest stored of each of said at least one selected initial reference document template, a step E4 for determining the region of interest of the corresponding acquired image, by projection into the acquired image of said stored region of interest depending on the determined global geometrical transformation. Such a projection may be added with a certain margin in order to overcome the lack of accuracy of the global geometrical transformation. For each selected initial reference document template, the zones of the acquired image containing points of interest corresponding to those of the template may thus be determined. The accuracy of this determination as compared with a given document template is improved by taking into account the global transformation of the document to be analyzed, particularly if the latter was determined during the step for determining a transformation E3 relatively to the reference image of this initial reference document template. Thus, if a global transformation has been determined during the transformation determination step E3 for each selected initial document template, the transformation used during the step for determining a region of interest E4 is the one corresponding to the appropriate initial reference document template.

The analysis device may also compute an acquired image corrected from the acquired image of the deformed document and from said estimated fine geometrical transformation, for example for archiving, displaying or human analysis purposes. Such a correction carried out from said fine geometrical transformation is thus more accurate than a correction made from a global transformation.

Such a method thus gives the possibility of analyzing the contents of the regions to be analyzed in the image of a document which may be deformed, from of an image of the document optionally shot at a distance, without requiring mechanical or software smoothing, without this identification being sensitive to the image shooting conditions of the document, and without requiring expensive computation of a complex transformation template reproducing the whole of the local deformations of the document to be analyzed.

Claims

UUJ(M2.)4I 28 CLAIMS

1. A method for analyzing contents of at least one image of a 5 structured document which may be deformed, said method being applied by an analysis device configured so as to be connected to a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least 10 one region to be analyzed, said points of interest of a given reference document template having a fixed location in any structured document according to said given reference document template, each reference document template being further associated with a reference image of a document according to said 15 template, the method comprising steps of: - acquiring said at least one digital image of said document; - selecting at least one initial reference document template from among the reference document templates stored in the storage 20 device; - for each stored region of interest of each of said selected initial reference document templates: * extracting points of interest in the acquired image, * matching the extracted points of interest with the stored 25 points of interest of said region of interest of the initial reference document template; - selecting a final reference document template from among said initial reference document templates selected depending on said matches; 30 - estimating a fine geometrical transformation between the acquired image and the reference image associated with the final reference template, taking into account the local deformations of

00J(M2.)4I

29

the document, from said matches, wherein the fine geometrical transformation is estimated by means of a reverse distance weighting algorithm or by interpolation with spines; - determining at least one region to be analyzed of said acquired 5 image by projecting said at least one region to be analyzed of the final reference document template in said acquired image depending on said estimated fine geometrical transformation; and - analyzing the contents of said at least one region to be analyzed of the determined acquired image. 10

2. The method according to claim 1 wherein the acquisition step is carried out at a distance.

3. The method according to any one of the preceding claims wherein, 15 a region to be analyzed of the final reference document template comprising a heading, the step for analyzing the contents of the region to be analyzed of the acquired image determined by projecting said region to be analyzed of the template further comprises an association of the analyzed contents with said 20 heading.

4. The method according to any one of the preceding claims further comprising: - a step for determining at least one global geometrical 25 transformation between the acquired image of said document and at least one reference image associated with a selected initial reference document template, and - for each stored region of interest of each of said selected initial reference document templates, a step for determining a region 30 of interest of the corresponding acquired image, by projection in the acquired image of said stored region of interest depending on the determined global geometrical transformation,

00J(M2.)4I

30

wherein the step for extracting points of interest comprises the extraction of said points of interest in the region of interest of the determined acquired image.

5 5. The method according to any one of the preceding claims, wherein said at least one initial reference document template is selected from among the reference document templates stored in the storage device depending on a machine readable zone (MRZ) of the acquired image. 10

6. The method according to any one of claims 1 to 4, wherein the selection of said at least one initial reference document template from among the reference document templates stored in the storage device comprises the determination of points of interest in 15 the acquired image, the matching of the determined points of interest with the points of interest of at least one reference document template stored in the storage device and the determination of said at least one initial reference document template depending on said matching. 20

7. The method according to any one of claims 4 to 6, wherein said global geometrical transformation is determined depending on the machine readable zones (MRZ) of the acquired image and of an initial reference document template. 25

8. The method according to any one of the preceding claims, wherein said step for selecting the final reference document template comprises the computation for each initial reference document template of a matching score depending on the matches carried out 30 for said initial reference document template, and the selection as a final reference document template of the initial reference document template having the maximum matching score.

UUJ(M2.)4I

31

9. The method according to any one of the preceding claims, wherein said document is from among an identity document, an invoice, a documentary proof and a form. 5 10. The method according to any one of the preceding claims, further comprising the computation of an acquired image corrected from the acquired image of the deformed document and from said estimated fine geometrical transformation. 10 11. The method according to any one of the preceding claims, wherein at least one region to be analyzed of the acquired image comprising a string of characters is read by an optical character recognition (OCR) method. 15 12. A computer program product comprising code instructions for executing an analysis method according to any one of the preceding claims when the program is executed by a processor.

20 13. A device for analyzing contents of at least one image of a structured document which may be deformed, the device comprising a data processing device configured so as to be connected to a storage device storing: reference document templates each associated with regions of interest and with points 25 of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document template having a fixed location in any document structured according to said given reference document template, each reference document template being further 30 associated with a reference image of a document according to said template,

00J(M2.)4I

32

and a device for capturing said at least one digital image of said document, the data processing device configured for performing steps of: - selecting at least one initial reference document template from 5 among the reference document templates stored in the storage device; - for each stored region of interest of each of said selected initial reference document templates: • extracting points of interest in the acquired image, and 10 • matching the extracted points of interest with the stored points of interest of said region of interest of the initial reference document template, - selecting a final reference document template from among the initial reference document templates selected depending on said 15 matches; - estimating a fine geometrical transformation between the acquired image and the reference image associated with the final reference template taking into account the local deformations of the document, from said matches, wherein the fine geometrical 20 transformation is estimated by means of a reverse distance weighting algorithm or by interpolation with splines; - determining at least one region to be analyzed of said acquired image by projecting said at least one region to be analyzed of the final reference document template in said acquired image 25 depending on said estimated fine geometrical transformation; and - analyzing the contents of said at least one region to be analyzed of the determined acquired image.

14. A system for analyzing contents of at least one image of a 30 structured document which may be deformed, the system comprising:

00J(M2.)4I

33

- a storage device storing: reference document templates each associated with regions of interest and with points of interest comprised in said regions of interest, and with at least one region to be analyzed, said points of interest of a given reference document 5 template having a fixed location in any document structured according to said given reference document template, - the analysis device according to claim 13 configured so as to be connected to said storage device.

10

15. A device for taking images from a distance comprising an analysis system according to claim 14.

16. A mobile terminal comprising a device for taking images from a distance according to claim 15.