CN116740741A

CN116740741A - Product information extraction method and device

Info

Publication number: CN116740741A
Application number: CN202210200742.3A
Authority: CN
Inventors: 曹文龙; 洪科元
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-03-01
Filing date: 2022-03-01
Publication date: 2023-09-12

Abstract

The embodiment of the application relates to the technical field of computers, and discloses a method and a device for extracting product information, wherein the method comprises the following steps: acquiring target text information in a target image and the context relation of each character in the target text information; calling a trained named entity recognition model, and processing the target text information and the context relation of each character to obtain an entity label of each character; determining at least one entity tag group in the entity tags of each character, wherein the tag types of the entity tags contained in each entity tag group are the same; and combining the characters corresponding to the entity tags in the same entity tag group to obtain product information. By adopting the embodiment of the application, the product information in the image can be accurately extracted through big data training, thereby improving the input efficiency of the product information.

Description

Product information extraction method and device

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for extracting product information.

Background

When a merchant starts to use a new system such as a bill, comment popularization and the like, as the merchant originally uses the product catalogues under the online condition, the patterns and the designs of the product catalogues are various, and the product information is various, if the merchant directly extracts the product information in a text recognition mode on the image containing the product catalogues, the extracted product information is often wrong; meanwhile, if the product information is manually input, the time and the labor are wasted under the condition that a large number of products exist. Therefore, how to accurately extract the product information in the image, so as to improve the input efficiency of the product information, is a technical problem which needs to be solved at present.

Disclosure of Invention

The embodiment of the application provides a method and a device for extracting product information, which can accurately extract the product information in an image, thereby improving the input efficiency of the product information.

In one aspect, an embodiment of the present application provides a method for extracting product information, including:

acquiring target text information in a target image and a context relation of each character in the target text information;

invoking a trained named entity recognition model, and processing the target text information and the context relation of each character to obtain an entity label of each character;

determining at least one entity tag group in the entity tags of the characters, wherein the entity tags contained in each entity tag group have the same tag type;

and combining the characters corresponding to the entity tags in the same entity tag group to obtain product information.

In one aspect, an embodiment of the present application provides a product information extraction apparatus, including an acquisition unit and a processing unit, where:

the acquisition unit is used for acquiring target text information in a target image and the context relation of each character in the target text information;

The processing unit is further used for calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character;

the processing unit is further configured to determine at least one entity tag group from the entity tags of the characters, where tag types of the entity tags included in each entity tag group are the same;

the processing unit is further configured to combine characters corresponding to each entity tag in the same entity tag group to obtain product information.

In one aspect, an embodiment of the present application provides an electronic device, where the electronic device includes an input interface and an output interface, and further includes:

a processor adapted to implement one or more instructions; the method comprises the steps of,

a computer storage medium storing one or more instructions adapted to be loaded by the processor and to perform the method of integrity checking of text data described above.

In one aspect, embodiments of the present application provide a computer storage medium having stored therein computer program instructions for executing the above-described product information extraction method when executed by a processor.

In one aspect, embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium; the processor of the electronic device reads the computer instructions from the computer readable storage medium, the processor executing the computer instructions, which when executed by the processor, are for performing the product information extraction method described above.

In the embodiment of the application, the target text information in the target image and the context relation of each character in the target text information are firstly obtained; calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character; then, determining at least one entity tag group in the entity tags of the characters, wherein the tag types of the entity tags contained in the entity tag groups are the same; and finally, combining the characters corresponding to the entity tags in the same entity tag group to obtain product information. According to the embodiment of the application, the trained named entity recognition model is called, each character in the target text information is marked, the entity label of each character is determined, and then the characters corresponding to each entity label belonging to the same entity label group are combined, so that the product information is obtained; the target text information is processed by taking the characters as units, and the corresponding characters are combined to obtain the product information, so that the problems of inaccurate identification and the like caused by text content adhesion, various patterns, designs and other factors of the product information in the target image when the product information in the target image is directly identified by a text identification mode are avoided, the product information in the image can be accurately extracted, and the efficiency of a product information input system is improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of a product information extraction system according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a product information extraction method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of acquiring target text information according to an embodiment of the present application;

FIG. 4 is a schematic diagram of another method for obtaining target text information according to an embodiment of the present application;

FIG. 5 is a schematic diagram of still another method for obtaining target text information according to an embodiment of the present application;

FIG. 6 is a schematic diagram of determining an entity tag group according to an embodiment of the present application;

FIG. 7 is a flowchart of another method for extracting product information according to an embodiment of the present application;

FIG. 8 is a schematic diagram of an encoding process provided by an embodiment of the present application;

FIG. 9 is a schematic illustration of a Span provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a process for extracting product information according to an embodiment of the present application;

FIG. 11 is a flowchart of another method for extracting product information according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of a product information extraction device according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

In order to accurately extract product information in an image and improve the input efficiency of the product information, the embodiment of the application provides a product information extraction scheme, which comprises the steps of firstly acquiring target text information in a target image and the context relation of each character in the target text information; calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character; then, at least one entity tag group is determined in the entity tags of all the characters, and the tag types of the entity tags contained in all the entity tag groups are the same; and finally, combining the characters corresponding to the entity tags in the same entity tag group to obtain product information. The target image refers to an image containing product information, and the target image may be an image acquired by the electronic device in response to a shooting operation of a user or an image selected from a plurality of images in an image database in response to a selection operation of the user. For example, the target image may be an image of a menu, an image of a price list of items of a gym, etc.

In one embodiment, the above product information extraction scheme may be performed by a terminal device, where the terminal device may include any one or more of a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart car, and a smart wearable device. The terminal equipment firstly acquires a target image, then acquires target text information in the target image and context relation of each character in the target text information; the terminal equipment invokes the trained named entity recognition model, and the target text information and the context relation of each character are processed to obtain entity labels of each character; then, the terminal equipment determines at least one entity tag group in the entity tags of each character, wherein the tag types of the entity tags contained in each entity tag group are the same; and finally, the terminal equipment combines the characters corresponding to the entity tags in the same entity tag group to obtain product information. Alternatively, the above product information extraction scheme may be performed by a server, where the server may be a stand-alone server, a cloud server, a server cluster, or a distributed system, and the like, which is not limited herein.

In another embodiment, the product information extraction scheme may be run in a product information extraction system, where the product information extraction system may include a terminal device and a server, and specifically, the terminal device may upload a target image to the server in response to an upload operation of a user, and the server may first obtain target text information in the target image and context relations of each character in the target text information, and then call a trained named entity recognition model, and process the target text information and the context relations of each character to obtain an entity tag of each character; then the server determines at least one entity tag group in the entity tags of each character, wherein the tag types of the entity tags contained in each entity tag group are the same; and finally, the server combines the characters corresponding to the entity tags in the same entity tag group to obtain product information, and sends the product information to the terminal equipment.

The above-described product information extraction system may be referred to as shown in fig. 1, and the product information extraction system shown in fig. 1 may include a terminal device 101 and a server 102. The terminal device 101 may include any one or more of a smart phone, a tablet computer, a notebook computer, a desktop computer, an intelligent vehicle, and an intelligent wearable device. The server 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligence platform. The terminal device 101 and the server 102 may be directly or indirectly connected through wired or wireless communication, which is not limited herein.

Based on the above product information extraction scheme, the embodiment of the application provides a product information extraction method. Referring to fig. 2, a flow chart of a product information extraction method according to an embodiment of the present application is shown. The product information extraction method shown in fig. 2 may be performed by a server or a terminal device. The product information extraction method shown in fig. 2 may include the steps of:

s201, acquiring target text information in a target image and the context relation of each character in the target text information.

In the embodiment of the application, the target text information refers to text information used for indicating the content of a product in the target image. Illustratively, the target image is an image containing a Chinese meal menu, and then the target text information can be the name of a dish in the Chinese meal menu, the price, the specification, the currency and the like of the dish, for example, the target text information is '10 yuan for braised eggplant/32 yuan for steamed crucian carp/dish'; and text information such as restaurant history introduction in the menu does not belong to the target text information. Further, the context of each character in the target text information refers to the position of the character in the target text information, such as which characters in the target text information precede character a, which characters follow character a, and so on. The context relation of each character in the target text information may be obtained by performing semantic analysis on the target text information, or alternatively, may be obtained by sorting each character in the target text information according to the reading sequence by using the position information of each character in the target image, which is not limited herein.

In one embodiment, the method for acquiring the target text information may be: 1) Performing text detection processing on the target image to obtain at least one text region and position information of each text region in the target image; 2) Sequencing at least one text region based on the position information of each text region in the target image to obtain sequenced text regions; 3) And performing splicing processing on characters contained in each text region in the sequenced text regions to obtain target text information.

Specifically, by sequencing at least one text region, the character sequence of the target text information obtained by subsequent splicing can be basically consistent with the character sequence in the target image, and the semantics of the target text information can be basically consistent with the semantics in the target image, so that the named entity recognition model after subsequent training can be favorable for marking the entity labels of all the characters more accurately.

Alternatively, the text detection processing may be performed on the target image by using an optical character recognition technology (Optical Character Recognition, OCR) to obtain at least one text region, or the text detection processing may be performed on the target image by using a trained text detection model to obtain at least one text region. By way of example, the text detection model may be a deep learning model such as DBNet (a differentiable binary segmentation network for text detection), CTPN (a network for text detection including convolutional neural network and cyclic neural network), segLink (a convolutional neural network capable of detecting text with a rotation angle), and the like, and different models may be flexibly selected based on different requirements, which is not limited herein. The training process of the text detection model is a technical means familiar to those skilled in the art, and is not described herein.

In one embodiment, the location information may include an abscissa and an ordinate of a region center point of each text region, and the method for sorting at least one text region based on the location information of each text region in the target image may be that: sequencing each target text region according to the sequence from small to large of the ordinate of the region center point of each text region; if the ordinate of the region center points of the text regions is the same or the difference between the ordinate of the region center points of the text regions is small, the text regions can be determined to be in the same row, then the target text regions in the text regions can be ordered according to the order from small to large of the abscissa of the region center points of the text regions, and finally the ordered target text regions are obtained.

For example, referring to fig. 3, a schematic diagram of acquiring target text information is shown, wherein 6 text regions are total in the target image, and the region center point coordinates of the text region a are (200 ); the region center point coordinates of the text region B (200,305), the region center point coordinates of the text region C (198,399), the region center point coordinates of the text region D (400,210), the region center point coordinates of the text region E (406,295), and the region center point coordinates of the text region F (405,390).

Then, a text region having a difference in the ordinate of the region center point of less than 10 pixels is set as a text region having the same ordinate, and then it can be determined that the ordinate of the region center point of the text region A, D is the same, the ordinate of the region center point of the text region B, E is the same, and the ordinate of the region center point of the text region C, F is the same. Comparing the abscissa of the region center point of text region A, D, the abscissa of the region center point of text region B, E, and the abscissa of the region center point of text region C, F, the resulting ordered target text region is: text area a, text area D, text area B, text area E, text area C, text area F. And finally, splicing the characters contained in each text region in the sequenced text regions to obtain target text information 302 '10.00 yuan for braised eggplant per serving of steamed crucian carp 32.00 yuan per jin of meat powder vermicelli 20.00 yuan per serving'.

In one embodiment, the location information of each text region in the target image includes an abscissa and an ordinate of each text region in the target image; then, based on the position information of each text region in the target image, the at least one text region is ordered, and the manner of obtaining the ordered text region may be:

1) And determining at least one text region queue based on the ordinate of each text region, wherein the ordinate of the text region in a target text region queue in the at least one text region queue is smaller than the ordinate of the text region in a next text region queue of the target text region queue, the target text region queue is any text region queue in the at least one text region queue, and the ordinate of each text region in the same text region queue is matched.

Specifically, the ordinate matching of each text region in the same text region queue may be that the ordinate of each text region is the same, or the difference between the ordinate of each text region is within a first preset threshold, which is not limited herein. The first preset threshold may be a pixel value of 20 pixels, 10 pixels, or the like, or may be a unit of 10 mm, 1 cm, or the like, which is not limited herein.

Optionally, the determining at least one text region queue based on the ordinate of each text region may be: and determining that the text areas with the same ordinate or the difference value between the ordinate and the ordinate within a first preset threshold value are the same text area queue based on the ordinate of each text area, so as to obtain at least one text area queue. And then, according to the ordinate of the text region in each text region queue or the average ordinate of the text regions in each text region queue, arranging each text region queue so that the ordinate of the text region in the target text region queue in at least one text region queue is smaller than the ordinate of the text region in the next text region queue in the target text region queue.

Optionally, at least one text region queue may also be determined based on the abscissa and the ordinate of each text region: firstly, determining a text region with the largest abscissa or the smallest abscissa in the target image as a first text region based on the abscissa of each text region, wherein if the difference value between the abscissa of the text region M and the abscissa of the first text region is smaller than a second preset threshold value, the text region M can be determined to be the first text region; and then, determining the text area with the same ordinate as the first text area or the difference value between the ordinates within a preset threshold value as a second text area in the text areas except the first text area in at least one text area, and finally determining the first text area and the second text area as the same text area queue. The second preset threshold may be a pixel value of 20 pixels, 10 pixels, or the like, or may be a unit of 10 mm, 1 cm, or the like, which is not limited herein.

2) Traversing at least one text region queue, and sorting all text regions in the currently traversed text region queue based on the abscissa of all text regions in the currently traversed text region queue to obtain sorted text regions. Specifically, the text regions in the currently traversed text region queue are ordered according to the descending order of the abscissa of the text regions in the currently traversed text region queue.

For example, referring to fig. 4, another schematic diagram of acquiring target text information is shown, and after text detection processing is performed on a target image 401, 12 text regions, namely text regions a-L, are obtained; setting the first preset threshold to be 10 pixels, then according to the ordinate of each text region, it may be determined that the text region A, B, C, D is the same text region queue, the text region E, F, G, H is the same text region queue, and the text region I, J, K, L is the same text region queue; based on the ordinate of the text area a, the text area E and the text area I, it may be determined that the text area queue containing the text area a is a first text area queue, the text area queue containing the text area E is a second text area queue, and the text area queue containing the text area I is a third text area queue.

Then traversing the three text area queues according to the sequence from the first text area queue to the third text area queue, and determining that the text area sequence of the first text area queue is a text area A, a text area B, a text area C and a text area D according to the sequence from the small to the large of the abscissa when traversing to the first text area queue; when traversing to the second text area queue, determining the text area sequence of the second text area queue as a text area E, a text area F, a text area G and a text area H according to the descending order of the abscissa; when traversing to the third text area queue, the text area sequence of the third text area queue can be determined to be a text area I, a text area J, a text area K and a text area L according to the descending order of the abscissa. Thus, the finally obtained ordered text regions are text region a, text region B, text region C, text region D, text region E, text region F, text region G, text region H, text region I, text region J, text region K, text region L. And finally, splicing characters contained in each text region in the sequenced text regions to obtain target text information 402 '10.00 yuan of braised eggplant per 40.00 yuan of cold-mixed donkey meat per 8.00 yuan of dish cold-mixed cucumber per 32.00 yuan of steamed crucian per jin of fried vegetables 12.00 yuan per 20.00 yuan of minced meat vermicelli per part'.

In one embodiment, the method for sorting at least one text region based on the location information of each text region in the target image may be that: invoking a word order correction model, and carrying out word order correction processing on at least one text region based on the position information of each text region to obtain a processed text region sequence, wherein the word order between characters in any text region in the processed text region sequence and characters in a previous text region in any text region and characters in a next text region in any text region meets a preset word order rule; and taking the processed text region sequence as the ordered text region.

Specifically, the preset word order rule may be a word order rule set manually according to the word order of the characters in the product information, or may be a word order rule summarized by the system according to the word order of the characters in the product information, which is not limited herein. For example, when the target image is an image containing a menu, a preset word order rule may be set to alternately show a menu name, a price, and a specification.

In addition, the word order correction model may be obtained by training a machine learning model such as a Recurrent Neural Network (RNN) through a large amount of text information related to a product. The process of calling the language order correction model and carrying out language order correction processing on at least one text area based on the position information of each text area to obtain the processed text area sequence specifically comprises the following steps: the method comprises the steps that firstly, characters in each text area are spliced based on the position information of each text area through a word order correction model to obtain spliced text information, then, word order adjustment is conducted on the characters in the spliced text information through the word order correction model to obtain adjusted spliced text information, and therefore the adjusted spliced text information meets preset word order rules; finally, sequencing at least one text region according to the characters in the adjusted spliced text information, so as to obtain a processed text region sequence; and finally, taking the processed text region sequence as the sequenced text region.

For example, referring to fig. 5, there is shown a schematic diagram of still another method for obtaining target text information, after performing text detection processing on a target image 501, 11 text regions can be obtained, then, a word order correction model is called to splice characters in each text region based on each position information, so as to obtain spliced text information 502 "pizza super-honor pizza durian pizza steak/Yi-face Orleans barbecue pizza Australia cold steak selective steak classical meat paste Yi-face spanish flavor Yi-flavor snack spelling"; finally, the menu is classified according to the category of the dish and the name of the dish, so that the word order correction model can carry out word order adjustment on the spliced text information 502 according to the category of the dish and the order of the names of the dishes belonging to the category, and finally, the adjusted spliced text information 503 ' pizza super-honor pizza, pizza otter, ying surface Australian cold steak, and selected steak classical beef sauce Ying surface Spanish flavor Italian flavor snack splice ', can be obtained, thereby determining that the ordered text area is text area a, text area b, text area c, text area e, text area d, text area f, text area g, text area i, text area j and text area k '.

Optionally, in order to avoid that the text region is inclined and is difficult to sort when the text detection processing is performed on the shot target image due to the inclination of the page during the process of shooting the page such as the menu, the item price list and the like, the angle correction may be performed on the target image before the text detection processing is performed on the target image, so that the page such as the menu, the item price list and the like in the target image is adjusted to be in a forward direction, that is, the page is in a non-inclined state.

Alternatively, in some practical scenarios, the characters in the pages such as the menu, the item price list, etc. may be vertically arranged, so that in the text detection process of the target image, the vertically arranged characters may be converted into horizontally arranged characters, and then the text regions may be ordered in a manner based on the abscissa and the ordinate of each text region and a manner of calling the trained word order correction model, which are mentioned in the above embodiments.

In addition, the arrangement mode of the texts may be determined in the text detection process of the target image, and if the arrangement mode is a lateral arrangement mode, the text regions may be ordered by referring to the mode based on the abscissa and the ordinate of each text region mentioned in the above embodiment and the mode of calling the trained word order correction model.

If the arrangement mode is vertical arrangement, based on the position information of each text region in the target image, at least one text region is ordered, and the mode of obtaining the ordered text region can be as follows: determining at least one text region queue based on the abscissa of each text region, wherein the abscissa of a text region in a target text region queue in the at least one text region queue is smaller than the abscissa of a text region in a next text region queue of the target text region queue, the target text region queue is any text region queue in the at least one text region queue, and the abscissas of all text regions in the same text region queue are matched; traversing at least one text region queue, and sorting all text regions in the currently traversed text region queue based on the ordinate of all text regions in the currently traversed text region queue to obtain sorted text regions.

Meanwhile, the method for sorting at least one text region based on the position information of each text region in the target image to obtain the sorted text region may further be: invoking a word order correction model, and carrying out word order correction processing on at least one text region based on the position information of each text region to obtain a processed text region sequence, wherein the word order between characters in any text region in the processed text region sequence and characters in a previous text region in any text region and characters in a next text region in any text region meets a preset word order rule; and taking the processed text region sequence as the ordered text region.

The text regions may be ordered in the manner of the abscissa and ordinate of each text region and the manner of calling the trained word order correction model, which are mentioned in the above embodiments.

S202, calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character.

In the embodiment of the application, the trained named entity recognition model comprises a trained encoder and a trained labeling module; the method for calling the trained named entity recognition model to process the context relation of the target text information and each character to obtain the entity label of each character may be as follows:

1) Invoking a trained encoder:

(1) processing the target text information and the context relation of each character to determine the feature vector of each character;

(2) determining a first character positioned before each character and a second character positioned after each character in the target text information;

(3) and carrying out coding processing on each character based on the feature vector of each character, the feature vector of the first character and the feature vector of the second character to obtain each coded character.

Wherein steps (1), (2) and (3) are all performed by the trained encoder.

Specifically, the trained encoder may be a trained BERT model (Bidirectional Encoder Representations from Transformer, bi-directional encoder characterization based on a transducer), a trained XLNet (autoregressive and self-encoding model), a LayoutLM (Pre-training of Text and Layout for Document Image Understanding), and other Pre-training language models, where advantages of different Pre-training language models are different, such as the BERT model is simpler, the LayoutLM combines spatial information, and may have better extraction effect in more complex scenarios, and different Pre-training language models may be selected based on different requirements, which is not limited herein. The feature vector refers to a vector used for representing the feature of each character, and the text information extracts the feature vector of the character before being input into the pre-training language model.

In particular implementations, when the trained encoder is a trained BERT model, the feature vectors for each character include a word vector (token vectors), a segment vector (segment embeddings), and a position vector (position embeddings) for each character. Specifically, the word vector of each character can be determined by querying a word vector table, the segment vector is determined based on the global semantics of each character and the target text information, and the position vector is determined by the context relation of each character because the semantics carried by the characters appearing in different positions of the text have differences. The determining process of the word vector, the segment vector and the position vector is a technical means commonly used by those skilled in the art, and is not described herein.

Because a Self-Attention mechanism is used in the pre-training language model, after the feature vector of each character is determined, the trained BERT model needs to determine a first character before each character and a second character after each character in the target text information through the context of each character; and then enhancing the semantic vector representation of each character by the feature vector based on each character, the feature vector of the first character and the feature vector of the second character, thereby completing the encoding processing of each character and obtaining each encoded character. The process of coding each character by the trained BERT model is a technical means familiar to those skilled in the art, and is not described herein.

2) And calling a trained labeling module, and labeling each coded character to obtain the entity label of each character.

In the embodiment of the present application, the trained labeling module may be a labeling module obtained by training sequence labeling models such as a hidden markov model (Hidden Markov Model, HMM), a maximum entropy model (The Maximum Entropy Principle, maxEnt), a BiLSTM-CRF model (Bi-directional Long Short-Term Memory and Conditional Random Field), and the like. In addition, the physical label of each character may include a label position and a label type, and the physical label of each character may be different according to a preset type, which is not limited herein. The preset type refers to the type of the product, if the product is a braised eggplant, the preset type is a vegetable name, the product is prasugrel, and the preset type is a body-building project.

Optionally, in order to further improve the efficiency of the product information input system, the entity tag may be set according to the product information required by the input system, so that the information extracted according to the entity tag may be directly used by the system. For example, when the product is a dish, the dish name, the dish price, the monetary units, and the measurement units of the dish are generally required, so that the entity tags may be set to include B-term, I-term, B-price, I-price, B-currency, I-currency, B-unit, I-unit, and O. Wherein term is tag type, and refers to dish name such as shredded fish, crystal sugar snow pear, honey pomelo tea, etc.; the price is the type of label and refers to the price of dishes; currency is a tag type, referring to monetary units such as meta, dollar, yen, etc.; unit is a label type, referring to a unit of measurement, such as jin, dish, part, 300g, position, etc.; B. i, O is the label position, B is the beginning, I is the interior, the character with the entity label B-term is the first character of a certain vegetable name, and the character with the entity label I-term is the internal character (i.e. the character except the first character) of a certain vegetable name; o refers to non-physical information, i.e. characters not belonging to any one of a dish name, a dish price, a monetary unit, and a metering unit.

Optionally, the trained named entity recognition model may also be a preset database in which the pre-arranged fields and field types are stored correspondingly. After the target text information is obtained, determining at least one field in the target text information according to the context relation between the target text information and each character, and searching a field type matched with each field in a preset database, wherein the field type is the entity label of each character in each field; finally, the fields belonging to the same product are determined and combined. The field belonging to the same product can be determined by judging through a rule set by people. For example, it may be set that: if the field types of two continuous fields in the target text information are all dish names, the two dish names are determined to belong to two dishes, so that the two fields are not combined. If the field types of two continuous fields in the target text information are both dish prices, the two fields are combined. If the field types of two consecutive fields in the target text information are the dish name and the dish price, respectively, it can be determined that the two fields belong to the same dish, and thus the two fields can be combined.

S203, determining at least one entity tag group in the entity tags of the characters, wherein the entity tags contained in each entity tag group have the same tag type.

S204, combining the characters corresponding to the entity tags in the same entity tag group to obtain product information.

In the embodiment of the present application, the manner of determining at least one entity tag group in the entity tags of each character may be: 1) If a target entity label exists in the entity labels of all the characters, constructing the entity labels with the same label type as the target entity label and the target entity label into an entity label group; 2) If a plurality of target entity labels exist in the entity labels of the characters, determining a plurality of label clusters, wherein each label cluster comprises an entity label between any target entity label in the entity labels of the characters and the next target entity label of any target entity label; and constructing the entity tags with the same tag type as any target entity tag in each tag cluster and any target entity tag into an entity tag group.

The method for determining the target entity tag in the entity tag of each character may be determining through the tag position of the entity tag of each character, and if the tag position of any entity tag is a preset tag position, determining that any entity tag is the target entity tag.

In the embodiment of the application, the product information refers to information related to a product. If the target image is a menu, the product in the menu is a dish, and the product information can comprise a dish name, a dish price, currency, specification and the like.

For example, referring to fig. 6, fig. 6 shows a schematic diagram for determining an entity tag group, where the tag positions may be preset to B and O. After obtaining the text information 601 'cold donkey meat_60 yuan/jin', inputting the text information 601 and the context relation of each character in the target text information into a trained named entity recognition model (consisting of a BERT model and a BiLSTM-CRF model), and outputting entity labels of each character by the trained named entity recognition model. The cool entity label of the character is B-term, the mixing entity label of the character is I-term, the donkey entity label of the character is I-term, the meat entity label of the character is I-term, the_meat entity label of the character is O, the 6 entity label of the character is B-price, the 0 entity label of the character is I-price, the meta entity label of the character is B-currency, the/' entity label of the character is O, and the jin entity label of the character is B-unit.

After the entity tags for each character are obtained, it may be determined that there are a plurality of target entity tags, respectively, "B-term", "O", "B-price", "B-currency", "O", and "B-unit". Then, determining the label clusters, wherein the next target entity label of the B-term is O, so that the B-term, the I-term and the I-term form one label cluster, and other label clusters can be obtained in the same manner and are not repeated herein. Then, since the tag types of all the entity tags of the tag clusters consisting of "B-term", "I-term" and "I-term" are "term", it is possible to determine that "B-term", "I-term" and "I-term" are the entity tag group 1, and similarly, the entity tag groups 2 to 6 can be obtained. And finally, combining the characters corresponding to the entity tags in the same entity tag group to obtain the product information. The product information obtained by combining the characters corresponding to the respective entity tags in the entity tag group 1 is 'cold donkey meat'. The final product information comprises 'cold donkey meat', '60', 'Yuan', '/' and 'jin'.

Alternatively, since O refers to non-entity information, that is, unimportant information that does not have great significance to product information, the tag position may be preset to B. Then it may be determined that the plurality of target entity tags are "B-term", "B-current", and "B-unit", respectively, and then it is determined that the next target entity tag of "B-term" is "B-term", so that "B-term", "I-term", and "O" constitute one tag group, and then it may be determined that "B-term", "I-term", and "I-term" are one entity tag group because the tag type of "O" is different from the tag types of "B-term", "I-term", and "I-term". Similarly, it may be determined that "B-price" and "I-price" are an entity tag group, "B-currency" is an entity tag group, and "B-unit" is an entity tag group. The final product information may then include "cold donkey," 60, "" yuan, "" jin.

Optionally, when outputting the product information, the product information may be correspondingly input into the system according to the label type of the entity label forming the character of the product information. Specifically, the tag type of the physical tag of the character of the cold donkey meat is the vegetable name, so that the cold donkey meat can be correspondingly input into the vegetable name part of the system to obtain the vegetable name: donkey meat is dressed in a cold sauce. Similarly, the product information of the input system finally obtained may be "vegetable name: cold-mixing donkey meat; dish price: 60; currency unit: a meta-element; measurement unit: weight of a jin.

Based on the above product information extraction scheme, the embodiment of the application provides another product information extraction method. Referring to fig. 7, a flow chart of another product information extraction method according to an embodiment of the present application is shown. The product information extraction method shown in fig. 7 may be performed by a server or a terminal device. The product information extraction method shown in fig. 7 may include the steps of:

s701, acquiring target text information in a target image and the context relation of each character in the target text information.

The specific embodiment of step S701 may refer to the specific embodiment of step S201, which is not described herein.

S702, calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character.

In the embodiment of the application, the process of training the named entity recognition model can be specifically as follows: 1) Acquiring a first training sample, wherein the first training sample comprises first training text information and target entity labels of all training characters in the first training text information; 2) Carrying out semantic analysis on the first training text information to obtain the context relation of each training character in the first training text information; 3) Invoking a named entity recognition model, and processing the first training text information and the context relation of each training character in the first training text information to obtain a predicted entity label of each training character in the first training text information; 4) Training the named entity recognition model based on the predicted entity label of each training character in the first training text information and the target entity label of each training character in the first training text information to obtain a trained named entity recognition model.

Specifically, the first training text information refers to text information related to a product, and target entity labels of all training characters in the first training text information can be obtained through artificial labeling. The training process for the named entity recognition model based on the predicted entity label of each training character in the first training text information and the target entity label of each training character in the first training text information is a training process with supervised learning, and the specific training process is a technical means familiar to those skilled in the art and is not repeated herein.

In one embodiment, a named entity recognition model includes an encoder and a labeling module; then the training of the named entity recognition model may also be preceded by:

1) Acquiring a second training sample, wherein the second training sample comprises second training text information, the second training text information comprises a plurality of training characters, and the information type of the second training text information is a preset type;

in particular, the second training text information refers to text information related to a product. The preset type can be the type of the product indicated by the text information related to the product contained in the second training text information, for example, the product is a braised eggplant, a cold donkey meat and the like, and then the preset type can be a vegetable name; the product is prasugrel, swimming, etc., the preset type can be a fitness project.

2) Carrying out semantic analysis on the second training text information to obtain the context relation of each training character in the second training text information;

specifically, semantic analysis is performed on the second training text information, and obtaining the context relationship of each training character in the second training text information is a common technical means, which is not described herein.

3) Generating a target attention parameter based on each training character in the second training text information and the context of each training character in the second training text information;

4) Updating the attention parameter in the encoder to be a target attention parameter to obtain an updated encoder;

specifically, there is a Self-Attention mechanism (Self-Attention) in the encoder, if the Attention parameters of two encoders in the Self-Attention mechanism are different, then when encoding the same text information, the Attention of the two encoders to the same character in the same text information is different, the assigned encoding weight and encoding logic are also different, and thus the encoded characters obtained by the two encoders are also different. Therefore, the attention parameter in the encoder is updated to be the target attention parameter, so that the updated encoder is obtained, the representation capability of the encoder in a specific scene can be improved, the character is encoded more accurately, and the robustness of the problem of the entity boundary in a Chinese scene is enhanced.

In a specific implementation, in a menu scene, training text information N with a menu type information type can be obtained, wherein the training text information N contains a great amount of information about dishes such as a dish name, a dish price, a dish specification and the like. Then, performing vocabulary enhancement training on the BERT model through training text information N, please refer to fig. 8, which shows a schematic diagram of coding processing, after determining feature vectors of each character/word in the training text information N, inputting the feature vectors of each character/word into Self-Attention, and performing ADD (weighting) and Norm (normalization) on the feature vectors of each character/word after processing based on the target Attention parameter; and then inputting the weighted and normalized data to a FNN (fully connected layer) for processing, and carrying out weighted and normalization again to obtain coded characters/words, and inputting the coded characters/words to a Linear layer or a CRF (conditional random field) for Linear processing.

The calculation process of the target Attention parameter in Self-Attention is specifically as follows:

(1) establishment ofSpanx _i And Spanx _j Is a matrix of relative distances of (a) and (b).

Specifically, span refers to text content representing a section of an input article in a named entity recognition task (NER). For example, referring to fig. 9, a schematic diagram of Span is shown, wherein the text content is "Yangtze river bridge in Nanjing city", and Span such as "Nanjing", "Nanjing city", "city long", "Yangtze river bridge", "bridge" can be determined by the text content.

Wherein x is _i Or x _j Can be used to indicate individual characters or individual words in the text information, as shown in fig. 8, x in the menu scenario _i Or x _j Refers to "cloud", "south", "Yunnan", "gap bridge wire", etc. In addition, as shown in FIG. 8, each Span is associated with a token, a head position (head), and a tail position (tail). the token represents a character or a word, and the head position and the tail position represent the positions of the first character and the last character of the token in the character string, respectively. Then, since the positional relationship between two different spans includes intersection (including), inclusion, and separation (separation), the Span can be established by _i And Spanx _j The relative distance matrix of (a) represents these relative positional relationships, and the formula of the dense matrix relative positional relationship matrix is as follows:

wherein,,represents x _i Head position and x of (2) _j Distance between head positions, < >>Represents x _i Head position and x of (2) _j The other two formulas are the same and are not repeated here. So that a relative distance matrix as shown at 801 can be obtained.

(2) Calculating a dense matrix: applying the activation function ReLU to the obtained relative distance matrix to obtain a dense matrix R _ij The formula is as follows:

wherein W is _r Refers to a parameter matrix which can be learned and can be used forAnd->Is calculated by taking the following formula>And->

Where k is a position-coded dimension index, where 2k represents even positions and 2k+1 refers to odd positions; the Bert model consists of multiple convectors, d _model The dimension output by the Transformers can be 512 or the like; general d _model ＝d _head ×H _num ，d _head Refers to the dimension of each head, H _num This is the concept introduced by the Transformers multi-headed mechanism for the number of heads.

(3) A target attention parameter (attention) is calculated, that is, the matching degree of a Query vector (Query) of a certain character/word and a Key vector (Key) corresponding to each character/word, wherein the Query vector and the Key vector can be obtained from the feature vector of the current character/word. The Attention matrix (Attention matrix) of the target Attention parameter can be obtained by the feature vector and the dense value R of the current character/word _ij The formula is as follows:

wherein,,are all parameters which can be learned, and the parameters are all parameters which can be learned,representation matrix W _q Transposed matrix of>Representation matrix->Transposed matrix of u ^T Representing the transposed matrix of u, v ^T Denote the transposed matrix of v, ">For the calculated Attention matrix in the current scene,/for the scene>Represents x _i Corresponding feature vector (Embedding vector), -a corresponding feature vector (Embedding vector) >Represents x _j Corresponding feature vector (Embedding vector), R _ij Is the dense matrix calculated in (2).

(4) And updating the attention parameters in the Bert model into target attention parameters to obtain an updated Bert model. The update process is formulated as follows, first by three matrices W _q ,W _k ,W _v Can be mapped to Q, K, V, then will be represented by the formulaCalculated A _i,j Updating to +.>Finally, updating the attention parameter in the BERT model into the target attention parameter through a normalization function:

[Q,K,V]＝E _x [W _q ,W _k ,W _v ]；

Attention(A,V)＝softmax(A)V；

where Q refers to the query vector, K refers to the key vector, V refers to the Value vector (Value), softmax is the normalization function,are all learnable parameter matrixes, E represents a table look-up or an upper layer Transformer layer, A _i,j Refers to the value of A at the position of coordinates (i, j), A is A _i,j A matrix of components.

Since the Bert model may be composed of a plurality of convertors, each Transformer layer refers to 8 Self-Attention layers (Self-Attention) contained by the multi-head Attention mechanism of the convertors, each of which outputs 64 layers in dimension, and thus each Transformer layer outputs 512 in feature dimension.

Then, optionally, invoking the named entity recognition model, and processing the first training text information and the context relation of each training character in the first training text information to obtain the predicted entity label of each training character in the first training text information may specifically be: 1) Invoking the updated Bert model, processing the first training text information and the context relation of each training character in the first training text information, and determining the feature vector of each training character; and determining a third character preceding each training character and a fourth character following each training character in the first training text information; and based on the feature vector of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information and the feature vector of the fourth character of each training character in the first training text information, carrying out coding processing on each training character in the first training text information to obtain each coded training character; 2) And calling a labeling module to label each coded training character to obtain the entity label of each training character in the first training text information.

S703, determining at least one entity tag group in the entity tags of each character, wherein the entity tags contained in each entity tag group have the same tag type.

S704, combining the characters corresponding to the entity tags in the same entity tag group to obtain product information.

The specific embodiments of step S703 to step S704 may refer to the specific embodiments of step S203 to step S204, which are not described herein.

S705, if target product information exists in the product information obtained based on the target text information, and target characters which are different from all characters in the preset character set exist in the target product information, searching the characters which are matched with the target character semantically in the preset character set.

S706, updating the target character in the target product information into the searched character to obtain updated target product information.

In the embodiment of the application, the character formats of all the characters in the preset character set are uniform. For example, all chinese characters in the preset character set a may be set, and then characters identified in the menu as to money "+," "RMB", "$" etc. may be target characters that are different from the respective characters in the preset character set a; then, the character matched with the character 'semantically' of the preset character set A is 'element', the character matched with the character 'RMB' semantically is 'element', and the character matched with the character '$' semantically is 'dollar'; the characters "@," RMB "and" $ "in the final product information are updated to" meta "," meta "and" dollar ", respectively.

In one embodiment, the method for searching the characters semantically matched with the target character in the preset character set may be to manually pre-arrange the preset character set, and mark similar words and keywords of each character in the preset character set to form a database, so that the characters semantically matched with the target character can be directly searched in the database. Or pre-training a synonym matching model, and searching the characters which are semantically matched with the target characters in a preset character set through the trained synonym matching model; the synonym matching model is a machine learning model, and the training process of the synonym matching model is a technical means familiar to those skilled in the art and is not described herein.

In one embodiment, it may also be: if the target product information exists in the product information obtained based on the target text information, the target product information does not meet the preset normalization rule, normalization processing can be performed on the target product information so as to update the target product information, and updated target product information is obtained.

Specifically, the preset normalization rule refers to a rule that can unify the formats of the characters in the product information, and may be set manually or set systematically, which is not limited herein. For example, in a menu scenario, preset normalization rules may be set to "0 for removing the tail of the price", "unified conversion of traditional Chinese characters into simplified form", "unified conversion of uppercase Chinese prices into Arabic numerals", and "unified currency representation into x-element/disc format", so that the formats of characters in the product information remain unified, and further the efficiency of the product information input system is improved.

For example, referring to fig. 10, a schematic diagram of a process of extracting product information is shown, and a preset normalization rule is set to "unified conversion of traditional Chinese characters into simplified Chinese characters". The menu image 1001 is uploaded to the product information extraction system 1002, and OCR recognition and word order correction are performed on the menu image 1001 by the product information extraction system 1002, so that the target text information W in the menu image 1001 and the context of each character in the target text information W are obtained. Then, the product information extraction system 1002 invokes a named entity recognition model to process the context relation between the target text information W and each character to obtain an entity tag of each character, and combines the characters corresponding to each entity tag determined to be in the same entity tag group to obtain product information, wherein all Chinese characters in the product information are complex at this time; finally, the product information extraction system 1002 performs normalization processing on the product information, that is, updates the traditional Chinese characters in the product information into simplified Chinese characters according to a preset normalization rule, and finally obtains updated product information 1003.

In the embodiment of the application, the target text information in the target image and the context relation of each character in the target text information are firstly obtained; calling the trained named entity recognition model, and processing the target text information and the context relation of each character to obtain the entity label of each character; and then, determining at least one entity tag group in the entity tags of each character, and combining the characters corresponding to each entity tag in the same entity tag group to obtain product information. Finally, if the product information contains target product information, and if the target product information contains target characters which are different from all characters in the preset character set, searching the characters which are semantically matched with the target characters in the preset character set; and updating the target character in the target product information into the searched character to obtain updated target product information. According to the embodiment of the application, the trained named entity recognition model is called, each character in the target text information is marked, the entity label of each character is determined, and then the characters corresponding to each entity label belonging to the same entity label group are combined, so that the product information is obtained; the target text information is processed by taking the characters as units, and the corresponding characters are combined to obtain the product information, so that the problems of inaccurate identification and the like caused by text content adhesion, various patterns, designs and other factors of the product information in the target image when the product information in the target image is directly identified by a text identification mode are avoided, the product information in the image can be accurately extracted, and the efficiency of a product information input system is improved. In addition, the method can unify the formats of the product information by searching the preset character set to search the characters semantically matched with the target characters and updating the target characters in the target product information into the searched characters, thereby being beneficial to further improving the efficiency of the product information input system.

Based on the product information extraction scheme and the product information extraction system, the embodiment of the application provides a product information extraction method. Referring to fig. 11, a flow chart of a product information extraction method according to an embodiment of the present application is shown. The product information extraction method shown in fig. 11 may be performed by the server and the terminal device shown in fig. 1. The product information extraction method shown in fig. 11 may include the steps of:

s1101, the terminal device transmits the target image to the server.

In the embodiment of the present application, the terminal device may send the target image to the server by wireless communication or wired communication, or may send the target image after encryption, or may send the target image through other methods, which is not limited herein.

S1102, the server acquires target text information in the target image and the context of each character in the target text information.

S1103, the server calls the trained named entity recognition model, and processes the context relation of the target text information and each character to obtain the entity label of each character.

S1104, the server determines at least one entity tag group in the entity tags of each character, wherein the entity tags contained in each entity tag group have the same tag type.

S1105, the server combines the characters corresponding to the entity tags in the same entity tag group to obtain product information.

The specific embodiments of step S1102 to step S1105 can be referred to the specific embodiments of step S201 to step S204, and are not described herein.

S1106, the server transmits the product information to the terminal device.

The manner of transmitting the product information to the terminal device in step S1106 is a technical means familiar to those skilled in the art, and is not described herein.

Based on the above product information extraction method embodiment, the embodiment of the application provides a product information extraction device. Referring to fig. 12, a schematic structural diagram of a product information extraction device according to an embodiment of the present application may include an acquisition unit 1201 and a processing unit 1202. The product information extraction apparatus shown in fig. 12 may operate as follows:

the acquiring unit 1201 is configured to acquire target text information in a target image, and a context of each character in the target text information;

the processing unit 1202 is further configured to invoke the trained named entity recognition model, and process the context of the target text information and each character to obtain an entity tag of each character;

the processing unit 1202 is further configured to determine at least one entity tag group from the entity tags of each character, where tag types of the entity tags included in each entity tag group are the same;

the processing unit 1202 is further configured to combine characters corresponding to each entity tag in the same entity tag group to obtain product information.

In one embodiment, the trained named entity recognition model includes a trained encoder and a trained labeling module;

The processing unit 1202 is further configured to: invoking a trained encoder, processing target text information and the context relation of each character, and determining the feature vector of each character; determining a first character positioned before each character and a second character positioned after each character in the target text information; based on the feature vector of each character, the feature vector of the first character and the feature vector of the second character, carrying out coding processing on each character to obtain each coded character; and calling a trained labeling module, and labeling each coded character to obtain the entity label of each character.

In one embodiment, the obtaining unit 1201 is further configured to: performing text detection processing on the target image to obtain at least one text region and position information of each text region in the target image; sequencing at least one text region based on the position information of each text region in the target image to obtain sequenced text regions; and performing splicing processing on characters contained in each text region in the sequenced text regions to obtain target text information.

In one embodiment, the location information of the respective text region in the target image includes an abscissa and an ordinate of the respective text region in the target image;

The acquiring unit 1201 is further configured to: determining at least one text region queue based on the ordinate of each text region, wherein the ordinate of the text region in a target text region queue in the at least one text region queue is smaller than the ordinate of the text region in the next text region queue of the target text region queue, the target text region queue is any text region queue in the at least one text region queue, and the ordinate of each text region in the same text region queue is matched; traversing at least one text region queue, and sorting all text regions in the currently traversed text region queue based on the abscissa of all text regions in the currently traversed text region queue to obtain sorted text regions.

In one embodiment, the obtaining unit 1201 is further configured to invoke a word order correction model, perform a word order correction process on at least one text region based on the location information of each text region, to obtain a processed text region sequence, where a word order between a character in any text region in the processed text region sequence and a character in a previous text region in any text region, and a word order between characters in a next text region in any text region satisfies a preset word order rule; and taking the processed text region sequence as the ordered text region.

In one embodiment, the processing unit 1202 is further configured to obtain a first training sample, where the first training sample includes first training text information, and a target entity tag of each training character in the first training text information; carrying out semantic analysis on the first training text information to obtain the context relation of each training character in the first training text information; invoking a named entity recognition model, and processing the first training text information and the context relation of each training character in the first training text information to obtain a predicted entity label of each training character in the first training text information; training the named entity recognition model based on the predicted entity label of each training character in the first training text information and the target entity label of each training character in the first training text information to obtain a trained named entity recognition model.

In one embodiment, the named entity recognition model includes an encoder and a labeling module; the processing unit 1202 is further configured to: acquiring a second training sample, wherein the second training sample comprises second training text information, the second training text information comprises a plurality of training characters, and the information type of the second training text information is a preset type; carrying out semantic analysis on the second training text information to obtain the context relation of each training character in the second training text information; generating a target attention parameter based on each training character in the second training text information and the context of each training character in the second training text information; updating the attention parameter in the encoder to the target attention parameter to obtain an updated encoder;

The processing unit 1202 is further configured to: invoking an updated encoder, processing the first training text information and the context relation of each training character in the first training text information, and determining the feature vector of each training character; determining a third character located before each training character and a fourth character located after each training character in the first training text information; based on the feature vector of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information and the feature vector of the fourth character of each training character in the first training text information, carrying out coding processing on each training character in the first training text information to obtain each coded training character; and calling a labeling module to label each coded training character to obtain the entity label of each training character in the first training text information.

In one embodiment, the processing unit 1202 is further configured to search, if there is target product information in the product information obtained based on the target text information, a target character that is different from each character in the preset character set in the target product information, and search a character that is semantically matched with the target character in the preset character set; and updating the target character in the target product information into the searched character to obtain updated target product information.

In one embodiment, the processing unit 1202 is further configured to construct the entity tag and the target entity tag with the same tag type as the target entity tag into an entity tag group if there is a target entity tag in the entity tags of each character; if a plurality of target entity labels exist in the entity labels of the characters, determining a plurality of label clusters, wherein each label cluster comprises an entity label between any target entity label in the entity labels of the characters and the next target entity label of any target entity label; and constructing the entity tags with the same tag type as any target entity tag in each tag cluster and any target entity tag into an entity tag group.

According to one embodiment of the present application, the steps involved in the product information extraction methods shown in fig. 2, 7 and 10 may be performed by the respective units in the product information extraction apparatus shown in fig. 12.

According to another embodiment of the present application, each unit in the product information extracting apparatus shown in fig. 12 may be separately or completely combined into one or several additional units, or some unit(s) thereof may be further split into a plurality of units having smaller functions, which may achieve the same operation without affecting the achievement of the technical effects of the embodiments of the present application. The above units are divided based on logic functions, and in practical applications, the functions of one unit may be implemented by a plurality of units, or the functions of a plurality of units may be implemented by one unit. In other embodiments of the present application, the product information extraction apparatus based on logical function division may also include other units, and in practical applications, these functions may also be implemented with assistance of other units, and may be implemented by cooperation of a plurality of units.

According to another embodiment of the present application, a product information extraction apparatus as shown in fig. 12 may be constructed by running a computer program (including program code) capable of executing the steps involved in the respective methods shown in fig. 2, 6 and 9 on a general-purpose computing device such as a computer including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read only storage medium (ROM), and the like, and a storage element, and implementing the product information extraction method of the embodiment of the present application. The computer program may be recorded on, for example, a computer readable storage medium, and loaded into and executed by the computing device described above.

Based on the method embodiment and the device embodiment, the application further provides electronic equipment. Referring to fig. 13, a schematic structural diagram of an electronic device according to an embodiment of the present application is provided. The electronic device shown in fig. 13 may include at least a processor 1301, an input interface 1302, an output interface 1303, and a computer storage medium 1304. Wherein the processor 1301, the input interface 1302, the output interface 1303, and the computer storage medium 1304 may be connected by a bus or other means.

The computer storage medium 1304 may be stored in a memory of an electronic device, the computer storage medium 1304 for storing a computer program comprising program instructions, and the processor 1301 for executing the program instructions stored by the computer storage medium 1304. Processor 1301 (or CPU (Central Processing Unit, central processing unit)) is a computing core and a control core of an electronic device, which is adapted to implement one or more instructions, in particular to load and execute one or more instructions to implement the above-described product information extraction method flow or corresponding functions.

The embodiment of the application also provides a computer storage medium (Memory), which is a Memory device in the electronic device and is used for storing programs and data. It will be appreciated that the computer storage medium herein may include both a built-in storage medium in the terminal and an extended storage medium supported by the terminal. The computer storage medium provides a storage space that stores an operating system of the terminal. Also stored in this memory space are one or more instructions, which may be one or more computer programs (including program code), adapted to be loaded and executed by processor 1301. Note that the computer storage medium may be a high-speed random access memory (random access memory, RAM) or a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory; optionally, at least one computer storage medium remote from the processor may be present.

In one embodiment, one or more instructions stored in a computer storage medium may be loaded and executed by processor 1301 to implement the corresponding steps of the methods described above in connection with the product information extraction method embodiments of fig. 2, 7, and 10, in a specific implementation, the one or more instructions in the computer storage medium are loaded and executed by processor 1301 to:

processor 1301 obtains target text information in the target image, and a context of each character in the target text information;

processor 1301 calls a trained named entity recognition model, and processes the context relation between the target text information and each character to obtain an entity tag of each character;

processor 1301 determines at least one entity tag group from the entity tags of each character, where each entity tag group includes the same tag type of the entity tag;

processor 1301 combines the characters corresponding to each of the physical tags in the same physical tag group to obtain product information.

In one embodiment, the trained named entity recognition model includes a trained encoder and a trained annotation module; processor 1301 invokes the trained encoder to process the target text information and the context of each character to determine the feature vector of each character; determining a first character positioned before each character and a second character positioned after each character in the target text information; based on the feature vector of each character, the feature vector of the first character and the feature vector of the second character, carrying out coding processing on each character to obtain each coded character;

Processor 1301 invokes the trained labeling module to label each encoded character to obtain an entity tag for each character.

In one embodiment, processor 1301 performs text detection processing on the target image to obtain at least one text region, and position information of each text region in the target image;

processor 1301 sorts at least one text region based on the position information of each text region in the target image, to obtain a sorted text region;

the processor 1301 performs a concatenation process on characters included in each text region in the ordered text regions, to obtain target text information.

In one embodiment, the location information of each text region in the target image includes an abscissa and an ordinate of each text region in the target image;

processor 1301 sorts at least one text region based on the position information of each text region in the target image, and obtains a sorted text region, including:

processor 1301 determines at least one text region queue based on the ordinate of each text region, the ordinate of a text region in a target text region queue in the at least one text region queue being smaller than the ordinate of a text region in a next text region queue of the target text region queue, the target text region queue being any text region queue in the at least one text region queue, the ordinate of each text region in the same text region queue being matched;

Processor 1301 traverses at least one text region queue, and sorts each text region in the currently traversed text region queue based on the abscissa of each text region in the currently traversed text region queue, resulting in a sorted text region.

In one embodiment, processor 1301 sorts at least one text region based on the location information of each text region in the target image, resulting in a sorted text region, including:

the processor 1301 calls a word order correction model, performs word order correction processing on at least one text region based on the position information of each text region, to obtain a processed text region sequence, wherein the word order between the characters in any text region in the processed text region sequence and the characters in the previous text region in any text region, and the word order between the characters in the next text region in any text region satisfies a preset word order rule;

processor 1301 takes the processed text region sequence as the ordered text region.

In one embodiment, processor 1301 obtains a first training sample, the first training sample comprising first training text information, and a target entity tag for each training character in the first training text information;

The processor 1301 performs semantic analysis on the first training text information to obtain a context relationship of each training character in the first training text information;

processor 1301 invokes a named entity recognition model to process the first training text information and the context of each training character in the first training text information to obtain a predicted entity label of each training character in the first training text information;

processor 1301 trains the named entity recognition model based on the predicted entity tags of each training character in the first training text information and the target entity tags of each training character in the first training text information, and obtains the trained named entity recognition model.

In one embodiment, a named entity recognition model includes an encoder and a labeling module;

processor 1301 obtains a second training sample, the second training sample including second training text information, the second training text information including a plurality of training characters, the information type of the second training text information being a preset type;

the processor 1301 performs semantic analysis on the second training text information to obtain a context relationship of each training character in the second training text information;

Processor 1301 generates a target attention parameter based on each training character in the second training text information and a context relationship of each training character in the second training text information;

processor 1301 updates the attention parameter in the encoder to the target attention parameter, resulting in an updated encoder;

processor 1301 invokes a named entity recognition model to process the first training text information and the context of each training character in the first training text information to obtain a predicted entity tag for each training character in the first training text information, comprising:

processor 1301 invokes the updated encoder to process the first training text information and the context of each training character in the first training text information, and determine a feature vector for each training character;

processor 1301 determines a third character preceding each training character and a fourth character following each training character in the first training text information;

processor 1301 encodes each training character in the first training text information based on the feature vector of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information, and the feature vector of the fourth character of each training character in the first training text information, to obtain each encoded training character;

Processor 1301 invokes a labeling module to label each coded training character to obtain an entity tag of each training character in the first training text information.

In one embodiment, if there is target product information in the product information obtained based on the target text information, and there is a target character in the target product information that is different from each character in the preset character set, the processor 1301 searches for a character that semantically matches the target character in the preset character set;

processor 1301 updates the target character in the target product information to the searched character, resulting in updated target product information.

In one embodiment, processor 1301 determines at least one entity tag group among the entity tags of each character, each entity tag group containing the same tag type of the entity tag, including:

if there is a target entity tag in the entity tags of each character, the processor 1301 constructs the entity tag and the target entity tag with the same tag type as the target entity tag into an entity tag group;

if there are multiple target entity tags in the entity tags of each character, processor 1301 determines multiple tag clusters, each tag cluster including an entity tag between any one of the entity tags of each character and the next target entity tag of any one of the target entity tags;

Processor 1301 constructs each tag cluster from an entity tag of the same tag type as any target entity tag and any target entity tag as a set of entity tags.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium and executes the computer instructions to cause the electronic device to perform the method embodiments described above and illustrated in fig. 2, 6 and 9. The computer readable storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.

It should be noted that, in the embodiment of the present application, the related scene of extracting the menu information is taken as an example to describe the product information extracting scheme mentioned in the embodiment of the present application, which does not limit the embodiment of the present application, and the product information extraction mentioned in the embodiment of the present application may also be applied to other scenes where product information needs to be extracted from the image, which is not limited by the embodiment of the present application.

The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.

Claims

1. A product information extraction method, characterized by comprising:

2. The method of claim 1, wherein the trained named entity recognition model comprises a trained encoder and a trained labeling module;

The step of calling the trained named entity recognition model, and processing the context relation between the target text information and each character to obtain the entity label of each character, which comprises the following steps:

invoking the trained encoder, processing the target text information and the context relation of each character, and determining the feature vector of each character;

determining a first character positioned before each character and a second character positioned after each character in the target text information;

based on the feature vector of each character, the feature vector of the first character and the feature vector of the second character, carrying out coding processing on each character to obtain each coded character;

and calling the trained labeling module to label the coded characters to obtain the entity labels of the characters.

3. The method of claim 1, wherein the obtaining the target text information in the target image comprises:

performing text detection processing on the target image to obtain at least one text region and position information of each text region in the target image;

Sorting the at least one text region based on the position information of each text region in the target image to obtain sorted text regions;

and performing splicing processing on characters contained in each text region in the sequenced text regions to obtain the target text information.

4. A method according to claim 3, wherein the location information of the respective text region in the target image comprises an abscissa and an ordinate of the respective text region in the target image;

the step of sorting the at least one text region based on the position information of each text region in the target image to obtain sorted text regions includes:

determining at least one text region queue based on the ordinate of each text region, wherein the ordinate of a text region in a target text region queue in the at least one text region queue is smaller than the ordinate of a text region in a next text region queue of the target text region queue, the target text region queue is any text region queue in the at least one text region queue, and the ordinate of each text region in the same text region queue is matched;

Traversing the at least one text region queue, and sorting all text regions in the currently traversed text region queue based on the abscissa of all text regions in the currently traversed text region queue to obtain the sorted text regions.

5. A method according to claim 3, wherein said ranking said at least one text region based on the location information of said respective text region in said target image, results in a ranked text region, comprises:

calling a word order correction model, and carrying out word order correction processing on at least one text region based on the position information of each text region to obtain a processed text region sequence, wherein the word order between characters in any text region in the processed text region sequence and characters in a previous text region of any text region and the word order between characters in a next text region of any text region meet a preset word order rule;

and taking the processed text region sequence as the sequenced text region.

6. The method according to claim 1, wherein the method further comprises:

Acquiring a first training sample, wherein the first training sample comprises first training text information and target entity labels of training characters in the first training text information;

carrying out semantic analysis on the first training text information to obtain the context relation of each training character in the first training text information;

invoking a named entity recognition model, and processing the first training text information and the context relation of each training character in the first training text information to obtain a predicted entity label of each training character in the first training text information;

and training the named entity recognition model based on the predicted entity label of each training character in the first training text information and the target entity label of each training character in the first training text information to obtain the trained named entity recognition model.

7. The method of claim 6, wherein the named entity recognition model includes an encoder and a labeling module;

the method further comprises the steps of:

acquiring a second training sample, wherein the second training sample comprises second training text information, the second training text information comprises a plurality of training characters, and the information type of the second training text information is a preset type;

Carrying out semantic analysis on the second training text information to obtain the context relation of each training character in the second training text information;

generating a target attention parameter based on each training character in the second training text information and the context of each training character in the second training text information;

updating the attention parameter in the encoder to the target attention parameter to obtain an updated encoder;

the calling of the named entity recognition model processes the first training text information and the context relation of each training character in the first training text information to obtain a predicted entity label of each training character in the first training text information, and the method comprises the following steps:

invoking the updated encoder, processing the first training text information and the context relation of each training character in the first training text information, and determining the feature vector of each training character;

determining a third character positioned before each training character and a fourth character positioned after each training character in the first training text information;

based on the feature vector of each training character in the first training text information, the feature vector of the third character of each training character in the first training text information and the feature vector of the fourth character of each training character in the first training text information, carrying out coding processing on each training character in the first training text information to obtain each coded training character;

And calling the marking module to mark the coded training characters to obtain entity labels of the training characters in the first training text information.

8. The method according to claim 1, wherein the method further comprises:

if target product information exists in the product information obtained based on the target text information, and target characters which are different from all characters in a preset character set exist in the target product information, searching the characters which are matched with the target character semantically in the preset character set;

and updating the target character in the target product information into the searched character to obtain updated target product information.

9. The method according to claim 1, wherein determining at least one entity tag group among the entity tags of the respective characters, the entity tags included in the respective entity tag groups having the same tag type, includes:

if a target entity tag exists in the entity tags of the characters, constructing an entity tag group by the entity tag with the same tag type as the target entity tag and the target entity tag;

If a plurality of target entity labels exist in the entity labels of the characters, determining a plurality of label clusters, wherein each label cluster comprises an entity label between any target entity label in the entity labels of the characters and the next target entity label of any target entity label;

and constructing an entity tag with the same tag type as any target entity tag in each tag cluster and any target entity tag into an entity tag group.

10. A product information extraction device, characterized in that the product information extraction device comprises an acquisition unit and a processing unit, wherein:

11. A computer device comprising a processor, a memory, wherein the memory is configured to store a computer program which, when executed by the processor, implements the product information extraction method of any one of claims 1 to 9.

12. A computer storage medium, characterized in that the computer storage medium has stored therein computer program instructions for executing the product information extraction method according to any of claims 1-9 when executed by a processor.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-9.