JP7164252B2 - Image processing method, device, electronic device and computer program - Google Patents

Description

This application benefits from the priority of the Chinese patent application with application number 201910872478.6 and titled "Image Processing Method, Apparatus and Electronic Apparatus" filed with the Chinese Patent Office on September 16, 2019 , the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD The present disclosure relates to the technical field of artificial intelligence, and in particular to an image processing method, an image description generating device and an electronic device.

Generating image descriptions is an analytical study for generating natural language descriptions that can express the meaning of photographs, and has broad and potentially applicable applications. For example, by generating a text description from an image, a visually impaired person can quickly and accurately understand the content of the image. In the field of early childhood education, generating intuitive and accurate descriptive texts from images for children will be useful for enlightening learning of young children.

With the successful use of neural networks in image recognition and machine translation, there are already many methods for generating image text descriptions based on neural network models. Traditionally, image description generation has mainly been done by using convolutional neural networks to represent the image code as a fixed vector, and then directly using recurrent neural networks to decode it into descriptive phrases. rice field. However, since the conventional decoding model is relatively simple, the decoding effect is significantly reduced when the phrase is long or the syntax is complicated.

It should be noted that the information disclosed in the background is merely used to enhance the understanding of the background of the present disclosure and may contain information that does not constitute prior art known to those skilled in the art.

Embodiments of the present disclosure provide an image processing method, an image processing apparatus, and an electronic device. This allows the generation of more accurate and fluent text descriptions while accurately and effectively extracting, at least to some extent, the natural language information contained in the images.

Other features and advantages of the disclosure will become apparent from the following detailed description, or will be learned in part through practice of the disclosure.

According to one embodiment of the present disclosure, obtaining an input image and encoding an object contained in each image region within the input image to obtain a first image feature; processing pixels in the first image feature according to a rule and identifying a second image feature based on the processed pixels; a region feature corresponding to each image region in the first image feature is decoded to obtain a word vector corresponding to each image region, and a word vector corresponding to the input image is obtained based on the word vector and generating a textual description. where the start word vector is the start mark of the text description.

According to one embodiment of the present disclosure, a feature extraction module that obtains an input image and encodes objects contained in each image region within the input image to obtain a first image feature; a feature transformation module for processing pixels in said first image feature according to a set of rules and identifying a second image feature based on the processed pixels; and based on said second image feature and a starting word vector. at different points in time, decoding the area feature corresponding to each image area in the first image feature to obtain a word vector corresponding to each image area; and a description generation module for generating a text description corresponding to the image. where the start word vector is the start mark of the text description.

The technical proposal of the present disclosure can more accurately and effectively extract the natural language information contained in the input image by decoding the image features corresponding to the input image using the decoding network model. Since the decoding network model will be applied even when the text is long or the syntax is complicated, it is expected to improve the accuracy and fluency of the text explanation.

It should be understood that the foregoing general description and the following detailed description are exemplary and interpretative only and are not restrictive of the present disclosure.

The drawings set forth below are incorporated in and constitute a part of this specification, illustrate embodiments in accordance with the disclosure, and are used together with the specification to explain the principles of the disclosure. Obviously, the drawings shown below are merely some embodiments of the present disclosure, and a person skilled in the art may derive other drawings based on these drawings without any inventive step. can also In these drawings:

1 is a schematic diagram illustrating an exemplary system architecture to which technical solutions according to embodiments of the present disclosure can be applied; FIG. 1 is a flowchart of an image processing method in related art; 4 is a flowchart of an image processing method according to an embodiment of the present disclosure; 2 is a schematic diagram illustrating the configuration of a reflection decoding network model according to an embodiment of the present disclosure; FIG. FIG. 3 is a schematic diagram illustrating the configuration of a visual attention module according to one embodiment of the present disclosure; [0018] Fig. 4 is a flow chart illustrating the processing procedure of a visual attention module according to one embodiment of the present disclosure; 4 is a flowchart of image processing according to an embodiment of the present disclosure; [0017] Fig. 5 is a flow chart illustrating the processing procedure of a reflective attention module according to one embodiment of the present disclosure; FIG. 3 is a schematic diagram illustrating a configuration of a reflective attention module according to one embodiment of the present disclosure; 4 is a flow chart illustrating a procedure for determining position perception loss by a reflective position module according to one embodiment of the present disclosure; 1 is a block diagram of an image processing device according to an embodiment of the present disclosure; FIG. 1 is a schematic diagram showing the configuration of a computer system suitable for implementing an image processing apparatus according to an embodiment of the present disclosure; FIG.

Exemplary embodiments will now be described more fully with reference to the drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein. Rather, these embodiments are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

Also, the features, configurations or characteristics described herein may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided in order to provide a thorough understanding of the embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the technical ideas of the present disclosure may be practiced without one or more of the specific details, or that other methods, components, devices, steps, etc. may be used. deaf. In other instances, well-known methods, devices, implementations or operations will not be shown or described in detail to avoid confusing aspects of the present disclosure.

The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to separate physical entities. That is, these functional entities are implemented in software, implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices. can be

The flowcharts shown in the figures are merely exemplary illustrations and do not necessarily include all content and operations/steps, nor need to be performed in the order described. For example, some operations/steps can be decomposed, some operations/steps can be combined, and some operations/steps can be partially combined, so the actual order of execution may be changed according to actual situations.

FIG. 1 is a schematic diagram illustrating an exemplary system architecture to which the technical solutions according to embodiments of the present disclosure can be applied.

As shown in FIG. 1, system architecture 100 comprises terminal device 101 , network 102 and server 103 . Network 102 serves as a medium for providing a communication link between terminal device 101 and server 103 . Network 102 may include various connection types, such as wired and wireless communication links.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely exemplary. Any number of terminal devices, networks and servers can be provided according to actual needs. For example, server 103 may be a server cluster consisting of multiple servers.

In one embodiment of the present disclosure, the terminal device 101 transmits an image to the server 103 via the network 102, and after obtaining the input image by the server 103, first divides the input image to form a plurality of image regions and , feature extraction is performed on the object in each image region by the coding network model, the region feature corresponding to each image region is obtained, and further based on the region feature corresponding to each image region, the first image corresponding to the input image is obtained. 1 image feature is obtained. Next, according to preset rules, pixels in the first image feature are processed, and based on the processed pixels, a second image feature is identified; and the first image feature, the second image feature. and input the starting word vector into the reflection decoding network model, decode the first image feature by the reflection decoding network model to obtain a word vector corresponding to each image region, and further obtain a word vector corresponding to each image region Based on this, a text description corresponding to the input image is generated. According to the technical solution according to the embodiment of the present disclosure, the performance of the model can be ensured even when the phrase is long or the syntax is complicated. are extracted systematically, producing more accurate and fluent text descriptions.

It should be noted that the image processing method provided by the embodiments of the present disclosure is typically executed by a server, and correspondingly, the image processing device is typically embedded in the server. However, in another embodiment of the present disclosure, the image processing method provided by the embodiments of the present disclosure may also be performed by a terminal device.

In the relevant art in the field, textual descriptions of images are mainly generated through encoding/decoding frames. FIG. 2 shows a flow chart of an image processing method in the related art. As shown in FIG. 2, image 201 is input to encoding network model 202 . This coding network model 202 includes a Faster R-CNN network and a ResNet-101 network, and if feature extraction of an input image is performed via the Faster R-CNN network, local feature information corresponding to each object in the input image can be obtained. is obtained, and by extracting the features of the input image via the ResNet-101 network, the overall feature information corresponding to the input image can be obtained. Subsequently, the local feature information and global feature information are input to the decoding network model 203 . This decoding network model 203 includes multiple overlapping network structures, which are recurrent neural networks with attention mechanisms. Specifically, the overall feature information is input to the first layer LSTM, feature extraction of the overall feature information is performed via the first layer LSTM, and the first hidden state is output. Next, the first hidden state and local feature information are input to the attention mechanism network layer, and the mixed feature is output via the attention mechanism network layer; jointly process the hidden states of and output a second hidden state. Finally, we perform softmax processing on the second hidden state to obtain the predicted word vector.

Although the image description generation algorithm shown in FIG. 2 can achieve good results, it is still limited. Specifically, as a possible way to improve the effectiveness of the model, we can only extract image features that are more representative fine-grained and separated down to the single object level, and focus on the language model itself. is ignored. In addition, since the decoding model is relatively simple, the decoding effect of the model is remarkably reduced when the phrase is long or the syntax is complicated.

Embodiments of the present disclosure provide an image processing method for the field of artificial intelligence. Artificial Intelligence (AI) uses digital computers or devices controlled by digital computers to simulate, extend, and augment human intelligence, sense the environment, acquire knowledge, and utilize knowledge. Theories, methods, techniques and application systems for obtaining the best results from In other words, artificial intelligence is an umbrella technology of computer science that seeks to understand the nature of intelligence and create new intelligent machines that can react in ways similar to human intelligence. Artificial intelligence is a technology that researches the design principles and implementation methods of various intelligent machines, and makes machines have the functions of sensing, reasoning and decision-making.

Artificial intelligence technology is a comprehensive field that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating/interactive systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning/deep learning.

Computer Vision (CV) is a technology that studies how machines "see", and more specifically, uses cameras and computers to perform machine vision tasks such as identifying, tracking and measuring targets. and image processing that causes a computer to process the image into an image more suitable for viewing by the human eye or transmission to equipment. As one of the science subjects, computer vision studies related theories and techniques, aiming to build artificial intelligence systems that can acquire information from images and multi-dimensional data. Computer vision techniques typically include image processing, image recognition, image semantic analysis, image retrieval, OCR, video processing, video semantic analysis, video content/action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality , synchronous positioning and map construction, etc., and also general biometric authentication technologies such as face recognition and fingerprint recognition.

Machine learning (ML) is a subject that spans various fields such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory, and specializes in methods for simulating or realizing human learning behavior. They aim to learn new knowledge and skills and reorganize existing knowledge structures to continually improve their performance. Machine learning is the core of artificial intelligence, the fundamental method for making computers intelligent, and is widely applied in various fields of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from instruction.

With the research and progress of artificial intelligence technology, the research and application of artificial intelligence technology, such as ordinary smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, driverless driving, automatic driving, drone, robot, etc. It has been advanced in many fields such as smart medical and smart customer service. With the development of technology, artificial intelligence technology is expected to be applied in more fields and play an increasingly important role.

A technical proposal according to an embodiment of the present disclosure relates to image semantic analysis technology using artificial intelligence. The description will be specifically made with reference to the following embodiments.

Embodiments of the present disclosure first propose an image processing method that can be applied in fields such as early childhood education, image retrieval, and guidance for the visually impaired. Detailed implementations of the technical solutions according to the embodiments of the present disclosure are described in detail below.

FIG. 3 is a flowchart of an image processing method according to one embodiment of the present disclosure. The image processing method may be performed by one or more computing devices. The one or more computing devices may be terminal device 101 and/or server 103 as shown in FIG. Referring to FIG. 3, the image processing method includes at least steps S310 to S330.

In step S310, an input image is obtained and an object contained in each image region within the input image is encoded to obtain a first image feature.

In one embodiment of the present disclosure, the input image is an image downloaded from a network, or an image stored locally in the terminal device 101, or an image that the user uses via a terminal having a shooting means such as a camera, a video camera, or a smartphone. It can be an acquired image. Once the image for which the text description needs to be generated is identified, it can be sent to the server 103 via the terminal device 101 . Furthermore, the terminal device 101 is not particularly limited to the embodiments of the present disclosure, but may be any terminal device having a display screen, such as a smart phone, a laptop computer, or a desktop computer.

In one embodiment of the present disclosure, after receiving an input image, the input image may be divided to form multiple image regions. Here, when dividing the input image, it can be divided by the number of pixels or by different objects in the image. After segmenting the input image to form multiple image regions, the objects in each image region can be encoded, ie, feature extracted. For example, the scene in a photograph is a child bouncing a rubber ball in a garden. can be ignored, making feature extraction for the background unnecessary. When encoding the object in each image region, a network structure such as Faster R-CNN, ResNet, VGG, etc. is used as the encoding network model, and the encoding network model extracts features from the object in each image region. to obtain region features corresponding to each image region. The region features are essentially fixed vector representations corresponding to image regions. Furthermore, a first image feature corresponding to the input image can be obtained based on the area feature corresponding to each image area.

In step S320, pixels in the first image feature are processed according to preset rules, and a second image feature is identified based on the processed pixels.

In one embodiment of the present disclosure, feature extraction is performed from each image region in the input image, and after obtaining the first image feature, based on the pixel value of each pixel in the first image feature, in the second image feature Pixel values can be specified. Specifically, the pixel average value of all pixels in the first image feature can be calculated, and the pixel average value can be used as the pixel value of each pixel in the second image feature. A reflection decoding network model decodes the first image feature based on the second image feature and the starting word vector, so that it can predict a word vector corresponding to each image region in the first image feature. A second image feature may be input to the reflection decoding network model as an input feature. Here, the start word vector in the embodiments of the present disclosure may be any character that has no substantial meaning, for example, a start mark code such as # or a start mark word such as BN, but the Embodiments do not specifically limit this.

In step S330, based on the second image feature and the starting word vector, decode the area feature corresponding to each image area in the first image feature to obtain a word vector corresponding to each image area. and generating a text description corresponding to the input image based on the word vector. where the start word vector is the start mark of the text description.

At step S330, the region features corresponding to the respective image regions in the first image feature are decoded at different times, and the previously decoded region features can be used to decode the current region feature.

In one embodiment of the present disclosure, after obtaining the second image feature, the second image feature is input to the reflection decoding network model as an input feature, and at different times, each image region in the first image feature A starting word vector can also be input to the reflection decoding network model to decode corresponding region features and obtain a word vector corresponding to each image region.

FIG. 4 is a schematic diagram showing the configuration of a reflection decoding network model. As shown in FIG. 4, the reflection decoding network model includes a plurality of sequentially arranged reflection decoding sub-networks. Each reflection decoding sub-network respectively decodes the region feature corresponding to each image region in the first image feature at a different time to obtain a word vector corresponding to each image region. The second image feature and the starting word vector are input as input features to the first reflective decoding sub-network, and the first reflective decoding sub-network generates a first image feature based on the second image feature and the starting word vector. can be decoded to obtain a word vector corresponding to the target region feature. For the M+1 th reflection decoding sub-network, the second image feature and the word vector output from the M th reflection decoding sub-network are input to the M+1 th reflection decoding sub-network, and the M+1 th reflection decoding sub-network A region of interest feature in the first image feature can be decoded to obtain a word vector corresponding to the region of interest feature. where M is a positive integer.

A method according to embodiments of the present disclosure comprises:
obtaining an input image and extracting a region feature of each image region in the input image to obtain a first image feature;
processing pixels in the first image feature according to preset rules and identifying a second image feature based on the processed pixels;
words corresponding to region features corresponding to each of said image regions in said first image feature at different times based on said second image feature and at least one word vector identified for said input image; identifying vectors, predicting positions in the text description of the word vectors, and generating a text description corresponding to the input image based on the word vectors and the positions.

In one embodiment of the present disclosure, each reflective decoding sub-network has the same structure, each of which consists of three parts: a visual attention module, a reflective attention module RAM (Reflective Attention Module) and a reflective position module RPM (RPM). Prepare. The visual attention module focuses primarily on visual features of the coded network model. Based on the output information of the visual attention module, the reflective attention module uses the text attention mechanism to model the degree of matching of the output information of the visual attention module at the present time and the past time, acquire a context vector, Generating words in the current moment, which allows us to capture more of the comprehensive historical lexical information. The reflective location module introduces the relative location information of each word in the generated text description, predicts the vocabulary with the reflection decoding network model, and at the same time predicts the relative location of the current vocabulary in the text description. This helps make the reflective decoding network model aware of the syntactic structure of words.

FIG. 5 is a schematic diagram showing the configuration of the visual attention module. As shown in FIG. 5, the visual attention module 500 comprises a first long short-term memory network (LSTM-1) 501, a second long short-term memory network (LSTM-2) 502 and an attention mechanism network (Attvis) 503. include. The first long short-term memory network 501 performs feature extraction based on the second image feature and the word vector acquired at the previous time. The second long short term memory network 502 performs feature extraction based on the output information of the first long short term memory network 501 and the output information of the attention mechanism network 503 . The attention mechanism network 503 performs feature extraction based on the first image features and the output information of the first long short-term memory network 501 .

Furthermore, FIG. 6 is a flow chart showing the processing procedure of the visual attention module. For convenience of understanding, the embodiments of the present disclosure take the processing procedure of the visual attention module in the t-th reflection decoding sub-network as an example. As shown in FIG. 6, the processing procedure of the visual attention module includes at least steps S601-S604. Specifically, it is as follows.

In step S601, the word vector output from the reflection decoding sub-network at the previous time is multiplied by the first weight matrix to obtain a target word vector.

及び前時点で反射復号サブネットワークによって出力された単語ベクトルは、ＬＳＴＭ－１の入力特徴である。入力された単語ベクトルの次元数がＬＳＴＭ－１によって処理されたデータの次元数と同じになることを確保するために、各反射復号サブネットワークに入力した単語ベクトル特徴に対して次元調整を行うことができ、具体的には、入力した単語ベクトル特徴Ｏｔ（ｔ＝１，．．．，Ｔ）に第１の重み行列Ｗｅを掛けて、目標単語ベクトルを取得し、入力特徴Ｏｔの次元変化を実現することができる。但し、当該第１の重み行列Ｗｅは各入力特徴Ｏｔに対して共有されるため、モデルをトレーニングする際に、第１の重み行列Ｗｅに対して１つのパラメータでトレーニングすればよい。 In one embodiment of the present disclosure, FIG. 7 illustrates a flowchart of image processing. As shown in FIG. 7, for LSTM-1, the second image feature identified based on the first image feature

and the word vectors output by the reflection decoding sub-network at the previous time point are the input features of the LSTM-1. Performing a dimensionality adjustment on the word vector features input to each reflection decoding sub-network to ensure that the dimensionality of the input word vectors is the same as the dimensionality of the data processed by the LSTM-1. Specifically, the input word vector feature Ot (t=1, . . . , T) is multiplied by the first weight matrix We to obtain the target word vector, and the dimensional change of the input feature Ot is can be realized. However, since the first weighting matrix We is shared for each input feature Ot, when training the model, it is sufficient to train the first weighting matrix We with one parameter.

In step S602, the first short-term memory network performs feature extraction on the second image feature and the target word vector to obtain first output information.

である。 In one embodiment of the present disclosure, after the second image feature and the target word vector are input to the LSTM-1, the LSTM-1 processes the target word vector and the second image feature to produce a first output Output information. This first output information is actually the Hidden state output by LSTM-1, for example shown in FIG.

is.

In step S603, the first output information and the first image feature are input to the attention mechanism network to perform visual matching to obtain target area features.

を対象領域特徴として注意機構ネットワークから出力させることができる。 In one embodiment of the present disclosure, the attention mechanism can selectively focus on only a portion of the overall information and ignore other visible information, similar to human vision. Prior to decoding by the reflection decoding network model, feature extraction is performed from the input image using a convolutional neural network such as Faster R-CNN to obtain the first image feature {ri} (i = 1, ..., k). Then, when the first output information output from LSTM-1 is acquired, the first output information and the first image feature are simultaneously input to the attention mechanism network, and the attention mechanism network Attvis is used to obtain the first image feature. Visual matching is performed on the output information of and the first image feature, the degree of matching between each area feature in the first image feature and the first output information is specified, and finally, the degree of matching is the highest. Determined region features, such as those shown in FIG.

can be output from the attention mechanism network as target region features.

In step S604, the second short-term memory network performs feature extraction on the first output information and the target area feature to obtain second output information.

である。なお、他の回帰型ニューラルネットワークを使用して本開示の実施形態におけるＬＳＴＭを置き換えることもでき、かつさらには、異なるタイプの回帰型ニューラルネットワークを使用して、本開示の実施形態におけるＬＳＴＭ－１およびＬＳＴＭ－２を置き換えることができる。しかし、長短期記憶ネットワーク（ＬＳＴＭ，Ｌｏｎｇ Ｓｈｏｒｔ－Ｔｅｒｍ Ｍｅｍｏｒｙ）は時間回帰型ニューラルネットワークであり、時系列において間隔及び遅延が比較的に長い重要なイベントを処理・予測するに適するので、語彙を更に精度よく予測し、流暢なテキスト説明を生成するために、本開示の実施形態における画像処理方法は、主にＬＳＴＭを利用して語彙の予測を行うとされている。 In one embodiment of the present disclosure, after the target region features are obtained, the target region features and the first output information are input to LSTM-2 as input features, and LSTM-2 receives the first output information and the target region Feature extraction may be performed on the features to obtain second output information corresponding to the region of interest features. This second output information is the hidden state output by LSTM-2, eg, shown in FIG.

is. It should be noted that other recurrent neural networks can be used to replace LSTM in embodiments of the present disclosure, and even different types of recurrent neural networks can be used to replace LSTM-1 in embodiments of the present disclosure. and LSTM-2 can be replaced. However, the long short-term memory network (LSTM) is a time-recurrent neural network and is suitable for processing and predicting important events with relatively long intervals and delays in the time series. In order to predict accurately and generate fluent text descriptions, the image processing method in the embodiments of the present disclosure is said to mainly utilize LSTM for lexical prediction.

Thereafter, in some embodiments, a word vector corresponding to the region-of-interest feature can be identified based on the first hidden state and the second hidden state.

In one embodiment of the present disclosure, when the phrase is long or the syntax is more complex, in order to improve the decoding effect, the embodiment of the present disclosure first uses the text attention mechanism by the reflective attention module. It is proposed to match the current hidden state with the past hidden state. As shown in FIG. 7, in the case of the reflective attention module RAM in the t-th reflective decoding sub-network, in addition to the second output information output from the corresponding LSTM-2, the 1st to (t-1)th further receive the second output information output from the LSTM-2 and the first output information output from the corresponding LSTM-1 in the reflection decoding subnetwork of the second output information at the past time and the current time According to the first output information and the second output information, identify the third output information corresponding to the target area feature at the present time.

FIG. 8 is a flow chart showing the processing procedure of the reflective attention module. As shown in FIG. 8, the processing procedure includes at least steps S801 to S805, specifically as follows.

In step S801, a target matrix is identified based on the second output information at all past times and the second output information at the present time.

と現時点における第２の出力情報

とで、相応する次元数を有する目標行列が構成され、例えば１０００×１の目標行列が構成される。 In one embodiment of the present disclosure, FIG. 9 is a schematic diagram illustrating the configuration of a reflective attention module. As shown in FIG. 9, the upper left cylinder represents the second output information. Second output information at past time

and the current second output information

to construct a target matrix with the corresponding number of dimensions, for example a 1000×1 target matrix.

In step S802, dimension reduction processing is performed on the target matrix to obtain first feature information, and dimension reduction processing is performed on the current first output information to obtain second feature information. Here, the first feature information and the second feature information have the same number of dimensions.

In one embodiment of the present disclosure, in order to improve computational efficiency, the target matrix and the current first output information are subjected to dimensionality reduction processing, so that the first feature information and the second feature of the same number of dimensions are respectively Information can be obtained. As shown in FIG. 9, the target matrix and the current first output information are each multiplied by a 512-dimensional weight matrix to reduce the number of dimensions of the target matrix and the number of dimensions of the first output information from 1000 to 512. Therefore, it is possible to greatly improve the processing efficiency.

In step S803, the third feature information is obtained by adding the first feature information and the second feature information based on the attention mechanism.

In one embodiment of the present disclosure, corresponding processing can be applied to the first feature information and the second feature information based on the text attention mechanism. For example, in the Attref shown in FIG. 9, the first feature information and the second feature information can be specifically added, and of course other specific processing methods can be used. The embodiments of the present disclosure are not particularly limited. By adding the first feature information and the second feature information, it is possible to obtain the third feature information combining the past hidden state and the current hidden state.

In step S804, the third feature information is weighted and normalized to obtain a second weight matrix.

In one embodiment of the present disclosure, after obtaining the third feature information, the reflex attention weight Wr may be multiplied by the third feature information to obtain a first feature matrix. The number of pieces of information included in the feature matrix is the same as the number of pieces of second output information in the target matrix, both of which are t pieces. Subsequently, softmax processing, that is, normalization processing is performed on the feature matrix, the ratio of each information to the total information is calculated, and the second weight matrix can be specified by the ratio corresponding to each second output information. .

In step S805, the first feature information is multiplied by the second weight matrix to obtain the sum of products, thereby obtaining third output information.

を得ることができる。 In one embodiment of the present disclosure, after obtaining the second weighting matrix corresponding to all the second output information, the first feature information specified according to all the second output information is converted to the second weighting matrix to obtain the sum of products, and the third output information, for example, the right cylinder shown in FIG.

can be obtained.

に対応する単語ベクトル、例えば図７に示されているＳｔを取得することができる。なお、ｔ時点に出力された単語ベクトルＳｔは、ｔ＋ｔ時点での入力ベクトルＯｔ＋１である。 In one embodiment of the present disclosure, after obtaining the third output information output from the reflective attention module, the third output information is multiplied by the third weight matrix Ws to obtain the target region feature

, for example St shown in FIG. The word vector St output at time t is the input vector Ot+1 at time t+t.

が１×１次元のベクトルに変換された後、続いて全結合層から出力されたベクトルが、対応する圧縮関数に従って圧縮層で圧縮され、相対的な位置が求められる。当該圧縮層の出力結果は０～１の間の数値であり、テキスト説明における単語ベクトルの位置を表す。例えば、テキスト説明は１０要素の単語を含む語句であり、圧縮層によって出力された数値は０．６であると想定された場合、当該語句におけるｔ番目の反射復号サブネットワークによって出力された単語ベクトルＳｔの位置は６番目である。 In one embodiment of the present disclosure, as shown in FIG. 7, when the third output information is output from the reflective attention module, the third output information is simultaneously input to the reflective location module. The reflected position module can predict the relative position in the text description of the currently output word vector based on the third output information. Specifically, the reflective position module includes a fully connected layer and a compressive layer. After the third output information is input to the reflection position module, it is first fully connected through a fully connected layer to obtain a 512×1-dimensional

is transformed into a 1×1-dimensional vector, then the vector output from the fully connected layer is compressed in the compression layer according to the corresponding compression function to obtain the relative position. The output result of the compression layer is a number between 0 and 1 representing the position of the word vector in the text description. For example, if the text description is a phrase containing 10 elements and the number output by the compression layer is assumed to be 0.6, then the word vector output by the tth reflective decoding sub-network in that phrase The position of St is 6th.

In one embodiment of the present disclosure, sequentially arranged reflective decoding sub-networks in the reflective decoding network model decode region features corresponding to each image region within the first image feature, and if there is punctuation at the end of a sentence, word vectors and obtain the word vectors {S, S2, . .

In one embodiment of the present disclosure, before the reflective decoding network model performs lexical prediction on the first image feature and generates a text description, the reflective decoding network model needs to be trained. Specifically, we first obtain an image sample and a text description sample corresponding to the image sample, then input the image sample into a reflection decoding network model to be trained to generate a corresponding text description, and generate a corresponding text description for the reflection decoding network model to be trained. The model parameters are adjusted based on how well the generated text descriptions match the corresponding text description samples so that the loss function corresponding to the decoded network model is minimized. In embodiments of the present disclosure, the reflection decoding network model loss function includes both a cross-entropy loss function and a location perception loss function. The cross-entropy loss function is the probability that the text description corresponding to the image sample produced by the reflection decoding network to be trained is correct, and the location perception loss function is the number of word vectors output by the reflection decoding network to be trained at the moment. The distance between the actual position and the predicted position in the text description sample.

In one embodiment of the present disclosure, to minimize the loss function of the reflection decoding network model, we need to minimize the location perception loss function while maximizing the cross-entropy loss function. Here, the cross-entropy loss function is obtained according to the following formula (1). Specifically, it is as follows.

where I is the input image, θ is the parameters for the reflection decoding network model including the weight matrices We, Ws, Wr, etc. in the previous embodiment, and S is the variable-length correct text description corresponding to the input image. , which can represent any phrase.

Since any word vector in the text description S depends on the previous word vector adjacent to it, we follow the chain rule to model and represent the joint probability distribution on the sentence-forming word vectors S1, S2, . . . , ST be able to. Furthermore, based on the above equation (1), the cross-entropy loss function Lxe is specified as shown in the following equation (2).

where N is the number of vocabularies included in the generated text description and St represents the word vector generated at time t.

In the training phase, (S,I) is the training image word pair and the sum of log-probabilities in equation (2) can be optimized by stochastic gradient descent (SGD).

を取得する。最後に、予測位置と、第３の出力情報に対応する単語ベクトルのテキスト説明サンプルにおける実位置とに基づいて位置知覚損失を特定する。ここで、語句における語彙の実位置

は、テキスト説明サンプルに含まれる語彙数と、対象領域特徴に対応する語彙のテキスト説明における位置によって求められ、これによって、実位置

と相対的な位置

とで位置知覚損失Ｌｐｏｓが特定され得る。具体的な計算方法は、下記式（３）に示される通りである。 In one embodiment of the present disclosure, Position-Perceptive Loss may be identified by a reflective position module. FIG. 10 is a flow chart illustrating the procedure for determining position perception loss with the reflection position module. As shown in FIG. 10, the third output information output by the reflective attention module is fully connected in the fully connected layer to generate fully connected information. This total connectivity information may be a 1×1 vector. and compressing the fully connected information according to a preset compression function corresponding to the compression layer, and the predicted position of the word vector corresponding to the third output information, i.e. the relative position of the predicted word vector in the text description.

to get Finally, a position perception loss is determined based on the predicted position and the actual position in the text description sample of the word vector corresponding to the third output information. where the real position of the vocabulary in the phrase

is determined by the number of vocabularies contained in the text description sample and the position in the text description of the vocabulary corresponding to the target region feature, which gives the actual position

position relative to

and the position perception loss Lpos can be specified. A specific calculation method is as shown in the following formula (3).

および

は、それぞれ現時点での単語ベクトルの語句における実位置と、予測された相対的な位置を表し、Ｌｐｏｓを最小化することによって両者間の距離を縮めることができる。 During the ceremony,

and

represent the actual position of the current word vector in the phrase and the predicted relative position, respectively, and the distance between them can be reduced by minimizing Lpos.

Furthermore, after obtaining the cross-entropy loss and the location perception loss, the magnitude of the loss function corresponding to the reflection decoding network model is obtained according to Equation (4) below. Specifically, it is as follows.

L=L _xe +λL _pos (4)

where the parameter λ is used to balance the role of the loss function in the optimization process of the entire reflection decoding network model, and can be set according to actual needs, but is not particularly limited in the embodiments of the present disclosure .

Next, a technical solution according to an embodiment of the present disclosure will be described, taking guidance for the visually impaired as an example. Smart devices such as smart glasses and portable smart cameras are worn by the visually impaired, allowing them to capture real-time images of the road ahead while the visually impaired are moving. Next, the image is analyzed via the image description device installed in the smart device, the corresponding text description is generated, and this text description is output via the corresponding voice output device to the visually impaired. Real-time recognition of road conditions and avoidance of obstacles. For example, when a visually impaired person walks to an intersection, when the red traffic light flashes, the image acquisition unit of the smart device can acquire an image including the traffic light, the pedestrian crossing, and the vehicle passing situation. to obtain a first image feature. then identifying a second image feature according to the pixel mean value of all pixels in the first image feature; and inputting the first image feature, the second image feature and the starting word vector into the reflection decoding network model. Then, the reflection decoding sub-network in the reflection decoding network model performs text prediction on the traffic lights, pedestrian crossings, and vehicles in the image in order. , there is a vehicle, there is no pedestrian”, and finally, based on the word vector corresponding to each image region, it outputs information such as “the traffic light is red, there is a vehicle on the crosswalk, and no pedestrians are allowed ” can be generated. This textual description can be sent in real time to the visually impaired and suggested to pass when the light turns green.

Taking early childhood education as an example, when children read picture books, they become attracted to various images. When a child sees a picture, the picture is captured by a camera mounted on the picture book and input to an image processing unit to generate a corresponding text description. Besides this, the pictures on each page of the picture book can also be pre-stored, and when the child sees a picture on a certain page, the picture on that page will be input into the image processing unit to generate a corresponding text description. . For example, a picture book page showing a lamb grazing on a hillside. Get image features. Next, an average value is obtained for all pixels in the first image feature, and the pixel values of all pixels are replaced with the pixel average value to form a second image feature. Then, the first image feature, the second image feature and the starting word vector are input to the reflection decoding network model, and the reflection decoding network model generates the current word according to the context vector, and Predict relative position. For example, a reflexive attention model may generate word vectors for one, lamb, hillside, grazing in sequence, and based on these word vectors, ``lamb is grazing on the hillside''. can generate a final text description of When the child is looking at the picture, the corresponding text description will be played through the voice output unit, which can help him understand the content of the picture and deepen his cognition of things.

The image processing method in the present disclosure decodes the first image feature encoded by the encoding network model with the reflective decoding network model, matches the hidden state at the present time with the hidden state at the past time with the reflective attention module, By obtaining the context vector to generate the current word vector and predicting the relative position of the current word vector in the text description with the reflective position module, pre- and post-phrase associations and chronological logic are strengthened. , which further improves the decoding ability of the language model and ensures the stability of the model performance when the phrases are long or complex, so that more natural and accurate image text descriptions can be generated. .

It should be noted that the embodiments of the present disclosure are mainly directed to the decoding input part of the long-term time series module and are improved by introducing the reflective attention module and the reflective position module, but other reinforcement learning, convolutional neural networks , and generative adversarial network techniques, and utilizing the reflective attention and reflective location modules in the present disclosure, can be improved to further improve the quality of image description generation.

Apparatus embodiments of the present disclosure are described below, which can be used to perform the image processing methods in the embodiments of the present disclosure. For details not disclosed in the embodiments relating to the apparatus of the present disclosure, please refer to the embodiments relating to the image processing method of the present disclosure.

FIG. 11 is a block diagram of an image processing device according to one embodiment of the present disclosure.

Referring to FIG. 11, an image processing device 1100 according to an embodiment of the present disclosure comprises a feature extraction module 1101, a feature transformation module 1102, and a description generation module 1103.

In it, the feature extraction module 1101 obtains an input image and encodes objects contained in each image region in said input image to obtain a first image feature. The feature transformation module 1102 processes pixels in the first image feature according to preset rules and identifies a second image feature based on the processed pixels. A description generation module 1103 decodes the region features corresponding to the respective image regions in the first image features at different times based on the second image features and the starting word vector, and and generate a text description corresponding to the input image based on the word vector, wherein the start word vector is the start mark of the text description.

In one embodiment of the present disclosure, the feature extraction module 1101 divides the input image to form a plurality of the image regions, and performs feature extraction on objects within the image regions via an encoding network model. to obtain an area feature corresponding to the image area, and to form the first image feature based on the area feature.

In one embodiment of the present disclosure, the feature transformation module 1102 obtains a pixel average value of all pixels in the first image feature, uses the pixel average value as the pixel value of each of the pixels, and It is configured to form a second image feature.

In one embodiment of the present disclosure, the description generation module 1103 generates a description for each of the image regions in the first image feature at different times based on the second image feature and a starting word vector by a reflection decoding network model. It is configured to decode corresponding region features to obtain a word vector corresponding to each said image region.

In one embodiment of the present disclosure, the reflective decoding network model includes a plurality of sequentially arranged reflective decoding sub-networks. The description generation module 1103 inputs the second image features and the word vectors output from the Mth reflection decoding sub-network to the M+1th reflection decoding subnetwork, and the M+1th reflection decoding subnetwork inputs the word vectors output from the M+1th reflection decoding subnetwork. It is configured to decode a region of interest feature within one image feature to obtain a word vector corresponding to the region of interest feature. where M is a positive integer.

In one embodiment of the present disclosure, said explanation generation module 1103 inputs said second image features and said starting word vector into a first reflective decoding sub-network, and said first and decoding a target region feature within the image feature of the target region feature to obtain a word vector corresponding to the target region feature.

In one embodiment of the present disclosure, said reflective decoding sub-network comprises a visual attention module, a reflective attention module and a reflective location module, wherein said reflective location module is a word currently output from said reflective decoding sub-network. Predict the relative position in the textual description of the vector.

In one embodiment of the present disclosure, said visual attention module includes a first long short term memory network, a second long short term memory network and an attention mechanism network. The image processing apparatus 1100 obtains a target word vector by multiplying a first weight matrix by the word vector output from the reflection decoding sub-network at the previous time, and obtains a target word vector by the first long-short-term memory network. performing feature extraction on the image feature and the target word vector to obtain first output information, and inputting the first output information and the first image feature to the attention mechanism network for visual matching to obtain a target region feature, perform feature extraction on the first output information and the target region feature by the second long-short-term memory network, and obtain second output information. It is

In an embodiment of the present disclosure, the image processing device 1100 further converts the second output information at a past point in time and the first output information and the second output information at a present point in time by the reflective attention module. based on, a word vector generation module that identifies third output information that currently corresponds to the region of interest feature.

In one embodiment of the present disclosure, the word vector generation module identifies a target matrix based on all the past time second output information and the current second output information, and Dimension reduction processing is performed on the first output information to obtain first feature information, and dimension reduction processing is performed on the first output information at the current time to obtain second feature information. The feature information and the second feature information have the same number of dimensions, and the first feature information and the second feature information are added based on an attention mechanism to obtain third feature information; weighting process and normalization process are performed on the third feature information to obtain a second weighting matrix; the second weighting matrix is multiplied by the first feature information to obtain a sum of products; 3 output information.

In one embodiment of the present disclosure, the explanation generation module 1103 is configured to multiply the third output information by a third weight matrix to obtain a word vector corresponding to the region-of-interest feature.

In one embodiment of the present disclosure, the image processing device 1100 further comprises a sample acquisition module for acquiring image samples and text description samples corresponding to said image samples; a model training module for training the reflection decoding network model to be trained such that a loss function corresponding to the decoding network model is minimized. Here, the loss function includes a cross-entropy loss function and a location perception loss function.

In one embodiment of the present disclosure, the cross-entropy loss function is the probability that the text description corresponding to the image sample produced by the reflection decoding network to be trained is correct, and the location perception loss function is currently the The distance between the actual and predicted positions in the text description samples of the word vectors output by the reflection decoding network to be trained.

In one embodiment of the present disclosure, a position perception loss corresponding to said position perception loss function is determined by said reflective position module. The image processing device 1100 generates fully connected information by fully connecting the features output from the reflective attention module using a fully connected layer, and compresses the fully connected information according to a preset compression function. obtaining a predicted position of a word vector corresponding to the output feature of the reflective attention module, and comparing the predicted position with the actual position in the text explanation sample of the word vector corresponding to the output feature of the reflective attention module based on which the position perception loss is determined.

FIG. 12 shows a schematic diagram showing the configuration of a computer system suitable for implementing the image processing apparatus according to the embodiment of the present disclosure.

It should be noted that the electronic device computer system 1200 shown in FIG. 12 is merely exemplary and is not intended to suggest any limitation as to the scope of functionality or use of the embodiments of the present disclosure.

As shown in FIG. 12, a computer system 1200 includes a Central Processing Unit (CPU) 1201, which stores programs, or storage parts, in a Read-Only Memory (ROM) 1202. According to a program loaded from 1208 to Random Access Memory (RAM) 1203, various appropriate operations and processes can be performed to realize the image processing method according to the above embodiments. The RAM 1203 also stores various programs and data necessary for system operation. CPU 1201 , ROM 1202 and RAM 1203 are interconnected via bus 1204 . Input/Output (I/O) interface 1205 is also connected to bus 1204 .

Components connected to the I/O interface 1205 include the following members: an input portion 1206 including a keyboard, mouse, etc.; Cathode Ray Tube (CRT), Liquid Crystal Display (LCD) and speaker, etc. a storage unit 1208 including a hard disk or the like; and a communication unit 1209 including a network interface card such as a LAN (Local Area Network) card, modem, or the like. A communication unit 1290 performs communication processing via a network such as the Internet. Drivers 1210 are also connected to I/O interfaces 1205 as needed. If necessary, a removable medium 1211 such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc. is attached to the drive 1210, so that the computer program read therefrom is installed in the storage unit 1208 as necessary. It will be done.

In particular, according to one embodiment of the present disclosure, the processes described below with reference to flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure comprises a computer program product comprising a computer program embodied on a computer readable medium, the computer program including program code for performing the method illustrated in the flowcharts. . In such embodiments, the computer program may be downloaded and installed from a network via communications portion 1209 and/or installed from removable media 1211 . When this computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this disclosure.

It should be noted that the computer-readable media presented in the embodiments of the present disclosure may be computer-readable signal media or computer-readable storage media, or any combination of the two. A computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. More specific examples of computer-readable media include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable It can include, but is not limited to, combinations. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program. The program may be used by or in combination with an instruction execution system, apparatus or device. In the present disclosure, a computer readable signal medium, which may include a data signal propagated in baseband or as part of a carrier wave, has computer readable program code embodied therein. This propagated data signal may be in various forms including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium. Such computer-readable media are capable of transmitting, propagating or transmitting programs for use by or in combination with any instruction execution system, apparatus or device. Program code contained on a computer readable medium may be transmitted by any suitable medium including, but not limited to, wireless, wired, etc. or any suitable combination of the above.

The flowcharts and block diagrams in the accompanying drawings illustrate possible architectures, functionality, and operation of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a module, program segment, or portion of code, wherein such module, program segment, or portion of code includes a given logic function. contains one or more executable instructions for implementing It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks listed subsequently may actually be executed substantially in parallel, or may be executed according to the reverse order, depending on the functionality involved. It should be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by dedicated hardware-based systems or dedicated hardware and computer systems that perform the specified functions or operations. can be realized by combining with the instructions of

The units described in the embodiments of the disclosure may be implemented in software or hardware. The described units may also be located in the processor. Therein, the names of these units do not limit the units themselves in any particular situation.

In another aspect, the disclosure further provides a computer-readable medium. The computer-readable medium may be included in the image processing apparatus described in the embodiments above, or may exist alone without being equipped with an electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device is enabled to implement the methods described in the embodiments.

Note that although the above disclosure details several modules or units of the device for performing operations, such partitions are not required. Indeed, according to embodiments of the present disclosure, features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further partitioned to be embodied by multiple modules or units.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with the necessary hardware. would get. Therefore, technical solutions according to embodiments of the present disclosure can be embodied in the form of software products. This software product is stored on a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk) or a network, and is stored on a computing device (personal computer, server, touch terminal or network device, etc.). It contains some instructions for performing a method according to an embodiment.

Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and the invention disclosed herein. This disclosure is intended to cover any adaptations, uses, or suitable modifications of the disclosure. These modifications, uses, or appropriate modifications follow the general principles of the present disclosure and include common knowledge or common technical means in the technical field that are not disclosed in the present disclosure.

It is to be understood that this disclosure is not limited to the precise constructions described above and shown in the drawings, and that various modifications and changes may be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

1100 Image description generator
1101 Feature Extraction Module
1102 Feature Conversion Module
1103 description generation module
1205 interface
1206 Input section
1207 Output section
1208 memory
1209 Communications Department
1210 driver
1211 removable media

Claims (15)

An image processing method performed by one or more computing devices, comprising:
obtaining an input image and extracting a region feature of each image region in the input image to obtain a first image feature;
processing pixels in the first image feature according to preset rules and identifying a second image feature based on the processed pixels;
identifying word vectors corresponding to region features corresponding to each of the image regions in the first image features based on the second image features and at least one word vector identified for the input image; , predicting positions in a text description of said word vectors and generating a text description corresponding to said input image based on said word vectors and said positions;
method including. The step of extracting a region feature of each image region in the input image to obtain a first image feature includes:
dividing the input image to form a plurality of the image regions;
performing feature extraction on objects in the image region by means of a coded network model to obtain region features corresponding to the objects in the image region;
forming said first image feature based on said region feature;
2. The method of claim 1, comprising: corresponding region features corresponding to each of the image regions in the first image feature at different times based on the second image feature and at least one word vector identified for the input image; The step of identifying word vectors includes:
corresponding to each of the image regions in the first image feature at different times based on the second image feature and at least one word vector identified for the input image by a reflection decoding network model; 3. The method of claim 1 or 2 , comprising identifying word vectors corresponding to the region features. the reflective decoding network model includes a plurality of sequentially arranged reflective decoding sub-networks;
for each image region in the first image feature at a different time based on the second image feature and at least one word vector identified for the input image by the reflection decoding network model; Identifying word vectors corresponding to the corresponding region features comprises:
inputting the second image feature and the word vector output from the Mth reflection decoding sub-network to the M+1th reflection decoding sub-network;
identifying word vectors corresponding to region-of-interest features in the first image feature by the M+1 th reflection decoding sub-network, where M is a positive integer;
4. The method of claim 3, comprising: for each image region in the first image feature at a different time based on the second image feature and at least one word vector identified for the input image by the reflection decoding network model; Identifying word vectors corresponding to the corresponding region features comprises:
inputting said second image feature and a starting word vector into a first reflective decoding sub-network, said first reflective decoding sub-network generating said first image feature and said starting word vector based on said second image feature and said starting word vector; identifying a word vector corresponding to a region-of-interest feature in the image features of , wherein the starting word vector is the starting mark of the textual description;
4. The method of claim 3, comprising: The reflection decoding sub-network comprises:
5. The method of claim 4, comprising a reflective position module for predicting relative positions in the textual description of word vectors output from the reflective decoding sub-network at present. the reflex decoding sub-network further comprising a visual attention module including a first long short-term memory network, a second long short-term memory network and an attention mechanism network;
wherein the step of identifying word vectors corresponding to region-of-interest features in the first image features by the M+1 th reflection decoding sub-network comprises:
multiplying the word vector output from the M+1 th reflection decoding sub-network at the previous time by a first weight matrix to obtain a target word vector;
performing feature extraction on the second image feature and the target word vector by the first long short term memory network to identify a first hidden state of the first long short term memory network;
inputting the first hidden state and the first image feature into the attention mechanism network for visual matching to obtain a region of interest feature;
performing feature extraction on the first hidden state and the region of interest features by the second long short term memory network to identify a second hidden state of the second long short term memory network;
identifying a word vector corresponding to the target region feature based on the first hidden state and the second hidden state;
5. The method of claim 4, comprising: the reflective decoding sub-network further comprising a reflective attention module;
wherein, based on the first hidden state and the second hidden state, identifying a word vector corresponding to the target region feature comprises:
The reflective attention module generates third output information corresponding to the target area feature at the present time based on the second hidden state at the past time point and the first hidden state and the second hidden state at the present time. a step of identifying;
multiplying the third output information by a third weight matrix to obtain a word vector corresponding to the target region feature;
8. The method of claim 7, comprising: Identifying third output information currently corresponding to the region of interest feature comprises:
identifying a target matrix based on all the past time point second hidden states and the current time point second hidden states;
A step of performing dimension reduction processing on the target matrix to obtain first feature information, and performing dimension reduction processing on the first hidden state at the current time point to obtain second feature information. wherein the first feature information and the second feature information have the same number of dimensions;
summing the first feature information and the second feature information based on an attention mechanism to obtain third feature information;
obtaining a second weight matrix by performing weighting and normalization on the third feature information;
multiplying the second weight matrix by the first feature information to obtain a sum of products to obtain third output information;
9. The method of claim 8, comprising: Before obtaining the input image, the method further comprises:
obtaining an image sample and a text description sample corresponding to the image sample;
training the reflection decoding network model to be trained such that the image samples and the text description samples minimize a loss function corresponding to the reflection decoding network model to be trained;
4. The method of claim 3, wherein the loss function comprises a cross-entropy loss function and a location perception loss function. the cross-entropy loss function is the probability that the text description corresponding to the image sample produced by the reflection decoding network to be trained is correct;
11. The method of claim 10, wherein the position perception loss function is the distance between actual and predicted positions in text description samples of word vectors currently output by the reflection decoding network to be trained. a position perception loss corresponding to the position perception loss function is determined by a reflective position module;
The method further comprises:
generating fully-connected information by fully-connecting features output from the reflective attention module by a fully-connected layer;
compressing the total connectivity information according to a preset compression function to obtain predicted positions of word vectors corresponding to output features of the reflective attention module;
determining the position perception loss based on the predicted positions and actual positions in the text description sample of word vectors corresponding to output features of the reflective attention module;
12. The method of claim 11, comprising: a feature extraction module that acquires an input image and extracts a region feature of each image region in the input image to acquire a first image feature;
a feature transformation module that processes pixels in the first image feature according to preset rules and identifies a second image feature based on the processed pixels;
Identifying word vectors corresponding to region features corresponding to each of the image regions in the first image features based on the second image features and at least one word vector identified for the input image. a description generation module for predicting positions in a text description of said word vectors and generating a text description corresponding to said input image based on said word vectors and said positions;
An image processing device comprising: one or more processors;
a storage device for storing one or more programs;
An electronic device that causes the one or more processors to implement the image processing method according to any one of claims 1 to 12 when one or more programs are executed by the one or more processors. A computer program , executed by one or more processors, implementing the image processing method according to any one of claims 1 to 12,
A computer program characterized by:

Images