JP7416614B2 - Learning model generation method, computer program, information processing device, and information processing method - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Published on June 4, 2020 at the 2019 National Conference of the Japanese Society for Artificial Intelligence (33rd) Data scientist: Yurufuwa recruitment on July 1, 2020 Published at a roundtable discussion Published at CCSE 2019 on July 13, 2020 Published at Data Driven Developer Meetup #6 on July 16, 2020

The present invention relates to a method for generating a learning model that recognizes character strings from images, a computer program, an information processing device, and an information processing method.

Various learning methods have been proposed for recognizing character strings in images using neural networks. Patent Document 1 discloses a method for improving the accuracy of character string recognition processing from image data of a form by learning using a database in which words that can be written on a form are registered.

Patent No. 6590355

Although character recognition using neural networks is possible in a variety of situations, it is still difficult to recognize characters from any image without any particular conditions. In order to improve character recognition accuracy using a neural network and to improve learning efficiency, as shown in Patent Document 1, the target image is a specific image and words that are not to be recognized are excluded. , it is necessary to make it possible to make analogies from the preceding and following characters.

The present invention provides a method for generating a learning model for character recognition, a computer program, an information processing device, and an information processing method that can learn accurately and efficiently even when the amount of teacher data is small. With the goal.

A learning model generation method according to an embodiment of the present disclosure includes a set of image data of an image in which a character string is shown and text data of the character string, and a set of image data and text data of the character string, and whether or not the character string is shown in the image data and the image data. A learning model that outputs text data and presence/absence data of a character string in the image when image data is input is obtained using the training data.

In the learning model generation method of the present disclosure, the learning model uses image data of an image without a character string, image data of an image with a character string, and correct data of the character string in the image. It is generated by training the same network to determine whether a character string is present or not and to recognize the character string.

The information processing device according to an embodiment of the present disclosure is trained to output text data and presence/absence data that are the recognition results of character strings appearing in the image of the image data when image data is input. A storage unit in which a model is stored, an image acquisition unit that acquires image data, and an image acquisition unit that inputs the image data acquired by the image acquisition unit into the model, based on the text data and presence/absence data output from the model. , an output unit that determines whether a character string is included in the image of the image data and outputs the included character string.

An information processing method according to an embodiment of the present disclosure is trained to output, when image data is input, text data and presence/absence data that are the recognition results of a character string appearing in an image of the image data. A model is memorized, image data is acquired, the acquired image data is input to the model, and based on the text data and presence/absence data output from the model, whether a character string is reflected in the image of the image data is determined. Includes processing to determine whether or not the image is displayed and to output the character string in the image.

A computer program according to an embodiment of the present disclosure is a model trained to output, when image data is input, text data and presence/absence data that are the recognition results of a character string appearing in an image of the image data. is stored in a computer, the acquired image data is input to the model, and based on the text data and presence/absence data output from the model, a character string is reflected in the image of the image data. The process of outputting whether or not there is a character in the image and the character string in the image is executed.

In the computer program of the present disclosure, recognition accuracy can be improved using a learning model even for a target with low resolution and little training data, such as characters in an image captured from a vehicle.

A computer program according to an embodiment of the present disclosure includes a first model trained to detect, when image data is input, a range in which a character string to be detected is included in an image of the image data; The first image data was acquired into a computer storing a second model that had been trained to output text data indicating character strings in the image and presence/absence data when image data was input. First image data is input to the first model, second image data is obtained by extracting the detection range detected by the first model from the image of the first image data, and second image data is input to the first model. The detection range and the confidence that a character string is included in the detection range are input to the second model and output from the first model, and the text data output from the second model and output together with the text data. A process of storing the certainty factor, which is a character string, in association with the first image data is executed.

According to the generation method of the present disclosure, even with a small amount of training data, a learning model that can recognize character strings with the same degree of accuracy as when there is a large amount of training data can be generated.

FIG. 2 is a block diagram of an information processing device that performs character recognition. It is a schematic diagram of a learning model. 3 is a flowchart illustrating an example of a learning model generation method. FIG. 2 is a schematic diagram of learning of a learning model. FIG. 2 is a schematic diagram of learning of a learning model. It is a graph showing the relationship between the amount of learning data and the accuracy. FIG. 3 is a schematic diagram of an information providing service according to a second embodiment. FIG. 2 is a block diagram showing the configuration of a communication device that implements an information providing service. FIG. 2 is a block diagram showing the configuration of a collection device, an information processing device, and a storage device. It is a flowchart which shows an example of the processing procedure by a collection device and an information processing device. The results of processing an image captured by a drive recorder are shown. FIG. 2 is a block diagram showing the configuration of an information providing device and an information terminal device. A display example of information provided by an information providing device is shown.

The present disclosure will be specifically described with reference to drawings showing embodiments thereof.

(Embodiment 1)
FIG. 1 is a block diagram of an information processing device 1 that performs character recognition. The information processing device 1 includes a control section 10, an image processing section 11, a storage section 12, a communication section 13, and a reading section 14. Although the information processing apparatus 1 and the operation thereof will be described below as one server computer, it may be configured such that processing is distributed among a plurality of computers.

The control unit 10 uses a processor such as a CPU (Central Processing Unit), a memory, and the like to control the constituent parts of the apparatus and realize various functions. The image processing unit 11 uses a GPU (Graphics
Using a processor such as a processing unit (Processing Unit) or a dedicated circuit and a memory, image processing is executed in accordance with control instructions from the control unit 10, and functions as a learning model 1M. Note that the control unit 10 and the image processing unit 11 may be integrated hardware. Furthermore, the control unit 10 and the image processing unit 11 may be configured as one piece of hardware (SoC: System On a Chip) that integrates a processor such as a CPU or GPU, a memory, and furthermore, a storage unit 12 and a communication unit 13. good.

The storage unit 12 uses a hard disk or flash memory. The storage unit 12 stores a learning program 1P and a DL (deep learning) library 1L. In addition, the storage unit 12 stores definition information that defines the learning model 1M generated by learning using the DL library 1L, parameter information including weighting coefficients of each layer in the trained (currently learning) learning model 1M, etc. be done.

The storage unit 12 stores a learning program (computer program) 1P for executing learning of the learning model 1M. The control unit 10 executes a network parameter learning process based on the definition information stored in the storage unit 12 based on the learning program 1P.

The communication unit 13 is an interface that inputs data from the outside and transmits data to the outside. The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet. The communication unit 13 may include a network card, a wireless communication device, or a carrier communication module. The communication unit 13 may be a USB interface.

The reading unit 14 can read the learning program 71P and the DL library 71L stored in the recording medium 7 using an optical disk or the like using, for example, a disk drive. The DL library 1L stored in the storage unit 12 may be a copy of the learning program 71P read by the reading unit 14 from the recording medium 7 into the storage unit 12 by the control unit 10.

The information processing device 1 configured as described above operates a convolutional neural network (hereinafter referred to as a CNN) and a recurrent NN based on the DL library 1L and the learning program 1P stored in the storage unit 12. , performs a process of learning parameters to make the learning model 1M.

FIG. 2 is a schematic diagram of the learning model 1M. The learning model 1M includes a CNN 121 that inputs character string image data and outputs feature amounts, and an RNN 122 that is a Bi-LSTM (Long Short Term Memory) that sequentially inputs the output from the CNN 121. The learning model 1M includes a CRF (Conditional Random Field) layer 123 that outputs a probability distribution of plausible characters from the output of the RNN 122.

The learning model 1M has a first output layer 124 which inputs the branched output from the CRF layer 123 and outputs the data of the character string appearing in the image, and a first output layer 124 which inputs the branched output and outputs the data of the character string shown in the image. and a second output layer 125 that outputs presence/absence data of the image.

In this way, the learning model 1M of the present disclosure uses the CNN 121 as an input, and is generated by multitask learning in which the first output layer 124 and the second output layer 125 that output different data are simultaneously trained.

The CNN 121 includes a filter that extracts mutually overlapping ranges in the parallel direction of character strings from an image of input image data. The CNN 121 sequentially outputs feature amounts for the sequentially extracted range of image data along the parallel direction of the character strings.

The RNN 122 sequentially inputs the feature quantities output from the CNN 121, and sequentially outputs, for each input range, the confidence of characters appearing in that range. At this time, the RNN outputs the confidence level including blank characters.

The CRF layer 123 outputs a combination of characters that is most likely to appear in the image of the image data input to the CNN 121 based on the characters and angles sequentially output from the RNN 122 .

The first output layer 124 considers that the combination of character strings output from the CRF layer 123 is extracted redundantly by the filter of CNN 121, and determines that consecutive character strings are the same character, The character string that is most likely to appear in the image of the image data input to CNN 121 is output. As shown in FIG. 2, the first output layer 124 uses CTC (Connectionist Temporal Classification), and if the same character is output continuously without a blank space, it is output as one character. do.

The second output layer 125 is a determination result of whether or not a character string appears in the image of the image data input to the CNN 121, based on the combination of character strings output from the CRF layer 123 and their probabilities. Output presence/absence data.

FIG. 3 is a flowchart showing an example of a method for generating the learning model 1M, and FIGS. 4 and 5 are schematic diagrams of learning of the learning model 1M. The schematic diagrams shown in FIGS. 4 and 5 show examples of the learning model 1M of FIG. 2 and teacher data.

The control unit 10 uses the image processing unit 11 based on the learning program 1P and the DL library 1L, and as shown in FIG. 125 is generated (step S101).

As shown in FIG. 4, the control unit 10 provides image data of an image containing a character string (in FIG. 4, a number string) to the CNN 121 of the network (step S102). The control unit 10 specifies the character string and its reliability output from the first output layer 124 of the network, and the presence/absence data (probability) output from the second output layer 125 (step S103).

The control unit 10 includes the text, confidence level, and presence/absence data of the character string identified in step S103, the text (correct answer) representing the character string shown in the image, which is teacher data, and the presence/absence of the character string. The loss is calculated using the presence/absence data (correct answer) shown (step S104).

The control unit 10 uses the image processing unit 11 to execute learning to back-propagate the loss obtained in step S104 to the network, and updates the parameters (step S105). In step S105, the control unit 10 repeatedly performs learning for one image data until the number of learning times reaches a predetermined number or the accuracy of the character string output from the first output layer 124 reaches a predetermined accuracy. good.

As shown in FIG. 5, the control unit 10 provides image data of an image without a character string to the CNN 121 of the network (step S106). The control unit 10 specifies only the presence/absence data (probability) output from the second output layer 125 without using the output from the first output layer 124 (step S107).

The control unit 10 calculates the loss using the presence/absence data identified in step S107 and the presence/absence data (correct answer) indicating that the character string does not exist in the image, which is the teacher data (step S108).

The control unit 10 uses the image processing unit 11 to perform learning to back-propagate the loss obtained in step S108 to the network, and updates the parameters (step S109). Also in step S109, the control unit 10 may repeatedly perform learning for one piece of image data until the accuracy of the presence/absence data output from the second output layer 125 reaches a predetermined accuracy.

The control unit 10 determines whether learning processing has been performed on all of the image data prepared as teacher data (step S110). When the control unit 10 determines that the learning process has not been performed for all of the items (S110: NO), the process returns to step S102. If it is determined that the learning process has been performed for all of them (S110: YES), the control unit 10 ends the learning process.

In the procedure shown in the flowchart of FIG. 3, the control unit 10 selects one image that includes a character string, executes the processing in steps S102 to S105, and then selects one image that does not include a character string. It has been described that the processing of steps S106 to S109 is repeated alternately until the processing is completed for all image data. However, the control unit 10 executes steps S102-S105 for all image data of images in which character strings are shown, and then processes steps S106-S109 for all image data of images in which character strings are not shown. Processing may also be executed.

When relearning, the control unit 10 preferably executes the processes of steps S102 to S110 using new image data.

In this way, the learning model 1M uses the image data of images that do not include character strings, the image data of images that do include character strings, and the correct data for the character strings that appear in those images. It is generated by training the same network to determine whether the image is present or not and to recognize character strings. This allows the learning model 1M to learn the shape of the character string (number) itself, and the recognition accuracy improves even with a small amount of training data.

In the first embodiment, the learning model 1M was described as being configured with CNN-RNN as shown in FIG. 2-5. The learning model 1M may be realized without using a filter that sequentially extracts images in the character string direction, and without using the RNN 122 and the CRF layer 123. For example, the learning model 1M may be realized using the CNN 121 that inputs image data and other known networks for image recognition.

FIG. 6 is a graph showing the relationship between the amount of learning data and the accuracy. FIG. 6 shows the transition of improvement in recognition accuracy with respect to the consumption rate of training data. In FIG. 6, the solid line indicates the transition in accuracy during the process of generating the learning model 1M using the learning method described above. The broken line shows the transition in accuracy in a model that does not use the second output layer 125, that is, a model that is simply trained to recognize character strings.

As shown in FIG. 6, it can be seen that multitask learning, which updates network parameters using multiple outputs, improves accuracy to the same extent as the accuracy after learning, even with a small amount of training data.

(Embodiment 2)
In the embodiment, a character string is recognized by the information processing device 1 described in Embodiment 1, and a database is created that stores the recognized character string and the position information where the image containing the character string is photographed in association with each other. and realizes an information provision service that provides information based on text data indicated by character strings. In the second embodiment, the recognized character string is a number string indicating the price of gasoline at a gas station.

FIG. 7 is a schematic diagram of the information providing service according to the second embodiment. The information providing service includes a vehicle V that travels on a road where a gas station is provided, an information processing device 1, a communication device 2, a collection device 3, a storage device 4, an information providing device 5, and an information terminal device 6.

The communication device 2 is mounted on the vehicle V. The communication device 2 transmits image data obtained by a drive recorder provided in the vehicle V to the collection device 3. A communication connection is possible between the communication device 2 and the collection device 3 via a network N including a public network N1 and a carrier network N2. The network N may include a dedicated network for a communication connection between the vehicle V and the collection device 3, or may be a network used for communication of road traffic information.

The collection device 3 collects image data from a plurality of vehicles V and stores it in the storage device 4. The collection device 3, the information processing device 1, the storage device 4, and the information providing device 5 are communicatively connected to each other via the service provider's local network LN, and to other devices via the external network N. It is possible. The collection device 3 provides the image data stored in the storage device 4 to the information processing device 1. The collection device 3 acquires data output from the information processing device 1 indicating whether or not a number string is included in the image of the image data, and if so, what the numerical value is indicated by the number string. The collection device 3 stores the acquired numerical data, that is, the numerical data indicating the price of gasoline, in the storage device 4 in association with the position data indicating the traveling position of the vehicle V when the image data was captured, and sequentially stores the numerical data in the storage device 4. Update to the latest data.

The information providing device 5 can be communicatively connected to the information terminal device 6 via the network N. The information providing device 5 provides a service for displaying the price of gasoline on a map based on numerical data indicating the price of gasoline that is associated with the position data stored in the storage device 4.

FIG. 8 is a block diagram showing the configuration of the communication device 2 that implements the information providing service. The communication device 2 includes a communication unit 22 and a GPS receiver 23, and sequentially acquires image data of moving images from a photographing device 20 (for example, a drive recorder) mounted on the vehicle V so as to photograph the outside of the vehicle. The communication device 2 uses the GPS receiver 23 to acquire position data indicating the position of the vehicle V at the time when the image data is acquired from the photographing device 20, and associates it with the image data. The communication device 2 transmits the associated image data and position data to the collection device 3 using a communication unit 22 that implements wireless communication such as a carrier network, a dedicated network, or a beacon. The communication device 2 may transmit identification information and time information of the vehicle V in which the photographing device 20 is installed in association with image data and position data.

FIG. 9 is a block diagram showing the configurations of the collection device 3, the information processing device 1, and the storage device 4. The collection device 3 includes a control section 30, an image processing section 31, a storage section 32, and a communication section 33. The control unit 30 uses a processor such as a CPU, a memory, and the like to control the components of the device and realize various functions. The image processing unit 31 uses a processor such as a GPU or a dedicated circuit and a memory to perform image processing in accordance with control instructions from the control unit 30. Note that the control section 30 and the image processing section 31 may be integrated hardware.

The storage unit 32 stores a processing program 30P and an image processing program 31P. The storage unit 32 stores definition data of a detection model 3M that is used by the image processing unit 31 to detect an area where the price of gasoline is likely to be captured.

The control unit 30 receives image data and position data from the communication device 2 based on the processing program 30P, associates at least an identification ID for identifying the image data, and data on the received date and time, and transmits the data to the storage device 4. and executes the process of storing it.

Based on the image processing program 31P, the control unit 30 uses the image processing unit 31 to execute area detection processing using the detection model 3M, which detects an area where the price of gasoline is likely to be captured from the acquired image data. do. The detection model 3M is trained to detect an area where the price of gasoline is shown from an image captured by a drive recorder. The detection model 3M uses, for example, an SSD (Single Shot Detector/Single Shot Multibox Detector) or the like, and has been trained in advance based on training data indicating that the area is an area where the price of gasoline is shown. The image processing unit 31 may also use a known method for extracting from the image a region that is likely to contain a specific object in the image data.

The communication unit 33 is a communication module that realizes a communication connection with the storage device 4 and the information processing device 1 via the local network LN or the network N. The communication unit 33 is, for example, a network card.

The storage device 4 includes a control section 40, a storage section 41, and a communication section 42. The control unit 40 uses a processor such as a CPU. The storage unit 41 uses a large-capacity nonvolatile memory such as a hard disk or SSD (Solid State Drive). The storage unit 41 stores image data and position data captured and transmitted by the drive recorder of the vehicle V, image data obtained by extracting a character string portion from the image data, and text data recognized from the extracted image data. (numerical data) and presence/absence data are stored in association with the identification ID.

FIG. 10 is a flowchart illustrating an example of a processing procedure by the collection device 3 and the information processing device 1.

The control unit 30 acquires image data from the communication device 2 through communication (step S301). The control unit 30 acquires position data that is transmitted in association with the image data (step S302).

The control unit 30 stores the image data and position data in the storage device 4 in association with the identification ID (step S303).

The control unit 30 inputs the image data to the detection model 3M (step S304), and determines the detection range output from the detection model 3M and the certainty that the character string (number string) indicating the price of gasoline is included in the detection range. Data indicating the degree is acquired (step S305).

The control unit 30 determines whether the area in which the price of gasoline is shown has been detected based on the confidence level (step S306). When a plurality of areas are detected in step S305, the control unit 40 executes the process in step S306 and the processes in and after step S307 for each area.

If it is determined in step S306 that the area in which the price is captured has been detected (S306: YES), the control unit 30 extracts the detection range from the image of the image data acquired in step S301 (step S307). , obtains image data of the extracted image (step S308).

The control unit 30 transmits the extracted image data to the information processing device 1 from the communication unit 33 (step S309).

The information processing device 1 acquires the extracted image data transmitted from the collection device 3 from the communication unit 13 (step S121). The control unit 10 of the information processing device 1 inputs the acquired image data to the image processing unit 11 that functions as the trained learning model 1M (step S122).

The image processing unit 11 outputs, as the learning model 1M, text data (numeric data) indicating the price, data on the reliability of the recognized numerical value, and presence/absence data indicating whether the price is included in the image (step S123). .

The control unit 10 transmits text data (numeric data) and presence/absence data from the communication unit 13 to the collection device 3 (step S124).

The control unit 30 of the collection device 3 uses the communication unit 33 to acquire the text data and presence/absence data transmitted from the information processing device 1 (step S310). The control unit 30 stores the acquired text data and presence/absence data in the storage device 4 in association with the identification ID (step S311), and ends the process.

In step S311, presence/absence data is accumulated in the storage device 4 even if the presence/absence data is image data indicating that no character string is captured.

If it is determined in step S306 that the area in which the price is shown has not been detected (S306: NO), the control unit 30 omits the processes in steps S307 to S311 and ends the process. In this case, the control unit 30 may delete the image data and position data stored in step S303.

Although the collection device 3 has been described as existing as hardware different from the information processing device 1 and executing the processing procedure shown in the flowchart of FIG. 10, the collection device 3 and the information processing device 1 are realized by one computer. Good too.

FIG. 11 shows processing results for images captured by a drive recorder. As shown in FIG. 11, based on the detection model 3M, an area in which a price is shown is detected and extracted with a predetermined confidence in the captured image. As shown in FIG. 11, when the extracted image is input to the learning model 1M, the numerical value indicating the price of gasoline, the confidence level of area detection, the confidence level of the numerical value, and the confidence level of the presence/absence result are output.

By being able to recognize character strings in images obtained using the information processing device 1 using the trained learning model 1M, a service that shows gasoline prices on a map is realized as shown below. be done.

FIG. 12 is a block diagram showing the configurations of the information providing device 5 and the information terminal device 6. As shown in FIG. The information providing device 5 is a server computer and includes a control section 50, a storage section 51, and a communication section 52.

The control unit 50 uses a processor such as a CPU or GPU, a memory, and the like to control the components of the device and realize various functions. The storage unit 51 stores an information providing program 5P, a web server program for accepting requests from the information terminal device 6, and map data. The storage unit 51 also stores data that the control unit 50 refers to.

The communication unit 52 is a network card or a wireless communication module for realizing a communication connection with the information terminal device 6 via the network N. The communication unit 52 includes a network card or a wireless communication module for realizing a communication connection via the network N in order to read data from the storage device 4.

Based on the information provision program 5P and based on a request from the information terminal device 6, the control unit 50 displays data on a web page that shows the price of gasoline on a map in a predetermined range including the location data specified in the request. It is transmitted to the terminal device 6.

The information terminal device 6 is a computer such as a personal computer, a smartphone, or a tablet terminal. The information terminal device 6 includes a control section 60, a storage section 61, a display section 62, an operation section 63, and a communication section 64.

The control unit 60 is a processor using a CPU or GPU. The control unit 60 includes a processor such as a CPU or GPU, a memory, and the like. The control unit 60 communicates with the information providing device 5 based on a general-purpose Web browser-based display program 6P stored in the storage unit 61, and operates the general-purpose computer as a terminal device that receives information providing services.

The storage unit 61 includes, for example, a nonvolatile memory such as a flash memory. The storage unit 61 stores the above-mentioned display program 6P.

The display unit 62 includes a display device such as a liquid crystal panel or an organic EL display. The operation unit 63 is an interface that accepts user operations, and includes physical buttons and a touch panel device with a built-in display. The operation unit 63 can accept operations on the screen displayed on the display unit 62 using physical buttons or a touch panel. The operation unit 63 may include a microphone or the like, and may recognize the operation content from the input voice using the microphone and accept the operation.

The communication unit 64 is a network card or a wireless communication module for realizing a communication connection with the information terminal device 6 via the network N.

Based on the display program 6P, the control unit 60 transmits the position data received by the operation unit 63 as a request to the information providing device 5, and causes the display unit 62 to display the price of gasoline around the location indicated by the position data. can.

FIG. 13 shows a display example of information provided by the information providing device 5. As shown in FIG. In FIG. 13, text or images of gasoline prices recognized from the image data of images captured at each location are displayed superimposed on the map image. This allows the operator of the information terminal device 6 to grasp the price of gasoline. The price of gasoline is updated to the latest data each time image data captured by the vehicle V is transmitted. Since it is stored in the storage device 4 as a history, the control unit 60 may display the change in gasoline prices on the display unit 62 in accordance with the operation.

In this way, even if it is a character string in a relatively small range of low-resolution images, such as the price of gasoline at a gas station captured by a drive recorder, recognition using learning model 1M can achieve high resolution. You can recognize prices with precision.

In the second embodiment, the image data collected by the collection device 3 is image data of an image captured by the drive recorder of the vehicle V. The vehicle V is not limited to a taxi, and may be a transport vehicle. The vehicle may also have an automatic driving function. Further, the image data to be recognized as a character string (number string) is not limited to the vehicle V, and may be data taken by a photographing device mounted on an unmanned aircraft such as a so-called drone.

The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is indicated by the claims, and includes all changes within the meaning and range equivalent to the claims.

1 Information processing device 10 Control unit 11 Image processing unit 12 Storage unit 13 Communication unit 1M Learning model (second model)
121 CNN
122 RNN
123 CRF layer 124 First output layer 125 Second output layer 1P Learning program 3 Collection device 30 Control section 32 Storage section 30P Processing program 31P Image processing program 3M Detection model (first model)
4 Storage device 41 Storage section 6 Information terminal device 62 Display section

Claims (15)

A method for generating a learning model that outputs text data and presence/absence data of character strings in the image when image data is input , the method comprising:
The learning model includes a convolutional neural network that separates and sequentially inputs image data, and a recurrent neural network that sequentially inputs feature data output from the convolutional neural network,
The recurrent neural network includes a first output layer that outputs a character string appearing in an image of input image data, and a second output layer that outputs presence/absence data of a character string in the image,
For the learning model, a set of image data of an image in which a character string is shown and text data of the character string, image data of an image in which a character string is shown and an image in which no character string is shown, and the image data. For each, obtain training data including a pair with presence/absence data indicating whether or not a character string is included in the image,
When image data of an image containing a character string included in the teacher data is input, the convolutional neural network uses the output from the first output layer and the output from the second output layer. learn the parameters in
When image data of an image that does not include a character string and is included in the teacher data is input, the parameters in the convolutional neural network and the recurrent neural network are set without using the output from the first output layer, A method for generating a learning model in which learning is performed using a loss function using the output from the second output layer . The convolutional neural network is configured to sequentially extract mutually overlapping predetermined ranges in the horizontal direction from images of input image data, and input image data of images in the extracted predetermined ranges. Generation method. When image data of an image showing the character string is input, the probability that the correct text data of the teacher data will be obtained using the output from the first output layer and the output from the second output layer. learning parameters in the convolutional neural network and recurrent neural network so as to maximize
The production method according to claim 1 or 2 . The learning model is
When image data of an image showing a character string is input, outputting text data of the character string and presence/absence data indicating that the character string is shown;
According to any one of claims 1 to 3 , the device is trained to output presence/absence data indicating that a character string is not included when image data of an image that does not include a character string is input. How to generate. The generation method according to any one of claims 1 to 4 , wherein the image data is image data photographed by a photographing device mounted outward on a vehicle. A computer program that generates, when image data is input to a computer, a learning model that outputs text data and presence/absence data of a character string in the image, the computer program comprising:
The learning model includes a convolutional neural network that separates and sequentially inputs image data, and a recurrent neural network that sequentially inputs feature data output from the convolutional neural network,
The recurrent neural network includes a first output layer that outputs a character string appearing in an image of input image data, and a second output layer that outputs presence/absence data of a character string in the image,
to the computer;
For the learning model, a set of image data of an image in which a character string is shown and text data of the character string, image data of an image in which a character string is shown and an image in which no character string is shown, and the image data. For each, obtain training data including a pair with presence/absence data indicating whether or not a character string is included in the image,
When image data of an image containing a character string included in the teacher data is input, the convolutional neural network uses the output from the first output layer and the output from the second output layer. learn the parameters in
When image data of an image that does not include a character string and is included in the teacher data is input, the parameters in the convolutional neural network and the recurrent neural network are set without using the output from the first output layer, Learning by a loss function using the output from the second output layer
A computer program that performs a process. A method for generating a learning model that outputs text data and presence/absence data of a character string in the image when a computer inputs image data, the method comprising:
The learning model includes a convolutional neural network that separates and sequentially inputs image data, and a recurrent neural network that sequentially inputs feature data output from the convolutional neural network,
The recurrent neural network includes a first output layer that outputs a character string appearing in an image of input image data, and a second output layer that outputs presence/absence data of a character string in the image,
The computer includes:
For the learning model, a set of image data of an image in which a character string is shown and text data of the character string, image data of an image in which a character string is shown and an image in which no character string is shown, and the image data. For each, obtain training data including a pair with presence/absence data indicating whether or not a character string is included in the image,
When image data of an image containing a character string included in the teacher data is input, the convolutional neural network uses the output from the first output layer and the output from the second output layer. learn the parameters in
When image data of an image that does not include a character string and is included in the teacher data is input, the parameters in the convolutional neural network and the recurrent neural network are set without using the output from the first output layer, Learning by a loss function using the output from the second output layer
Information processing method. A computer storing a model trained by the generation method according to any one of claims 1 to 5 ,
Get the image data,
Input the acquired image data into the model,
A computer program that executes a process of determining whether or not a character string appears in an image of the image data and outputting the character string that appears, based on text data and presence/absence data output from the model. to the computer;
A request for executing a process of associating and storing the text data output from the model, information regarding the certainty that a character string is reflected in the image of the image data, which is output together with the text data, and the image data. The computer program according to item 8 . to the computer;
A claim that executes a process of storing the image data in association with data indicating the absence of a character string when the presence/absence data output from the model indicates that the character string is not included in the image. 10. The computer program according to 8 or 9 . The computer program according to any one of claims 8 to 10 , wherein the image data is image data photographed by a photographing device mounted outward on a vehicle. to the computer;
obtaining position data of the vehicle at the timing when the image data was taken;
12. The computer program according to claim 11 , wherein the computer program executes a process of storing the acquired position data in association with text data and presence/absence data output from the model by inputting the image data into the model. A first model trained to detect a range in which a character string to be detected is included in an image of the image data when image data is input; A computer that stores a second model trained by the generation method described above ,
Obtain first image data,
inputting the acquired first image data into the first model;
obtaining second image data in which a detection range detected by the first model is extracted from the first image data;
inputting second image data into the second model;
The detection range output from the first model and the confidence that the character string is included in the detection range, the text data output from the second model and the confidence that the character string is output together with the text data. A computer program that causes a computer program to execute a process of storing an image data in association with the first image data. A storage unit storing a model trained by the generation method according to any one of claims 1 to 5 ;
an image acquisition unit that acquires image data;
By inputting the image data acquired by the image acquisition unit into the model, it is possible to determine whether or not a character string appears in the image of the image data, based on the text data and presence/absence data output from the model. An information processing device comprising: an output unit that outputs a character string; Storing a trained model by the generation method according to any one of claims 1 to 5 ,
Get the image data,
Input the acquired image data into the model,
An information processing method including processing for determining whether or not a character string is shown in an image of the image data and outputting a character string that is shown based on text data and presence/absence data output from the model.

Images