JP7793492B2 - Learning devices and programs - Google Patents

Description

Embodiments of the present invention relate to a machine learning model learning device and a program for causing a computer to function as the learning device.

In order to improve the accuracy of a machine learning model, it is necessary to train the model using a large amount of training data. Training data is data that combines input data and correct labels that indicate the expected results for that input data. In other words, preparing a large amount of training data means preparing a large amount of input data that covers a wide range of possible patterns, along with correct labels for that input data. However, preparing such a large amount of training data in advance is extremely difficult. Therefore, there is a technology that generates new input data by performing various processing operations on the raw training data input data, and combines it with the correct labels of the raw training data to expand the training data.

The correct labels for training data expanded using such technology are equal to the correct labels for the original training data. However, depending on the processing performed on the input data, even the areas that the machine learning model focuses on when obtaining recognition results may be processed, resulting in the output of correct labels even though the correct labels cannot actually be obtained from the input data. There is a concern that training using such training data will result in a decrease in the accuracy of the machine learning model.

Japanese Patent Application Laid-Open No. 2020-197833

The problem that embodiments of the present invention aim to solve is to provide a learning device and program that can prevent a decrease in the accuracy of a machine learning model by training it with expanded training data.

In one embodiment, a learning device that trains a machine learning model using training data that combines input data and a correct label indicating a result expected for the input data includes an input means, an identification means, a processing means, and a determination means. The input means inputs the training data to the machine learning model. The identification means identifies an area of interest that is focused on in the input data of the training data when the machine learning model that has received the training data as input obtains a result indicated by the correct label of the training data. The processing means processes the input data in the training data. The determination means determines a correct label for the training data resulting from processing the input data based on the processing area of the input data processed by the processing means and the area of interest identified by the identification means.

FIG. 1 is a schematic diagram illustrating an overview of a product registration system according to an embodiment. FIG. 2 is a block diagram showing the functional configuration of a discount sticker recognition unit that uses machine learning. FIG. 3 is a block diagram showing circuit components required when the learning device, which is one function of the discount label recognition unit, is configured in hardware. FIG. 4 is a schematic diagram showing the main data structure of the processing table of the learning device. FIG. 5 is a flowchart showing the main steps of the information processing executed by the CPU of the learning device in accordance with the learning program. FIG. 6 is a flowchart showing one procedure of the data extension process in the information processing shown in FIG. FIG. 7 is a diagram showing an example of an image used to explain the outline of the identification unit, which is one function of the learning device. FIG. 8 is a diagram showing an example of an image used to explain the outline of the processing unit, which is one function of the learning device.

Hereinafter, an embodiment of a learning device for a machine learning model will be described with reference to the drawings.
This embodiment illustrates a learning device for a machine learning model used when acquiring information on discount stickers through image recognition.

In supermarkets and other retail stores, it is common to affix discount stickers to products to sell them at a discount when their expiration date or best-before date is approaching. When registering a product with a discount sticker attached to a POS (Point Of Sales) terminal, an operator holds the discount sticker, along with the barcode attached to the product, over an image recognition scanner. The image recognition scanner recognizes the barcode from an image captured by a camera. The scanner also recognizes the discount sticker from the image and outputs the discount sticker information, such as 10 yen off or 10% off, along with the barcode information, to the POS terminal.

In product registration systems using such scanners, machine learning techniques such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), and RNN (Recurrent Neural Network) are used as image recognition technologies to recognize discount labels from captured images. Using machine learning makes it possible to recognize information on discount labels in an extremely short time. However, improving the recognition accuracy of machine learning requires improving the accuracy of the machine learning model that derives results from input data. To improve the accuracy of a machine learning model, the machine learning model can be trained using a large amount of training data. Training data is data that combines input data and correct labels that indicate the results expected from that input data. In other words, in this embodiment, images of discount labels are used as input data, and a large amount of training data is prepared in which the information on the discount labels is used as correct labels, and the machine learning model is trained using this data.

However, even the same discount sticker will appear differently depending on the angle at which the operator holds it over the scanner. For this reason, it is necessary to prepare training data using images of discount stickers rotated at various angles as input data. In addition, the intensity of the lighting illuminating the area near the scanner varies from store to store. For this reason, it is also necessary to prepare training data using images of discount stickers illuminated by lighting of various intensities as input data. Furthermore, in the case of a self-service system in which customers themselves hold the product's barcode over the scanner, it is possible that the customer, who is also the operator, may cover part of the discount sticker with their finger or other object before holding it over the scanner. Therefore, it is also necessary to prepare training data using images of the discount sticker partially obscured with a finger or other object as input data. Such training data can be expanded by applying processing such as rotation, gamma conversion, lighting effects, occlusion, and finger placement to the discount sticker images that form the raw training data input data.

This embodiment provides a learning device and program for training a machine learning model using a large amount of training data expanded by processing discount sticker images using the processing described above. First, an overview of the product registration system 100 according to this embodiment will be described using the schematic diagram in Figure 1.

As shown in FIG. 1, the product registration system 100 includes a scanner 1, an image processing device 2, and a POS terminal 3. The scanner 1 is an image recognition model. That is, the scanner 1 is equipped with a camera and takes a picture of a product 4 held up to the camera, and supplies the image obtained to the image processing device 2. The image processing device 2 recognizes a barcode 5 attached to the product 4 from the image. The image processing device 2 also recognizes a discount label 6 attached to the product 4 from the image. The image processing device 2 can recognize the barcode 5 and the discount label 6 from a single image. The image processing device 2 can also recognize the barcode 5 and the discount label 6 from separate captured images.

The image processing device 2 is a computer that functions as an image storage unit 11, a barcode recognition unit 12, a discount label recognition unit 13, and a recognition result output unit 14. The image storage unit 11 stores frame images captured by the scanner 1 camera in chronological order. The barcode recognition unit 12 recognizes a barcode 5 from the frame images stored in the image storage unit 11. If the barcode 5 is recognized, the barcode recognition unit 12 provides information about the barcode 5 to the recognition result output unit 14. The discount label recognition unit 13 recognizes a discount label 6 from the frame images stored in the image storage unit 11. If the discount label 6 is recognized, the discount label recognition unit 13 provides information about the discount label 6 to the recognition result output unit 14. If barcode information is provided by the barcode recognition unit 12, the recognition result output unit 14 outputs the barcode information to the POS terminal 3. When information about the discount sticker 6 is provided by the discount sticker recognition unit 13, the recognition result output unit 14 outputs the information about the discount sticker 6 to the POS terminal 3.

When the POS terminal 3 acquires barcode 5 information from the image processing device 2, it registers product sales data such as the price, product name, sales quantity, and sales amount of the product identified by the product code represented by the barcode 5. When the POS terminal 3 acquires discount label 6 information from the image processing device 2, it executes discount processing. For example, the POS terminal 3 calculates the discount amount based on the information on the discount label 6 and adds the discount amount to the product sales data registered immediately before. Alternatively, the POS terminal 3 adds the discount amount to the product sales data registered immediately after calculating the discount label. For example, if the discount label information indicates a 10 yen discount, the discount amount is 10 yen. For example, if the discount label information indicates a 10% discount and the price in the product sales data immediately before or after is 200 yen, the discount amount is 20 yen.

In a product registration system 100 configured as described above, the image processing device 2 may be configured integrally with the scanner 1 or POS terminal 3, or may be configured separately from the scanner 1 and POS terminal 3. When the scanner 1 and image processing device 2 are configured integrally, the device may be referred to as a scanner device or simply a scanner. When the POS terminal 3 and image processing device 2 are configured integrally, the device may be referred to as a product registration device or simply a POS terminal. When the image processing device 2 is a standalone device, the device may be referred to as an information processing device or a data processing device.

In this embodiment, the discount sticker recognition unit 13 is realized using machine learning. Next, the functional configuration of the discount sticker recognition unit 13 using machine learning will be explained using the block diagram in Figure 2.

As shown in FIG. 2, the discount sticker recognition unit 13 includes an inference device 20, a machine learning model 30, and a learning device 40. The inference device 20 is a device specialized in inferring information about a discount sticker 6 displayed in an image using a trained machine learning model 30. The inference device 20 has the functions of an image acquisition unit 21, an image clipping unit 22, and a recognition processing unit 23. The image acquisition unit 21 acquires an image of a discount sticker 6 from the image storage unit 11. The image clipping unit 22 clips an image of the discount sticker 6 from the image acquired by the image acquisition unit 21. The recognition processing unit 23 inputs the image of the discount sticker 6 clipped by the image clipping unit 22 into the machine learning model 30 and acquires the recognition result of the machine learning model 30 in response to the input. The recognition processing unit 23 provides the acquired recognition result to the recognition result output unit 14.

The machine learning model 30 analyzes the input image of the discount sticker 6 using a machine learning technique such as DNN, CNN, or RNN, and outputs the recognition result. Specifically, if the machine learning model 30 can recognize the discount information written on the discount sticker from the input image, such as the discount amount or discount rate, it outputs the discount information as the recognition result. If the machine learning model 30 cannot recognize the discount information written on the discount sticker from the input image, it outputs information indicating that it cannot be recognized as the recognition result. The recognition result is output to the source of the image input. In other words, if the image of the discount sticker is input from the inference device 20, the machine learning model 30 outputs the recognition result to the inference device 20. If the image of the discount sticker is input from the learning device 40, the machine learning model 30 outputs the recognition result to the learning device 40.

As described above, the recognition results of the machine learning model 30 output to the inference device 20 are output to the POS terminal 3 via the recognition result output unit 14. In the POS terminal 3, when discount sticker information is provided as the recognition result of the machine learning model 30, a discount process is executed. However, if the recognition result of the machine learning model 30 is inappropriate, the operation of the POS terminal 3 results in an error. Therefore, to minimize the chance of an inappropriate recognition result, it is necessary to train the machine learning model 30 with a large amount of training data covering various conceivable patterns of discount stickers 6. On the other hand, if the machine learning model 30 incorrectly recognizes the information on the discount sticker, there is a possibility that a product will be erroneously discounted. For example, if a discount sticker 6 that says "80 yen off" has the left half of the "8" covered with a finger and is held over the scanner 1, the machine learning model 30 may erroneously recognize the information on the discount sticker 6 as "30 yen off." Such malfunctions pose a risk to the store's credibility and must be avoided as much as possible.

The learning device 40 therefore aims to train the machine learning model 30 not only to improve the recognition accuracy of the discount sticker 6 but also to prevent misrecognition. To solve this problem, the learning device 40 has the functions of an acquisition unit 41, an input unit 42, an identification unit 43, a processing unit 44, and a determination unit 45.

The acquisition unit 41 has a function for acquiring a training dataset. A training dataset is a collection of training data used in supervised learning. In other words, the training data is data in which an image of a discount sticker 6 is used as input data to the machine learning model 30, and the recognition result expected from the machine learning model 30 for that input data is used as the correct label. The acquisition unit 41 acquires a training dataset distributed, for example, via a network. The acquisition unit 41 may also acquire the training dataset by reading it from a storage medium such as a flash memory. The acquisition unit 41 can be referred to as an acquisition means.

The input unit 42 has the function of selecting one piece of training data from the training dataset acquired by the acquisition unit 41 and inputting it into the machine learning model 30. The input unit 42 also has the function of acquiring the recognition result output from the machine learning model 30 by inputting the training data into the machine learning model 30. The input unit 42 can be referred to as an input means.

The identification unit 43 is a function that identifies the attention area that the machine learning model 30, which has input training data, focuses on in the input data of the training data when obtaining a recognition result. This function uses technology that visualizes which parts of the input image the neural network focuses on. For example, it uses Grad-CAM (Class Activation Mapping), which visualizes areas of high attention using a heat map. Glad-CAM can generate a heat map. This heat map represents the degree of attention in the input image. Therefore, areas of high attention can be interpreted as attention areas where information necessary for recognition is present. The identification unit 43 can be referred to as an identification means.

The processing unit 44 is a function that processes input data in order to expand the training data that constitutes the training dataset. One or more appropriate processing methods are selected to suit the actual operational form. In this embodiment, the input data is an image of a discount sticker, so six types of processing are selected and implemented: rotation, occlusion, gamma conversion, finger placement, low lighting effect, and high lighting effect. The processing unit 44 sequentially applies the six types of processing to each piece of training data to expand the training data. The processing unit 44 may randomly select zero or more processing methods for each piece of training data and perform them. Incidentally, if there are zero processing methods, no processing is performed and the raw image is used as is. The processing unit 44 can be referred to as processing means.

The determination unit 45 is a function that, for each piece of training data, compares the processing area of the input data processed by the processing unit 44 with the area of interest identified by the identification unit 43 and determines the correct label of the training data expanded by processing the input data. For example, if the processing area of the input data overlaps at least a portion of the area of interest, the determination unit 45 changes the correct label of the training data resulting from the processing of the input data from the correct label combined with the input data before processing. Alternatively, if the area where the processing area of the input data overlaps with the area of interest is equal to or greater than a threshold, the determination unit 45 changes the correct label of the training data resulting from the processing of the input data from the correct label combined with the input data before processing. Alternatively, if the distance from the center of gravity of the area of interest to the processing area of the input data is equal to or less than a threshold, the determination unit 45 changes the correct label of the training data resulting from the processing of the input data from the correct label combined with the input data before processing. Specifically, if the processing area of the input data and the area of interest satisfy the correct label change condition, the determination unit 45 changes the correct label to "unrecognizable (reject)." The determination unit 45 may change the correct label to "unknown" instead of "unrecognizable." Alternatively, the determination unit 45 may change the correct label so that it is treated as a negative sample. The determination unit 45 can be referred to as a determination means.

Figure 3 is a block diagram showing the circuit components required when configuring the learning device 40 using hardware. The circuit components shown in Figure 3 can also be considered to be the circuit components that make up the learning device 40 and the inference device 20. They can also be considered to be the circuit components of the discount sticker recognition unit 13, which includes the machine learning model 30. Alternatively, they can be considered to be the circuit components of the image processing device 2.

As shown in FIG. 3, the learning device 40 includes a CPU (Central Processing Unit) 51, a GPU (Graphics Processing Unit) 52, a ROM (Read Only Memory) 53, a RAM (Random Access Memory) 54, an auxiliary storage device 55, a device interface 56, a communication interface 57, and a system transmission path 58. The system transmission path 58 includes an address bus, a data bus, control signal lines, and the like. The learning device 40 constitutes a computer by connecting the CPU 51 with the GPU 52, ROM 53, RAM 54, the auxiliary storage device 55, the device interface 56, and the communication interface 57 via the system transmission path 58.

The CPU 51 corresponds to the central part of the computer. The CPU 51 controls each part to realize the various functions of the learning device 40 in accordance with the operating system or application program.

The GPU 52 is a computing device that functions as an accelerator to assist the computational processing performed by the CPU 51. In the learning device 40, the GPU 52 is primarily used to perform image processing calculations at high speed.

ROM 53 and RAM 54 correspond to the main memory of the computer. ROM 53, a non-volatile memory area, stores an operating system or application programs. ROM 53 may also store data necessary for CPU 51 to execute processes to control each part. RAM 54, a volatile memory area, is used as a work area where data can be rewritten as needed. RAM 54 may also store application programs or data necessary for CPU 51 to execute processes to control each part.

The auxiliary storage device 55 corresponds to the auxiliary storage portion of the computer. For example, the auxiliary storage device 55 can be an EEPROM (Electric Erasable Programmable Read-Only Memory), a HDD (Hard Disk Drive), or an SSD (Solid State Drive). The auxiliary storage device 55 stores data used by the CPU 51 or GPU 52 when performing various processes, data created by those processes, etc. The auxiliary storage device 55 may also store the application programs mentioned above.

The device interface 56 has the function of inputting data signals input from the input device 59 and the function of outputting display data to the display device 60. The input device 59 is a keyboard, pointing device, touch panel, etc. The display device 60 is a display, touch panel, etc.

The communication interface 57 is a circuit for communicating data with external devices. External devices include a computer device that distributes the training dataset, a storage device that stores the machine learning model 30, etc.

In the learning device 40 configured as described above, the CPU 51 implements a learning program to cooperate with the other GPU 52, ROM 53, RAM 54, auxiliary storage device 55, device interface 56, and communication interface 57 to realize the functions of the acquisition unit 41, input unit 42, identification unit 43, processing unit 44, and determination unit 45. The learning program is application software installed in the ROM 53 or auxiliary storage device 55. There are no particular limitations on the method for installing the learning program in the ROM 53 or auxiliary storage device 55. The learning program can be installed in the ROM 53 or auxiliary storage device 55 by recording it on a removable recording medium or by distributing it via communication over a network. The recording medium can be in any form, such as an SD memory card or USB memory, as long as it can store the program and is readable by the device.

In addition, to support the function of the processing unit 44, the learning device 40 stores a processing table 70 (see Figure 4) in RAM 54 or the auxiliary storage device 55.

Figure 4 is a schematic diagram showing the main data structure of the processing table 70. As shown, the processing table 70 is a data table that stores data such as processing codes, processing names, and missing type flags in association with a series of table numbers. The processing code field stores a unique processing code that is set to individually identify multiple types of processing used to process the image of the discount sticker 6, which is the input data for the learning data. The processing name field stores the name of the processing identified by the corresponding processing code. The missing type flag field stores a missing type flag, which is one-bit data that identifies whether the processing identified by the corresponding processing code is a type that does not result in the loss of information necessary for recognition. For example, the missing type flag is set to "0" for processing that does not result in the loss of information necessary for recognition, and "1" for processing that does result in the loss of information necessary for recognition.

In this embodiment, the processing code "1001" is the identification code for the "rotation" processing. The "rotation" processing is a process for rotating the image of the discount sticker 6 at an arbitrary angle. Images processed using the "rotation" processing do not lose any information necessary for recognition. Therefore, the loss type flag is set to "0".

The processing code "1002" is the identification code for the "Occlusal" processing. The "Occlusal" processing is a process that processes the image of the discount sticker 6 into an image in which part of it is obscured by a component of any shape. Images processed using the "Occlusal" processing may lack information necessary for recognition. Therefore, the loss type flag is set to "1".

The processing code "1003" is the identification code for the "gamma conversion" processing. The "gamma conversion" processing is a process that processes the image of the discount sticker 6 into an image with brightness adjusted using an arbitrary γ (gamma) value. Images processed using the "gamma conversion" processing do not lose any information necessary for recognition. Therefore, the loss type flag is set to "0".

The processing code "1004" is the identification code for the "finger place" processing. The "finger place" processing is a process that processes the image of the discount sticker 6 into an image in which part of it is covered by a finger. Images processed using the "finger place" processing may lack information necessary for recognition. Therefore, the loss type flag is set to "1".

The processing code "1005" is the identification code for the "weak lighting effect" processing. The "weak lighting effect" processing is a process that processes the image of the discount sticker 6 into an image illuminated with weak lighting. Images processed using the "weak lighting effect" processing do not lose any information necessary for recognition. Therefore, the loss type flag is set to "0".

The processing code "1006" is the identification code for the "Strong Lighting Effect" processing. The "Strong Lighting Effect" processing processes the image of the discount sticker 6 into an image illuminated with strong lighting light to the extent that whiteout occurs. Images processed using the "Strong Lighting Effect" processing may lack information necessary for recognition. Therefore, the loss type flag is set to "1".

Here, the processing table 70 functions as a setting means for setting, for each processing, whether or not the processing area is of a type related to the area of interest.
Next, the main operations of the learning device 40 will be explained using the flowcharts in Figures 5 and 6, and the example images in Figures 7 and 8. Note that the operation procedure explained below is an example. The procedure can be changed as appropriate as long as the same operational effect is achieved.

Figure 5 is a flowchart showing the main steps of the information processing executed by the CPU 51 of the learning device 40 in accordance with the learning program. For example, when an input signal from the input device 59 commands the start of the learning program, the CPU 51 starts the information processing steps shown in the flowchart of Figure 5. First, in ACT 1, the CPU 51 acquires a learning dataset by functioning as the acquisition unit 41.

After acquiring the training data set, the CPU 51 proceeds to ACT 2. In ACT 2, the CPU 51 selects one piece of training data from the training data set using its function as the input unit 42 and inputs it into the machine learning model 30. Then, in ACT 3, the CPU 51 causes the machine learning model 30 to perform training.

Once learning using the selected learning data is complete, the CPU 51 proceeds to ACT 4. In ACT 4, the CPU 51 identifies the region of interest using the function of the identification unit 43. For example, the CPU 51 binarizes a heat map generated using visualization technology such as Grad-CAM, and acquires the region that becomes the foreground region as the region of interest.

Figure 7 shows an example of an image used to explain the overview of the identification unit 43. In the figure, image 80 shows a discount sticker with discount information "30 yen off." In other words, the correct label for learning data that uses image 80 of this discount sticker as input data is "30 yen off." By focusing on the part of image 80 where "30 yen" is indicated, the machine learning model 30 can obtain the result of "30 yen off," which is indicated by the correct label. Therefore, the identification unit 43 identifies the area surrounding "30 yen" as the area of interest 91 in image 90 showing the entire discount sticker.

Once the region of interest has been identified, the CPU 51 checks in ACT 5 whether unprocessed training data exists in the training dataset. If unprocessed training data exists, the CPU 51 returns to ACT 2. That is, the CPU 51 selects the unprocessed training data and inputs it into the machine learning model 30, causing the machine learning model 30 to perform training. The CPU 51 then identifies the region of interest for that training data.

Once the CPU 51 has identified the regions of interest for all of the learning data set in the learning dataset in this way, the CPU 51 proceeds to ACT 6. In ACT 6, the CPU 51 executes data augmentation processing using the functions of the processing unit 44 and the determination unit 45. Details of the data augmentation processing will be described later, but the data augmentation processing generates one or more pieces of augmented learning data in which the input data has been processed for each piece of learning data set in the learning dataset, and the CPU 51 inputs the augmented learning data to the machine learning model 30 in ACT 7. Then, in ACT 8, the CPU 51 causes the machine learning model 30 to perform learning.

In ACT 9, the CPU 51 determines whether re-learning of the machine learning model 30 is necessary. For example, the CPU 51 determines whether re-learning is necessary by verifying whether the machine learning model 30 has reached the expected recognition performance. Specifically, the CPU 51 determines that re-learning is necessary if the machine learning model 30 has not reached the expected recognition performance. Alternatively, the CPU 51 determines whether re-learning is necessary by verifying whether re-learning has been performed a preset number of times. Specifically, the CPU 51 determines that re-learning is necessary if the number of re-learning attempts has not reached the set number. The CPU 51 may also determine whether re-learning is necessary using other determination methods.

If the CPU 51 determines in ACT 9 that re-learning is necessary, it returns to ACT 6. That is, the CPU 51 executes the data expansion process again. If the CPU 51 determines in ACT 9 that re-learning is not necessary, it ends the information processing of the procedure based on the learning program.

Figure 6 is a flow chart showing one procedure for data augmentation processing. This procedure is a data augmentation procedure in which, for each training data, six types of processing - rotation, occlusion, gamma conversion, finger placement, low lighting effect, and high lighting effect - are sequentially performed on the input data to obtain extended training data. Other procedures, for example, may be a procedure in which extended training data is obtained by randomly selecting and performing zero or more processing operations on each training data. Furthermore, other procedures may be used as long as they achieve the same effects as the above procedure.

When the data extension process begins in ACT 6 of Figure 5, the CPU 51 starts information processing according to the procedure shown in the flowchart of Figure 6. First, in ACT 11, the CPU 51 resets the counter N to "0". Next, in ACT 12, the CPU 51 counts up the counter N by "1". Then, in ACT 13, the CPU 51 checks whether the counter N has exceeded the maximum value of "6", the table number in the processing table 70.

If the counter N does not exceed the maximum value of the table number "6", the CPU 51 proceeds to ACT 14. The CPU 51 executes the processes of ACT 14 to ACT 17 by functioning as the processing unit 44. That is, in ACT 14, the CPU 51 selects a processing operation identified by the processing code set in the table number N of the processing operation table 70. The "N" in the table number N is the value of the counter N. At this time, the counter N is "1". Therefore, the CPU 51 selects the processing operation "rotation".

After selecting a processing operation, the CPU 51 determines the processing level for that processing operation in ACT 15. Currently, the processing operation "rotation" is selected. The processing level for the processing operation "rotation" is the rotation angle of the image. Different rotation angles are set depending on the number of times the data extension process has been performed. Since the first data extension process is currently being performed, the CPU 51 determines the rotation angle corresponding to the first time as the processing level. Incidentally, if it is determined in ACT 9 of Figure 5 that re-learning is necessary and the data extension process is to be performed for the second or subsequent time, the CPU 51 determines the rotation angle corresponding to each time as the processing level.

Once the processing level of the processing has been determined, the CPU 51 proceeds to ACT 16. In ACT 16, the CPU 51 extracts one piece of learning data from the learning dataset. Then, in ACT 17, the CPU 51 performs the processing selected in the processing of ACT 14 on the input data of the extracted learning data at the processing level determined in the processing of ACT 15.

Figure 8 shows an example of an image used to explain the overview of the processing unit 44. In the figure, image 80 shows an image of a discount sticker before processing. When the processing process "rotation" is performed on this image 80 at a determined processing level, for example, image 81 is generated.

Once the processing is complete, the CPU 51 executes the processing in ACT 18 to ACT 24 by functioning as the determination unit 45. That is, in ACT 18, the CPU 51 checks whether the processing is of a type that causes a loss of information necessary for recognition, i.e., whether it is a loss-type processing. If the loss-type flag corresponding to table number N is "0", the processing is not a loss-type processing. If the loss-type flag corresponding to table number N is "1", the processing is a loss-type processing.

The processing "rotation" is not of the missing type. The CPU 51 proceeds to ACT 19. In ACT 19, the CPU 51 sets the correct label of the training data in which the processed image 81 is input data to the value of the correct label of the training data in which the unprocessed image 80 is input data. For example, in the case of the training data in which the input data is image 80, the correct label is "30 yen off", so the correct label of the training data in which the input data is image 81 is "30 yen off".

After completing the processing in ACT 19, the CPU 51 proceeds to ACT 23. In ACT 23, the CPU 51 adds the training data expanded by the processing process to the training dataset. That is, the CPU 51 adds expanded training data, which uses image 81 as input data and has the correct label "30 yen discount," to the training dataset.

In ACT 24, the CPU 51 checks whether or not it has finished extracting unprocessed learning data from the learning dataset. Unprocessed learning data is learning data that constitutes the learning dataset acquired in the processing of ACT 1 and that has not been extracted in the processing of ACT 16. Hereinafter, the learning dataset acquired in the processing of ACT 1 will be referred to as raw learning data to distinguish it from the extended learning data added in the processing of ACT 23.

If unprocessed learning data exists among the raw learning data, the CPU 51 returns to ACT 16. The CPU 51 extracts the unprocessed learning data from the learning dataset. The CPU 51 then performs the "rotation" processing on the input data of that learning data at the determined rotation angle, and adds expanded learning data that combines the processed input data with the correct label to the learning dataset.

The CPU 51 then performs the "rotation" processing process on each piece of unprocessed training data input data at the determined rotation angle, and repeats the process of adding expanded training data that combines the processed input data with the correct label to the training dataset.

Once the CPU 51 has completed the processing of ACT 16 and subsequent steps for all raw learning data, it returns to ACT 12. The CPU 51 increments the counter N by 1. As a result, the counter N becomes 2, and the CPU 51 proceeds to ACT 14. That is, the CPU 51 selects the "shielding" processing. The CPU 51 also determines the shape of the shielding member and the position of the image to be shied as the processing level for the "shielding" processing. The processing level for the "shielding" processing is also set according to the number of times the data expansion processing has been performed. Since the current data expansion processing is the first time, the CPU 51 determines the shape of the shielding member and the position of the image to be shied that correspond to the first time as the processing level. Incidentally, if it is determined in ACT 9 of Figure 5 that re-learning is necessary and the data expansion processing is to be performed for the second or subsequent time, the CPU 51 determines the shape of the shielding member and the position of the image to be shied that correspond to each time as the processing level.

Once the "masking" processing is selected and the processing level is determined, the CPU 51 sequentially extracts the raw learning data set from the learning dataset. The CPU 51 then performs the "masking" processing on the input data of the extracted learning data at the determined processing level.

In Figure 8, when the "masking" processing is performed on the unprocessed discount sticker image 80 at the determined processing level, image 82, for example, is generated.

When the processing is completed, the CPU 51 checks in ACT 18 whether the processing is a loss-type processing. The processing "obscuration" is a loss-type processing. Therefore, the CPU 51 proceeds from ACT 18 to ACT 20. In ACT 20, the CPU 51 identifies the processing area. In the case of image 82 in Figure 8, the processing area is area 821 obscured by a member.

After identifying the processing area 821, the CPU 51 proceeds to ACT 21. In ACT 21, the CPU 51 checks whether the processing area 821 overlaps at least partially with the attention area 91 for the image 80. In the example image of Figure 8, a portion of the processing area 821 overlaps with the attention area 91. Therefore, the CPU 51 proceeds from ACT 21 to ACT 22. In ACT 22, the CPU 51 determines that the correct label of the learning data in which the processed image 82 is used as input data cannot be recognized (rejects).

On the other hand, depending on the degree of processing of the "obscuration" processing, it is possible that the processing area 821 does not overlap with the area of interest 91 for the image 80. In that case, the CPU 51 proceeds from ACT 21 to ACT 19. That is, the CPU 51 sets the correct label of the learning data in which the processed image is used as input data to "30 yen off," which is the correct label of the learning data in which the unprocessed image 80 is used as input data.

After completing the processing in ACT 19 or ACT 22, the CPU 51 proceeds to ACT 23. In ACT 23, the CPU 51 adds the training data expanded by the processing process to the training dataset. That is, the CPU 51 adds the expanded training data in which the image 82 is used as input data and the correct label is "unrecognizable (reject)" to the training dataset. Alternatively, the CPU 51 adds the expanded training data in which the processed image is used as input data and the correct label is "30 yen discount" to the training dataset.

Then, in ACT 24, the CPU 51 checks whether or not the extraction of unprocessed training data from the training dataset has been completed. If unprocessed training data exists, the CPU 51 returns to ACT 16. The CPU 51 extracts unprocessed training data from the training dataset. The CPU 51 then performs the "masking" processing on the input data of the training data at the determined processing level, and adds extended training data that combines the processed input data with the correct label to the training dataset. That is, if the processing area 821 does not overlap the attention area 91, the CPU 51 uses the processed image as input data and adds extended training data that combines the correct label "30 yen off". If the processing area 821 overlaps at least a portion of the attention area 91, the CPU 51 uses the processed image as input data and adds extended training data that combines the correct label "unrecognizable".

Once the processing from ACT 16 onwards has been performed on all raw learning data in this way, the CPU 51 returns to ACT 12. The CPU 51 counts up the counter N by "1". As a result, the counter N becomes "3", and the CPU 51 proceeds to ACT 14. That is, the CPU 51 selects the processing "gamma conversion". The CPU 51 also determines a gamma value as the processing level for the processing "gamma conversion". The processing level for the processing "gamma conversion" is also set according to the number of times the data extension processing has been performed. Since the current time is the first data extension processing, the CPU 51 determines the gamma value corresponding to the first time as the processing level. Incidentally, if it is determined in ACT 9 of Figure 5 that re-learning is necessary and the data extension processing is to be performed for the second or subsequent times, the CPU 51 determines the gamma value corresponding to each time as the processing level.

Once the "gamma conversion" processing is selected and the degree of processing is determined, the CPU 51 sequentially extracts the raw learning data set set in the learning dataset. The CPU 51 then performs the "gamma conversion" processing on the input data of the extracted learning data at the determined degree of processing.

In Figure 8, when the "gamma conversion" processing is performed on the unprocessed discount sticker image 80 at the determined processing level, image 83, for example, is generated.

When the processing is complete, the CPU 51 checks in ACT 18 whether the processing is a loss-type processing. The processing "gamma conversion" is not a loss-type processing. Therefore, the CPU 51 performs the processing, correct label determination process, and extended learning data addition process for all raw learning data, just as in the case of the processing "rotation." At this time, the correct label for all extended learning data is "30 yen discount."

Next, since counter N becomes "4", CPU 51 selects the "finger placement" processing. CPU 51 also determines the position of the finger to place on the image as the processing level of the "finger placement" processing. The processing level of the "finger placement" processing is also set according to the number of times the data extension processing has been performed. Since the current data extension processing is the first time, CPU 51 determines the finger position corresponding to the first time as the processing level. Incidentally, if it is determined in ACT 9 of Figure 5 that re-learning is necessary and the data extension processing is performed for the second or subsequent time, CPU 51 determines the finger position corresponding to each time as the processing level.

Once the "finger placement" processing is selected and the processing level is determined, the CPU 51 sequentially extracts the raw learning data set from the learning dataset. The CPU 51 then performs the "finger placement" processing on the input data of the extracted learning data at the determined processing level.

In Figure 8, when the "finger touch" processing is performed on the unprocessed discount sticker image 80 at the determined processing level, image 84, for example, is generated.

When the processing is completed, the CPU 51 checks in ACT 18 whether the processing is a missing-type processing. The processing "finger contact" is a missing-type processing. Therefore, the CPU 51 identifies the processing area in ACT 20. In the case of image 84 in Figure 8, the processing area is area 841 indicating the finger.

After identifying the processing area 841, the CPU 51 checks in ACT 21 whether the processing area 841 overlaps at least partially with the attention area 91 for the image 80. In the example image of Figure 8, a portion of the processing area 841 overlaps with the attention area 91. Therefore, in ACT 22, the CPU 51 determines that the correct answer label of the learning data in which the processed image 84 is used as input data cannot be recognized (rejects).

On the other hand, depending on the degree of processing of the "finger touch" processing, it is possible that the processing area 841 does not overlap with the attention area 91 for the image 80. In that case, in ACT 19, the CPU 51 sets the correct label of the learning data in which the processed image is used as input data to "30 yen off," which is the correct label of the learning data in which the unprocessed image 80 is used as input data.

After completing the processing in ACT 19 or ACT 22, the CPU 51 adds the training data expanded by the processing process in ACT 23 to the training dataset. That is, the CPU 51 adds the expanded training data in which the image 84 is used as input data and the correct label is "unrecognizable (reject)" to the training dataset. Alternatively, the CPU 51 adds the expanded training data in which the processed image is used as input data and the correct label is "30 yen discount" to the training dataset.

Then, in ACT 24, the CPU 51 checks whether or not the extraction of unprocessed learning data from the learning dataset has been completed. If unprocessed learning data exists, the CPU 51 returns to ACT 16. The CPU 51 extracts unprocessed learning data from the learning dataset. The CPU 51 then performs the "finger placement" processing on the input data of the learning data at the determined processing level, and adds extended learning data that combines the processed input data with the correct label to the learning dataset. That is, if the processing area 841 does not overlap the attention area 91, the CPU 51 uses the processed image as input data and adds extended learning data that combines the correct label "30 yen off". If the processing area 841 overlaps at least a portion of the attention area 91, the CPU 51 uses the processed image as input data and adds extended learning data that combines the correct label "unrecognizable".

In this way, when the processing from ACT 16 onwards has been executed for all raw learning data, the CPU 51 returns to ACT 12. The CPU 51 counts up the counter N by "1". As a result, the counter N becomes "5", so the CPU 51 proceeds to ACT 14. That is, the CPU 51 selects the processing "weak lighting effect". In addition, the CPU 51 determines the intensity and position of the lighting to be applied to the image as the processing level of the processing "weak lighting effect".
The processing level of the "weak lighting effect" processing is also set according to the number of times the data extension process has been performed. Since the first data extension process is currently being performed, the CPU 51 determines the lighting intensity and position corresponding to the first time as the processing level. Incidentally, if it is determined in ACT 9 of FIG. 5 that re-learning is necessary and the data extension process is to be performed for the second or subsequent time, the CPU 51 determines the lighting intensity and position corresponding to each time as the processing level.

Once the "weak lighting effect" processing is selected and the processing level is determined, the CPU 51 sequentially extracts the raw learning data set from the learning dataset. The CPU 51 then applies the "weak lighting effect" processing to the input data of the extracted learning data at the determined processing level.

In Figure 8, when the "weak lighting effect" processing is performed at a determined processing level on the unprocessed discount sticker image 80, image 85, for example, is generated.

When the processing is complete, the CPU 51 checks in ACT 18 whether the processing is a missing type processing. The processing "weak lighting effect" is not a missing type processing. Therefore, the CPU 51 performs the processing, correct label determination process, and extended learning data addition process for all raw learning data, just as in the case of the processing "rotation." At this time, the correct label for all extended learning data is "30 yen discount."

Next, since counter N becomes "6", CPU 51 selects the "strong lighting effect" processing. CPU 51 also determines the intensity and position of the lighting to be applied to the image as the processing level for the "strong lighting effect" processing. The processing level for the "strong lighting effect" processing is also set according to the number of times the data extension processing has been performed. Since the first data extension processing is currently being performed, CPU 51 determines the lighting intensity and position corresponding to the first time as the processing level. Incidentally, if it is determined in ACT 9 of Figure 5 that re-learning is necessary and the data extension processing is to be performed for the second or subsequent time, CPU 51 determines the lighting intensity and position corresponding to each time as the processing level.

Once the "strong lighting effect" processing is selected and the processing level is determined, the CPU 51 sequentially extracts the raw learning data set from the learning dataset. The CPU 51 then applies the "strong lighting effect" processing to the input data of the extracted learning data at the determined processing level.

In Figure 8, when the "strong lighting effect" processing is performed at a determined processing level on the unprocessed discount sticker image 80, image 86, for example, is generated.

When the processing is complete, the CPU 51 checks in ACT 18 whether the processing is a loss-type processing. The processing "Strong Lighting Effect" is a loss-type processing. Therefore, the CPU 51 identifies the processing area in ACT 20. In the case of image 86 in Figure 8, the processing area is area 861 where blown-out highlights occur.

After identifying the processing area 861, the CPU 51 checks in ACT 21 whether the processing area 861 overlaps at least partially with the attention area 91 for the image 80. In the example image of Figure 8, a portion of the processing area 861 overlaps with the attention area 91. Therefore, in ACT 22, the CPU 51 determines that the correct answer label of the learning data in which the processed image 86 is used as input data cannot be recognized (rejects).

On the other hand, depending on the degree of processing of the "strong lighting effect" processing, it is possible that the processing area 861 does not overlap with the area of interest 91 for the image 80. In that case, in ACT 19, the CPU 51 sets the correct label of the learning data in which the processed image is used as input data to "30 yen off," which is the correct label of the learning data in which the unprocessed image 80 is used as input data.

After completing the processing in ACT 19 or ACT 22, the CPU 51 adds the training data expanded by the processing process in ACT 23 to the training dataset. That is, the CPU 51 adds expanded training data in which image 86 is used as input data and the correct label is "unrecognizable (reject)" to the training dataset. Alternatively, the CPU 51 adds expanded training data in which the processed image is used as input data and the correct label is "30 yen discount" to the training dataset.

Then, in ACT 24, the CPU 51 checks whether or not the extraction of unprocessed learning data from the learning dataset has been completed. If unprocessed learning data exists, the CPU 51 returns to ACT 16. The CPU 51 extracts unprocessed learning data from the learning dataset. The CPU 51 then performs the "strong lighting effect" processing on the input data of the learning data at the determined processing level, and adds extended learning data that combines the processed input data with the correct label to the learning dataset. That is, if the processing area 861 does not overlap the attention area 91, the CPU 51 uses the processed image as input data and adds extended learning data that combines the correct label "30 yen off". If the processing area 861 overlaps at least a portion of the attention area 91, the CPU 51 uses the processed image as input data and adds extended learning data that combines the correct label "unrecognizable".

Once the processing from ACT 16 onwards has been performed for all raw learning data, the CPU 51 returns to ACT 12. The CPU 51 counts up the counter N by "1". As a result, the counter N becomes "7", and the CPU 51 ends the data extension processing.

As described above, the learning device 40 in this embodiment processes the input data of each learning data set, which is a collection of multiple learning data sets, to obtain extended learning data and add it to the learning data set. In this way, the learning device 40 can train the machine learning model 30 with a large amount of learning data that combines a large number of input data sets that cover a variety of expected patterns and correct labels for those input data sets.

Furthermore, the learning device 40 identifies a region of interest that was focused on in the input data of the training data when the machine learning model 30, which received input training data before processing, obtained a result indicated by the correct label of the training data. If at least a portion of the processed region when the input data of the training data was processed overlaps with the region of interest, the learning device 40 changes the correct label of the extended training data to a correct label such as "unrecognizable (reject)" instead of the correct label of the raw training data. Depending on the processing performed on the input data, even the region of interest that the machine learning model 30 focused on when obtaining the recognition result may have been processed, making it impossible to actually obtain a correct label from the input data. In such cases, in this embodiment, the recognition result is "unrecognizable (reject)," so no erroneous recognition occurs. Therefore, learning with the extended training data can prevent a decrease in the accuracy of the machine learning model 30.

In this embodiment, whether or not the processing area is related to the area of interest is set in advance depending on the type of processing. For processing operations in which the processing area is not related to the area of interest, the correct label of the raw learning data is unconditionally used as the correct label of the extended learning data. Therefore, compared to executing a process to determine whether the processing area overlaps with at least a portion of the area of interest for all processing operations, this can effectively reduce the processing load on the CPU 51.

The above describes one embodiment of a learning device and its program, but this embodiment is presented as an example and is not intended to limit the scope of the invention.

For example, in the above embodiment, a learning device 40 was shown that trains a machine learning model 30 used for image recognition of discount labels 6, but the images to be recognized are not limited to images of discount labels 6. For example, the product registration system 100 may also be a learning device for a machine learning model 30 for image recognition of barcodes or image recognition of products. It can also be similarly applied to a learning device that trains a machine learning model used for voice recognition, or a learning device that trains a machine learning model used for character recognition.

These novel embodiments may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope of the invention, as well as within the scope of the inventions and their equivalents as set forth in the claims.

1...Scanner, 2...Image processing device, 3...POS terminal, 4...Product, 5...Barcode, 6...Discount sticker, 11...Image storage unit, 12...Barcode recognition unit, 13...Discount sticker recognition unit, 14...Recognition result output unit, 20...Inference device, 21...Image acquisition unit, 22...Image extraction unit, 23...Recognition processing unit, 30...Machine learning model, 40...Learning device, 41...Acquisition unit, 42...Input unit, 43...Identification unit, 44...Processing unit, 45...Determination unit, 51...CPU, 52...GPU, 53...ROM, 54...RAM, 55...Auxiliary storage device, 56...Device interface, 57...Communication interface, 58...System transmission path, 59...Input device, 60...Display device, 70...Processing processing table, 80, 81, 82, 83, 84, 85, 86, 90...Image, 821, 841, 861...Processing area, 91...Area of interest.

Claims (6)

A learning device that trains a machine learning model using training data that combines input data and a correct answer label that indicates an expected result for the input data,
an input means for inputting the learning data into the machine learning model;
an identification means for identifying an area of interest that is focused on the input data of the learning data when the machine learning model, to which the learning data is input, obtains a result indicated by the correct label of the learning data;
processing means for processing the input data in the learning data;
a determining means for determining a correct label of the learning data obtained by processing the input data based on the processing area of the input data processed by the processing means and the attention area identified by the identifying means;
A learning device comprising: The learning device of claim 1, wherein the determination means, when the processing area of the input data overlaps at least a portion of the area of interest, changes the correct answer label of the training data obtained by processing the input data from the correct answer label combined with the input data before processing. The learning device of claim 1, wherein the determination means changes the correct label of the learning data resulting from processing the input data from the correct label combined with the input data before processing when the area where the processing region of the input data overlaps with the region of interest is equal to or greater than a threshold. The learning device of claim 1, wherein the determination means changes the correct label of the learning data resulting from processing the input data from the correct label combined with the input data before processing when the distance from the center of gravity position of the region of interest to the processing region of the input data is equal to or less than a threshold value. the processing means processes the input data using a plurality of types of processing;
a setting means for setting, for each of the plurality of types of processing, whether or not the processing area is a type related to the area of interest;
Further comprising:
5. The learning device according to claim 1, wherein the determining means determines a correct label of learning data in the input data that has been subjected to a type of processing in which the processing region relates to the region of interest. A computer of a learning device that trains a machine learning model using training data that combines input data and correct labels that indicate expected results for the input data,
an input means for inputting the training data into the machine learning model;
an identification means for identifying an area of interest that is focused on the input data of the learning data when the machine learning model to which the learning data is input obtains a result indicated by the correct label of the learning data;
processing means for processing the input data in the training data; and
a determination means for determining a correct label of the learning data obtained by processing the input data based on the processing area of the input data processed by the processing means and the attention area identified by the identification means;
A program to function as a