JP7795320B2 - Optical information reader - Google Patents

Description

The present invention relates to an optical information reading device that optically reads information.

In recent years, there has been growing emphasis on traceability, which allows goods to be tracked along their distribution route, from the manufacturing stage to the consumption or disposal stage, and code readers designed for this purpose are becoming increasingly popular. Furthermore, code readers are also used in a variety of fields other than traceability.

Generally, code readers are configured to use a camera to capture a barcode, two-dimensional code, or other code attached to a workpiece, then extract and binarize the code contained in the resulting image using image processing, and decode it to read the information. Because they are devices that read information optically, they are also called optical information readers.

One known example of this type of optical information reading device is one equipped with a machine learning device that learns a model structure that shows the relationship between a code image acquired by a robot's visual sensor and an ideal code image, as disclosed in Patent Document 1. Patent Document 1 describes how the results of learning by the machine learning device are applied to code images acquired by the visual sensor during operation, thereby restoring the image to one suitable for reading.

Patent No. 6683666

By the way, the amount of calculation required for image processing by a neural network in a machine learning device generally depends on the size of the image input to the neural network, so the larger the size of the input image, the greater the amount of calculation required by the neural network. This increased calculation load puts a heavier load on the processor, ultimately resulting in a decrease in processing speed.

On the other hand, there are virtually no cases where a code is present throughout the entire image captured by an optical information reading device. During normal operation, the code is present only in a small area of the image, and if only that area can be restored using a neural network, it is thought that reading accuracy can be improved.

The present invention was made in light of these issues, and its purpose is to reduce the size of the image input to the neural network while preventing the loss of information necessary for reading, thereby achieving both faster processing speeds and higher reading accuracy.

To achieve the above objective, one aspect of the present disclosure can be based on an optical information reading device that reads a code attached to a workpiece. The optical information reading device includes a camera that photographs the code and generates a read image; a memory unit that stores the structure and parameters of a neural network that has been generated in advance by machine learning multiple defective images that have portions unsuitable for reading the code and multiple ideal images that correspond to the multiple defective images; and a processor that extracts code candidate areas that are likely to contain a code from the read image generated by the camera, inputs partial images corresponding to the extracted code candidate areas into a neural network configured with the structure and parameters stored in the memory unit, attempts to repair the partial image in accordance with the structure and parameters stored in the memory, and performs a decoding process on the repaired partial image.

With this configuration, when the optical information reading device is in operation, if a code candidate area is extracted from the scanned image generated by the camera, the partial image corresponding to that code candidate area is input into the neural network, and an attempt is made to repair the partial image. As mentioned above, since it is rare for a code to be present in the entire scanned image, the size of the partial image corresponding to the code candidate area is smaller than the size of the scanned image. This reduces the size of the image input to the neural network, reducing the computational load on the processor and ultimately achieving faster processing speeds. Furthermore, because the partial image corresponds to an area where a code is likely to be present, it is possible to create an image that includes the information necessary for reading, i.e., the entire code. Because an image including this entire code is input into the neural network, a decrease in reading accuracy is suppressed.

In another aspect of the present disclosure, the processor searches for a code in the scanned image generated by the camera based on information for identifying the code, and extracts an area containing the searched code as the code candidate area.

With this configuration, the optical information reader automatically extracts code candidate areas without the user having to perform any operations to extract the code candidate areas, thereby reducing the burden on the user.

In another aspect of the present disclosure, the optical information reading device includes a light irradiation unit that forms a mark by irradiating visible light of a color different from ambient light within the field of view of the camera. The processor can identify a portion of the scanned image generated by the camera that corresponds to the mark formed by the light irradiation unit and extract the identified portion as the code candidate region.

With this configuration, a marker is formed using visible light within the camera's field of view. For example, in an optical information reader equipped with a handheld housing with a handle for the user to hold during operation, the user can align the marker with the code, allowing the camera to capture an area that reliably contains the code. Therefore, in the scanned image generated by the camera, there is a high probability that the code is present in the portion corresponding to the marker. By extracting the portion corresponding to this marker as a code candidate area, the likelihood that the extracted area contains a code is further increased, thereby improving the success rate of the decoding process.

In another aspect of the present disclosure, the processor identifies the center of the scanned image generated by the camera and extracts that center as the code candidate area. For example, in an optical information reader equipped with a handheld housing with a handle for a user to hold during operation, it is thought that the user will often move the optical information reader so that the code is positioned in the center of the camera's field of view. Therefore, by extracting the center of the scanned image as the code candidate area, the likelihood that the extracted area will contain a code is further increased, thereby improving the success rate of the decoding process.

In another aspect of the present disclosure, the processor is configured to accept a user's designation of a specific portion of the scanned image generated by the camera, and extract the designated portion as the code candidate area.

In other words, the scanned image may contain things other than codes, making it difficult to distinguish them from codes. However, if the user can specify a part of the image as in this configuration, the user can specify the code in the scanned image and reliably identify the code.

In another aspect of the present disclosure, the optical information reading device further includes a tuning execution unit that sets various conditions during configuration of the optical information reading device so that the conditions are suitable for the decoding process. The tuning execution unit can acquire the size of the code included in the scanned image generated by the camera. The processor can determine the input size of the partial image to the neural network based on the code size acquired by the tuning execution unit.

This allows the input size to be appropriate for the code size, reducing the chance of the input size being unnecessarily large and taking a long time to process, or the input size being too small and resulting in parts of the code being missing, causing the decoding process to fail.

In another aspect of the present disclosure, the tuning execution unit, when setting up the optical information reading device, acquires the size of the code to be decoded based on an image generated by capturing an image of the code with the camera. With this configuration, the input size to the neural network can be adjusted to match the input size of the code to be decoded.

In another aspect of the present disclosure, the processor increases the input size of the partial image to the neural network by a predetermined amount greater than the size of the code obtained by the tuning execution unit.

In other words, while inputting a partial image into a neural network can reduce the computational load, if the code is rotated or the code position cannot be detected precisely, there is a risk that part of the code in the partial image will be missing, resulting in a failed decoding process. In this configuration, the input size to the neural network is not the same as the size of the code obtained by the tuning execution unit, but is larger by a predetermined amount, preventing part of the code from being missing from the partial image. Furthermore, by making the input size larger by a predetermined amount than the code size, an upper limit can be placed on the input size to the neural network, thereby preventing the computational load from becoming too heavy. The "predetermined amount" means, for example, that the horizontal (or vertical) length of the partial image input to the neural network is at least 1.5 times, 2.0 times, or 3.0 times the horizontal (or vertical) length of the code obtained by the tuning execution unit.

In another aspect of the present disclosure, the processor can determine the input size of the partial image to the neural network based on the number of pixels in one module that makes up the code obtained by the tuning execution unit and the number of modules arranged vertically or horizontally in the code.

In another aspect of the present disclosure, the system further includes an image restoration unit that inputs a scanned image to a neural network and performs an inference process to generate a restored image by restoring the scanned image; a first decoding processing unit that performs a first decoding process on the scanned image generated by the camera; and a second decoding processing unit that performs a second decoding process on the restored image generated by the image restoration unit. The second decoding processing unit extracts a code candidate area from the scanned image and sends a trigger signal to the image restoration unit to execute the inference process. Upon receiving the trigger signal from the second decoding processing unit, the image restoration unit inputs a partial image corresponding to the code candidate area to the neural network and executes an inference process to generate a restored image by restoring the partial image. The second decoding processing unit can determine grid positions indicating the positions of each cell of the code based on the restored image and decode the restored image based on the determined grid positions. In other words, the first decoding processing unit decodes the normal scanned image that has not been subjected to inference processing, the image restoration unit generates a restored image through inference processing, and the second decoding processing unit decodes the restored image restored through inference processing. This allows for the construction of processing circuits suited to each process, thereby increasing processing speed. In addition, the image restoration unit only needs to restore the partial image corresponding to the code candidate area, making it possible to further increase processing speed.

In another aspect of the present disclosure, the inference processing by the image restoration unit and the first decoding processing by the first decoding processing unit are performed in parallel, thereby increasing the processing speed. Furthermore, at least a portion of the period during which the second decoding processing unit is performing the second decoding processing includes a pause period during which the inference processing by the image restoration unit is not performed, thereby reducing the load on the image restoration unit and suppressing heat generation. In this case, the second decoding processing unit may be configured with multiple cores.

In another aspect of the present disclosure, the first decoding processing unit can execute the first decoding process in a shorter time than the time required for the second decoding process. Furthermore, the setting unit can switch whether or not to execute the second decoding process, i.e., whether or not to execute decoding of the restored image. The second decoding process, which executes decoding of the restored image, takes longer than the first decoding process, which executes decoding of a normal scanned image. Given this premise, if the second decoding process is set to be executed, a decoding timeout period (decoding time limit) longer than the time required to execute the second decoding process can be set. On the other hand, if the second decoding process is set not to be executed, a decoding timeout period shorter than the time required to execute the second decoding process can be set. Therefore, if the decoding of the restored image exceeds the decoding timeout period, the process can quickly move on to decoding of the next captured image.

A decoding processing unit according to another aspect of the present disclosure can perform a second decoding process on a restored image in parallel with a first decoding process on a scanned image generated by a camera, thereby increasing the processing speed even when scanned images and restored images are mixed. When the setting unit is configured not to perform the second decoding process, this decoding processing unit can allocate processing resources used for the second decoding process to the first decoding process. Processing resources include, for example, memory usage areas and cores that make up a multi-core CPU. When a memory area or core is pre-set as a processing resource used for the second decoding process, when the setting is made not to perform the second decoding process, that memory area or core can be used for the first decoding process, thereby further increasing the speed of the first decoding process.

A decoding processing unit according to another aspect of the present disclosure extracts code candidate areas from a scanned image that are likely to contain a code. The image restoration unit inputs a partial image corresponding to the code candidate area into a neural network and generates a restored image by restoring the partial image. The decoding processing unit determines grid positions based on the restored image generated by the image restoration unit, and decodes the restored image based on the determined grid positions. In other words, since generating a restored image often takes time, the time required to generate the restored image can be shortened by extracting code candidate areas based on, for example, feature amounts that indicate code-likeness before generating the restored image. Furthermore, since grid positions can be determined with high accuracy based on the restored image, reading performance is improved.

In another aspect of the present disclosure, a tuning process is performed to determine optimal imaging conditions and decoding conditions, and to set the size of the code to be read. The decoding processing unit can determine the size of an extracted image containing a code candidate area from the scanned image based on the code size set in the tuning process. By enlarging or reducing the extracted image to a predetermined size and inputting it into the neural network, the PPC of the code image input to the neural network can be kept within a predetermined range, allowing for a stable image restoration effect by the neural network. Furthermore, although the code size to be read can vary, the size of the code ultimately input to the neural network can be fixed, thereby maintaining a constant balance between high processing speed and ease of decoding.

A camera according to another aspect of the present disclosure receives light that passes through a first polarizer and is reflected from the code via a second polarizer, generating a read image with lower contrast than when the light is not transmitted through the first and second polarizers. By using a polarizer, the specular reflection component of the workpiece is removed, resulting in a read image with reduced influence from the specular reflection component. While polarizers are suitable for images with a high specular reflection component, such as when capturing an image of a metal workpiece, reduced light intensity can cause the read image to become dark and reduce contrast. In such cases, reading performance can be improved by converting the image into a high-contrast restored image using neural network inference processing.

In another aspect of the present disclosure, the tuning execution unit can set multiple imaging conditions and code conditions, including first imaging conditions and code conditions and second imaging conditions and code conditions. The decoding processing unit inputs a read image generated under the first imaging conditions and code conditions into a neural network to generate a repaired image, and if decoding of the repaired image fails, inputs a read image generated under second imaging conditions and code conditions different from the first imaging conditions and code conditions into the neural network to generate a repaired image, and then decodes the repaired image.

As explained above, code candidate areas where a code is likely to exist are extracted from the scanned image generated by the camera, the partial image corresponding to the extracted code candidate area is input into a neural network to attempt to repair the partial image, and the decoding process is performed on the repaired partial image, thereby achieving both faster processing speed and higher reading accuracy.

FIG. 10 is a diagram illustrating the stationary optical information reader during operation. FIG. 1 is a perspective view of a stationary optical information reading device. FIG. 1 is a block diagram of an optical information reader. FIG. 1 is a perspective view of a handheld optical information reading device. FIG. 2 is a diagram illustrating each unit configured by a processor. FIG. 1 is a conceptual diagram of a neural network. 1 is a flowchart showing an example of a basic procedure for learning a neural network. 1 shows a pair of a defective image and an ideal image, where (A) is an example of the defective image and (B) is an example of the ideal image. FIG. 1 is a conceptual diagram of a convolutional neural network used for image conversion. 10 is a flowchart showing an example of the procedure of a tuning process performed when setting up an optical information reader. 10 is a flowchart illustrating an example of a decoding process procedure before a reduction ratio and an enlargement ratio are determined. 10 is a flowchart illustrating an example of a decoding process procedure after a reduction ratio and an enlargement ratio are determined. 1A shows an example of a scanned image generated by a camera, and FIG. 1B shows an example of an image after an image processing filter is applied. 10A shows an example of an image after reduction processing, and FIG. 10B shows an example of an image after code region extraction. FIG. 10 is a diagram showing an example of restoration of a scanned image using a neural network. FIG. 10 is a diagram illustrating an example of a decoding process procedure according to another embodiment. 10 is a time chart of the decoding process by the first decoding processing unit and the second decoding processing unit. 10 is a time chart of a decoding process by a decoding processing unit. 10 is a flowchart illustrating an example of a first decoding process and a second decoding process. 10A to 10C are diagrams illustrating example images of each process. 10A and 10B are diagrams illustrating a case where a plurality of neural networks are used to deal with black and white inversion of a code. FIG. 10 is a diagram illustrating a case where one neural network is used to handle black and white inversion of a code. 10 is a graph showing the relationship between contrast and matching level. 10 is a flowchart illustrating an example of a process for calculating a matching level. FIG. 4 is a diagram illustrating an example of a user interface displayed on a display unit. FIG. 10 is a diagram showing an example in which an imaging element with an AI chip is used.

Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Note that the following description of the preferred embodiment is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

(Stationary optical information reader)
FIG. 1 is a schematic diagram illustrating a stationary optical information reading device 1 according to an embodiment of the present invention during operation. In this example, multiple workpieces W are placed on the upper surface of a conveyor belt B and transported in the direction of arrow Y in FIG. 1 . The optical information reading device 1 according to the embodiment is installed above and away from the workpieces W. The optical information reading device 1 is a code reader configured to photograph a code attached to the workpiece W and decode the code contained in the photographed image to read information. In the example shown in FIG. 1 , the optical information reading device 1 is stationary. When this stationary optical information reading device 1 is in operation, it is fixed to a bracket or the like (not shown) to prevent movement. The stationary optical information reading device 1 may also be used while being held by a robot (not shown). Alternatively, the optical information reading device 1 may be configured to read the code on a stationary workpiece W. The stationary optical information reading device 1 is in operation when it is performing the operation of reading the codes of the works W being transported by the transport belt conveyor B in order.

In addition, a code is attached to the outer surface of each workpiece W. The code includes both barcodes and two-dimensional codes. Examples of two-dimensional codes include QR Code (registered trademark), micro QR Code, Data Matrix (Data Code) (registered trademark) , Veri Code, Aztec Code, PDF417, and Maxi Code. Two-dimensional codes are classified into stack type and matrix type, and the present invention can be applied to any two-dimensional code. The code may be attached by printing or engraving directly on the workpiece W, or by printing on a label and then attaching it to the workpiece W, and the means and method are not important.

The optical information reader 1 is wired to the computer 100 and programmable logic controller (PLC) 101 via signal lines 100a and 101a, respectively. However, this is not limited to this; the optical information reader 1, computer 100, and PLC 101 may each have a built-in communication module, and the optical information reader 1 may be wirelessly connected to the computer 100 and PLC 101. The PLC 101 is a control device for sequence control of the transport belt conveyor B and the optical information reader 1, and a general-purpose PLC may be used. The computer 100 may be a general-purpose or dedicated electronic computer, a portable terminal, or the like.

During operation, the optical information reading device 1 receives a read start trigger signal from the PLC 101 via signal line 101a, which signals the start timing of code reading. The optical information reading device 1 then captures and decodes the code based on this read start trigger signal. The decoded results are then transmitted to the PLC 101 via signal line 101a. Thus, during operation, the optical information reading device 1 repeatedly inputs the read start trigger signal and outputs the decoded results via signal line 101a between the optical information reading device 1 and an external control device such as the PLC 101. Note that the input of the read start trigger signal and the output of the decoded results may be performed via signal line 101a between the optical information reading device 1 and the PLC 101, as described above, or via other signal lines (not shown). For example, a sensor for detecting the arrival of work W may be directly connected to the optical information reading device 1, and the read start trigger signal may be input from the sensor to the optical information reading device 1.

As shown in Figure 2, the optical information reading device 1 includes a box-shaped housing 2, a polarizing filter attachment 3, an illumination unit 4, a camera 5, a display unit 6, a power connector 7, and a signal line connector 8. Furthermore, the housing 2 includes an indicator 9, an aimer light emitting unit (light emitting unit) 10, and operation buttons 11 and 12; the indicator 9, aimer light emitting unit 10, and operation buttons 11 and 12 are also components of the optical information reading device 1.

The housing 2 is elongated in a specific direction, but the shape of the housing 2 is not limited to the shape shown in the figure. A polarizing filter attachment 3 is detachably attached to the front outer surface of the housing 2. Inside the housing 2, an illumination unit 4, a camera 5, an aimer light irradiation unit 10, a processor 20, a memory unit 30, a ROM 40, a RAM 41, etc. are housed. The processor 20, memory unit 30, ROM 40, and RAM 41 are also components of the optical information reading device 1.

An illumination unit 4 is provided on the front side of the housing 2. The illumination unit 4 illuminates at least the code on the workpiece W by emitting light toward the front of the optical information reading device 1. As shown in FIG. 3, the illumination unit 4 includes a first illumination unit 4a consisting of multiple light-emitting diodes (LEDs), a second illumination unit 4b consisting of multiple light-emitting diodes, and an illumination drive unit 4c consisting of an LED driver or the like that drives the first illumination unit 4a and the second illumination unit 4b. The first illumination unit 4a and the second illumination unit 4b are driven individually by the illumination drive unit 4c, allowing them to be turned on and off separately. The illumination drive unit 4c is connected to the processor 20, which controls the illumination drive unit 4c. Note that one of the first illumination unit 4a and the second illumination unit 4b may be omitted.

As shown in Figure 2, a camera 5 is provided in the center of the front side of the housing 2. The optical axis direction of the camera 5 is approximately aligned with the direction of light emitted by the illumination unit 4. The camera 5 is the part that photographs the code and generates a scanned image. As shown in Figure 3, the camera 5 is equipped with an image sensor 5a that receives reflected light from the code attached to the workpiece W and illuminated by the illumination unit 4, an optical system 5b having a lens, etc., and an AF module (autofocus module) 5c. Light reflected from the part of the workpiece W where the code is attached is incident on the optical system 5b, and the incident light is emitted toward the image sensor 5a and forms an image on the imaging surface of the image sensor 5a.

The imaging element 5a is an image sensor consisting of light-receiving elements such as a CCD (charge-coupled device) or CMOS (complementary metal oxide semiconductor) that converts the image of the code obtained through the optical system 5b into an electrical signal. The imaging element 5a is connected to the processor 20, and the electrical signal converted by the imaging element 5a is input to the processor 20 as scanned image data. The AF module 5c is a mechanism that adjusts the focus by changing the position and refractive index of the focusing lens that makes up the optical system 5b. The AF module 5c is also connected to and controlled by the processor 20.

As shown in FIG. 2, a display unit 6 is provided on the side of the housing 2. The display unit 6 is, for example, an organic EL display or a liquid crystal display. The display unit 6 is connected to the processor 20 and can display, for example, the code captured by the imaging unit 5, the character string resulting from decoding the code, the read success rate, the matching level, etc. The read success rate is the average read success rate when multiple reading processes are performed. The matching level is the reading margin that indicates the ease of reading a successfully decoded code. This can be determined from the number of error corrections that occur during decoding, and can be expressed, for example, as a numerical value. The fewer error corrections, the higher the matching level (reading margin), and conversely, the more error corrections, the lower the matching level (reading margin).

A power cable (not shown) is connected to the power connector 7 to supply power to the optical information reader 1 from an external source. Furthermore, signal lines 100a, 101a, etc. are connected to the signal line connector 8 to communicate with the computer 100 and the PLC 101. The signal line connector 8 can be configured, for example, as an Ethernet connector, a serial communication connector such as RS232C, a USB connector, etc.

The housing 2 is provided with an indicator 9. The indicator 9 is connected to the processor 20 and can be composed of a light-emitting element such as a light-emitting diode. The operating status of the optical information reading device 1 can be notified to the outside by the lighting state of the indicator 9.

A pair of aimer light emitting units 10 are provided on the front side of the housing 2, sandwiching the camera 5. As shown in FIG. 3, the aimer light emitting unit 10 includes an aimer 10a, such as a light-emitting diode, and an aimer driver 10b that drives the aimer 10a. The aimer 10a emits light (aimer light) toward the front of the optical information reading device 1 to indicate the camera 5's shooting range, the center of the field of view, and the optical axis of the illumination unit 4. Specifically, the aimer 10a emits visible light of a color different from the ambient light (e.g., red or green) toward the camera 5's shooting field of view, forming a mark visible to the naked eye on the surface illuminated by the visible light. The mark may be various shapes, symbols, letters, etc. The user can also install the optical information reading device 1 by referring to the light emitted from the aimer 10a.

As shown in FIG. 2, operation buttons 11 and 12 are provided on the side of the housing 2 to be used when setting up the optical information reader 1. The operation buttons 11 and 12 include, for example, a select button and an enter button. In addition to the operation buttons 11 and 12, a touch panel-type operation means may also be provided. The operation buttons 11 and 12 are connected to the processor 20, which is capable of detecting the operation status of the operation buttons 11 and 12. By operating the operation buttons 11 and 12, it is possible to select one option from multiple options displayed on the display unit 6 and to confirm the selection.

(Handheld optical information reader)
In the above example, the optical information reader 1 is a stationary type, but the present invention can be applied to devices other than the stationary type optical information reader 1. Fig. 4 shows a handheld type optical information reader 1A, and the present invention can also be applied to a handheld type optical information reader 1A such as that shown in this figure.

The housing 2A of the handheld optical information reader 1A is elongated in the vertical direction. Note that the orientation of the optical information reader 1A during use is not limited to the orientation shown in the figure, and it can be used in a variety of orientations. However, for the sake of convenience, the vertical direction of the optical information reader 1A is specified.

A display unit 6A is provided on the upper portion of the housing 2A. The display unit 6A is configured in the same manner as the display unit 6 of the stationary optical information reader 1. The lower portion of the housing 2A is a gripping unit 2B that the user holds during operation. The gripping unit 2B is a part that can be held in one hand by an average adult, and its shape and size can be freely set. By holding this gripping unit 2B, the optical information reader 1A can be carried around. In other words, this optical information reader 1A is a portable terminal device, and can also be called, for example, a handheld terminal.

Like the stationary optical information reader 1, the handheld housing 2A also houses an illumination unit, camera, aimer light emitting unit, processor, memory unit, ROM, RAM, etc. (not shown). The optical axes of the illumination unit, camera, and aimer light emitting unit point diagonally upward from near the top end of the housing 2A. The handheld housing 2A also includes a buzzer (not shown).

Multiple operation buttons 11A and trigger keys 11B are provided on or near the gripper 2B. The operation buttons 11A are similar to the operation buttons 11 on the stationary optical information reading device 1. When the user points the tip (top) of the optical reading device 1A toward the workpiece W and presses the trigger key 11B, an aimer beam is emitted from the tip of the optical reading device 1A. The user adjusts the orientation of the optical reading device 1A while visually observing the aimer beam reflected from the surface of the workpiece W, and aligns the aimer beam with the code to be read. The code is then automatically read and decoded. When reading is complete, a buzzer sounds to notify completion.

One example of the use of the handheld optical information reader 1A is for picking work in a logistics warehouse. For example, when ordered products are to be shipped from the logistics warehouse, the required products are picked from the shelves in the warehouse. This picking work is carried out by a user holding an order slip with a code written on it, who then goes to the product shelf and compares the code on the order slip with the code attached to the product or the product shelf. In this case, the handheld optical information reader 1A alternately reads the code on the order slip and the code attached to the product or the product shelf.

(Processor Configuration)
The following description is common to both the stationary optical information reader 1 and the handheld optical information reader 1A, and can be applied to either device 1 or 1A unless otherwise specified. As shown in FIG. 3, the processor 20 includes a CPU core 21, a DSP core 22, and an AI chip 23. The CPU core 21, the DSP core 22, and the AI chip 23 are so-called System-on-a-chip (SoC) and are mounted on a single board. Note that the DSP core 22 and the AI chip 23 do not have to be SoCs, in which case they are not mounted on the same board, but this case is also within the scope of the present invention.

High-speed RAM 41 is connected to the CPU core 21, DSP core 22, and AI chip 23, and all of the CPU core 21, DSP core 22, and AI chip 23 can access RAM 41. In addition, ROM 40 is connected to the processor 20, and all of the CPU core 21, DSP core 22, and AI chip 23 can access ROM 40.

The CPU core 21 is a general-purpose processor that performs, for example, AF control, lighting control, camera control, and decoding of scanned images. The DSP core 22 performs, for example, various filter processes on scanned images. The AI chip 23 is an integrated circuit dedicated to attempting to restore scanned images using a neural network, and is specialized for performing ultra-high-speed product-sum operations required for neural network processing.

As shown in FIG. 5, the processor 20 configures an AF control unit 20a, an imaging control unit 20b, a filter processing unit 20c, a tuning execution unit 20d, a decode processing unit 20f, an extraction unit 20g, a reduction unit 20h, an enlargement unit 20i, and an image restoration unit 20j. The AF control unit 20a, the imaging control unit 20b, a filter processing unit 20c, a tuning execution unit 20d, a decode processing unit 20f, an extraction unit 20g, a reduction unit 20h, and an enlargement unit 20i are configured by the arithmetic processing of the CPU core 21 or the DSP core 22. On the other hand, the image restoration unit 20j is configured by the AI chip 23.

(Configuration of AF control unit)
The AF control unit 20a is a unit that controls the AF module 5c shown in FIG. 3, and is configured to be able to focus the optical system 5b by conventionally known contrast AF or phase difference AF.

(Configuration of imaging control unit)
The imaging control unit 20b is a unit that adjusts the gain of the camera 5, controls the amount of light from the illumination unit 4, and controls the exposure time (shutter speed) of the image sensor 5a. Here, the gain of the camera 5 refers to the amplification factor (also called magnification) used when amplifying the brightness of the image output from the image sensor 5a through digital image processing. The amount of light from the illumination unit 4 can be changed by separately controlling the first illumination unit 4a and the second illumination unit 4b. The gain, the amount of light from the illumination unit 4, and the exposure time are imaging conditions for the camera 5.

(Configuration of the filter processing unit)
The filter processing unit 20c is a part that applies image processing filters to the read image. The filter processing unit 20c applies a noise removal filter that removes noise contained in the image generated by the camera 5, a contrast correction filter that corrects contrast, an averaging filter, etc. The image processing filters applied by the filter processing unit 20c are not limited to a noise removal filter, a contrast correction filter, and an averaging filter, and may include other image processing filters.

The filter processing unit 20c is configured to apply an image processing filter to the scanned image before image restoration, which will be described later. The filter processing unit 20c is also configured to apply an image processing filter to the scanned image before enlargement or reduction, which will be described later. The filter processing unit 20c may also be configured to apply an image processing filter to the scanned image after image restoration, or to apply an image processing filter to the scanned image after enlargement or reduction.

(Configuration of tuning execution unit)
The tuning execution unit 20d shown in FIG. 5 repeatedly performs the image capture and decoding processes by changing the imaging conditions of the camera 5 and the decoding conditions of the decoding process. Based on a matching level indicating the ease of code reading calculated under each imaging and decoding condition, the tuning execution unit 20d determines optimal imaging and decoding conditions and sets the size of the code to be read. Specifically, when setting up the optical information reading device 1, 1A, the tuning execution unit 20d sets various conditions (tuning parameters) to achieve optimal decoding conditions by changing imaging conditions such as the gain of the camera 5, the light intensity and exposure time of the illumination unit 4, and the image processing conditions of the filter processing unit 20c. The image processing conditions of the filter processing unit 20c include the coefficients (strength of the image processing filter), switching between image processing filters if multiple image processing filters are used, and combining different types of image processing filters. Appropriate imaging and image processing conditions vary depending on factors such as the effect of external light on the workpiece W during transport and the color and material of the surface on which the code is attached. Therefore, the tuning execution unit 20d searches for more appropriate imaging conditions and image processing conditions, and sets the processing to be performed by the AF control unit 20a, the imaging control unit 20b, and the filter processing unit 20c.

The tuning execution unit 20d is configured to be able to acquire the size of the code contained in the scanned image generated by the camera 5. When acquiring the code size, the tuning execution unit 20d first searches for the code based on features that indicate its likelihood of being a code. Next, the tuning execution unit 20d acquires the PPC, code type, code size, etc. of the searched code. The PPC, code type, code size, etc. are included in the code parameters or code conditions, and therefore the tuning execution unit 20d is configured to be able to acquire the code parameters or code conditions of the searched code.

A two-dimensional code is composed of multiple white and black modules arranged randomly. For example, in QR code model 2, the number of modules ranges from 21 x 21 to 177 x 177. In the case of a stationary optical information reader 1, the number of modules and PPC are limited by reading the code to be read in advance, and the code size (pixel size) is limited to the number of modules x PPC.

When focusing on one of the modules that make up a code, the PPC is a code parameter that indicates how many pixels (picture elements) that module is made up of. After identifying a module, the tuning execution unit 20d can obtain the PPC by counting the number of pixels that make up that module.

The code type refers to the type of code, such as a QR code, data matrix code, or VeriCode. Each code has its own characteristics, so the tuning execution unit 20d can determine the code type, for example, based on the presence or absence of a finder pattern.

The code size can also be calculated from the number of modules arranged vertically or horizontally in the code and the PPC. The tuning execution unit 20d calculates the number of modules arranged vertically or horizontally in the code, calculates the PPC, and obtains the code size by multiplying the number of modules arranged vertically or horizontally by the PPC.

(Configuration of the Decode Processing Unit)
The decoding processing unit 20f decodes the black and white binary data. A table showing the correspondence between coded data can be used for decoding. Furthermore, the decoding processing unit 20f checks whether the decoded result is correct according to a predetermined check method. If an error is found in the data, an error correction function is used to calculate the correct data. The error correction function differs depending on the type of code.

In this embodiment, the decoding processing unit 20f decodes the code contained in the scanned image after image restoration, which will be described later, but it can also decode the code contained in the scanned image before image restoration. The decoding processing unit 20f is configured to write the decoded result obtained by decoding the code to the decoding result storage unit 30b of the storage unit 30 shown in Figure 3.

(Configuration of extraction unit)
The extraction unit 20g extracts code candidate areas from the scanned image generated by the camera 5 that are likely to contain a code. The code candidate areas can be extracted based on features that indicate the likelihood of a code. In this case, the features that indicate the likelihood of a code serve as information for identifying a code. For example, the extraction unit 20g can acquire a scanned image and search for a code in the acquired scanned image based on the features that indicate the likelihood of a code. Specifically, the extraction unit 20g searches the acquired scanned image for a portion that has a predetermined or higher level of features that indicate the likelihood of a code. If a portion with a feature that indicates the likelihood of a code is found, the extraction unit 20g extracts the portion as a code candidate area. The code candidate area may include areas other than a code, but it must at least include a portion that is more likely to be a code than a predetermined level. Note that the code candidate area is merely an area that is likely to contain a code, and therefore may not actually contain a code.

The extraction unit 20g can also extract a code candidate area using position information of the aimer light emitted from the aimer light emitting unit 10 as information for identifying the code. In this case, the extraction unit 20g identifies a portion of the scanned image corresponding to the mark formed by the aimer light emitting unit 10 and extracts the identified portion as the code candidate area. As described above, the aimer light indicates the camera 5's shooting range and field of view center, so the mark formed by the aimer light can be aligned with the camera 5's field of view in advance. In particular, with the handheld optical information reading device 1A, the user aims the aimer light at the code to be read, so there is a high probability that the mark formed by the aimer light will overlap with the code. By aligning the portion corresponding to the mark formed by the aimer light emitting unit 10 with the camera 5's field of view center, the extraction unit 20g extracts an area including the camera 5's field of view, which is an area where a code is likely to exist. In this case, the part corresponding to the mark formed by the aimer light irradiation unit 10 does not have to be perfectly aligned with the center of the field of view of the camera 5; it may be slightly misaligned vertically or horizontally. Also, because the code has a predetermined size, the area extracted by the extraction unit 20g is not just the center of the field of view of the camera 5, but an area of a predetermined size that includes the center of the field of view.

The extraction unit 20g can also identify the center of the scanned image and extract that center as a code candidate area. For example, by acquiring the center of the field of view of the camera 5 in advance as information for identifying the code, it is possible to identify the center of the field of view of the camera 5 on the scanned image. In particular, in the case of a handheld optical information reading device 1A, the center identified on the scanned image corresponds to the mark formed by the aimer light irradiation unit 10, and therefore this center is an area where a code is likely to exist.

The extraction unit 20g can be configured to accept a user's designation of a specific portion of the scanned image and extract the designated portion as a code candidate area. That is, when the user designates a region of any size at any position (coordinate) on the scanned image, the extraction unit 20g accepts the designation of the specific portion of the scanned image based on the coordinates and size information of that position. The extraction unit 20g extracts the accepted portion as a code candidate area. For example, in the case of a stationary optical information reading device 1, as shown in FIG. 1, a workpiece W being transported by a conveyor belt B is photographed and a code decoding process is performed. However, the workpiece W is not necessarily located in the center of the width of the conveyor belt B, but may be located at an edge. In this case, by designating a portion of the scanned image corresponding to the edge of the conveyor belt B, it is possible to accurately extract the region as a region likely to contain a code. In addition, there are cases where a code is displayed near the edge of a large workpiece W, away from the center. In such cases, by specifying a part of the scanned image that corresponds to the edge of the workpiece W, it is possible to accurately extract the area where the code is likely to be present.

(Configuration of storage unit)
The storage unit 30 shown in FIG. 3 can be configured as a readable/writable storage device such as an SSD (solid state drive). However, the following storage units 30a, 30b, 30c, and 30d may be provided in a ROM 40 instead of the storage device. That is, this embodiment also covers a configuration in which the image data storage unit 30a, the decoded result storage unit 30b, the parameter set storage unit 30c, and the neural network storage unit 30d are included in the ROM 40. The image data storage unit 30a stores a scanned image generated by the camera 5. The decoded result storage unit 30b stores the decoded result of the code executed by the decoding processing unit 20f. The parameter set storage unit 30c stores the results of tuning executed by the tuning execution unit 20d, various conditions set by the user, and various conditions set by the user. The neural network storage unit 30d stores the structure and parameters of a trained neural network, which will be described later.

(Image restoration using neural networks)
The optical information reading device 1, 1A has an image restoration function that uses a neural network to restore a read image acquired by the camera 5. In this embodiment, the optical information reading device 1, 1A does not perform machine learning, but rather the optical information reading device 1, 1A stores in advance the structure and parameters of a neural network that has already completed learning, and performs image restoration using a neural network configured with the stored structure and parameters, which is different from conventional optical information reading devices.

As shown in Figure 6, a neural network has an input layer to which input data (image data in this example) is input, an output layer that outputs output data, and an intermediate layer located between the input and output layers. For example, multiple intermediate layers can be provided, thereby creating a neural network with a multi-layer structure.

(Neural network learning)
First, the basic procedure for training a neural network will be described with reference to the flowchart shown in Figure 7. Training of the neural network can be performed using a computer prepared for training purposes other than the optical information reading device 1, 1A, but it can also be performed using a general-purpose computer other than one for training purposes. Furthermore, the neural network training method can be a conventionally known method.

After starting, in step SA1, data for a previously prepared defective image is read. A defective image is an image that has parts that are inappropriate for reading a code, and an example of this is an image like the one shown on the left side of Figure 8. Inappropriate parts include, for example, dirty or colored parts. Then, proceeding to step SA2, data for a previously prepared ideal image is read. An ideal image is an image that is appropriate for reading a code, and an example of this is an image like the one shown on the right side of Figure 8. A defective image and an ideal image are paired, and multiple such pairs are prepared.

In step SA2, a loss function is calculated to determine the difference between the defective image and the ideal image. In step SA3, the neural network parameters are updated to reflect the difference determined in step SA2.

In step SA4, it is determined whether the machine learning completion conditions are met. The machine learning completion conditions can be set based on the difference determined in step SA2; for example, if the difference is less than a predetermined value, it can be determined that the machine learning completion conditions are met. If step SA4 returns YES, meaning the machine learning completion conditions are met, the machine learning ends. On the other hand, if step SA4 returns NO, meaning the machine learning completion conditions are not met, the process returns to step SA1, where another defective image is read, and then the process proceeds to step SA2, where another ideal image paired with the other defective image is read.

In this way, a trained neural network can be generated in advance by performing machine learning on multiple defective images and multiple ideal images corresponding to each of the multiple defective images. By storing the structure and parameters of the trained neural network in the optical information reading device 1, 1A, the trained neural network can be constructed within the optical information reading device 1, 1A. In other words, when performing machine learning, a huge number of defective images and ideal images are input into the neural network, which requires extremely high computing power. Ensuring such high computing power within the optical information reading device 1, 1A is difficult and could result in an inability to meet the demand for a compact and lightweight device. By performing machine learning outside the optical information reading device 1, 1A and storing the resulting neural network structure and parameters within the optical information reading device 1, 1A, as in this example, image restoration using a neural network becomes possible while achieving a compact and lightweight optical information reading device 1, 1A.

Figure 9 is a conceptual diagram of a convolutional neural network (CNN) configured as described above. "Convolution" extracts features from the input image. This layer is composed of convolution operations similar to image processing filters. The filter weights are called kernels, and feature extraction is performed according to the kernels. Convolution layers generally have multiple different kernels, and the number of maps (D) increases depending on the number of kernels. Kernel sizes can be, for example, 3x3 or 5x5.

"Pooling" performs a reduction process to combine the responses of each kernel. "Deconvolution" uses a deconvolution filter to reconstruct an image from feature values. "Unpooling" performs an expansion process to sharpen the response values.

The structure and parameters of the trained neural network are stored in the neural network storage unit 30d of the storage unit 30 shown in Figure 3. The neural network structure includes the number of intermediate layers (D) provided between the input layer and output layer, the number of image processing filters, etc. The neural network parameters are the parameters set in step SA3 of the flowchart shown in Figure 7.

The neural network storage unit 30d is also capable of storing the structures and parameters of multiple neural networks with different numbers of layers or image processing filters. In this case, the structures and parameters of multiple neural networks can be stored in the neural network storage unit 30d in an associated state with code conditions. For example, the structure and parameters of a first neural network are associated with the first code conditions corresponding to the first neural network. When associating, the number of layers or filters of the first neural network can be optimally determined in advance using the first code conditions, particularly the first PPC, so the association is made to maintain this relationship. Similarly, the structure and parameters of a second neural network different from the first neural network are associated with second code conditions different from the first code conditions.

If the neural network structures and parameters of multiple neural networks are stored in the neural network storage unit 30d, the tuning execution unit 20d operates as follows when setting the optical information reading device 1, 1A. That is, when setting the code conditions contained in the read image generated by the camera 5, the tuning execution unit 20d reads the neural network structure and parameters associated with the set code conditions from the neural network storage unit 30d and identifies it as the neural network to be used during operation. For example, if the tuning execution unit 20d sets a first code condition as the code condition contained in the read image, it reads the first neural network structure and parameters associated with the first code condition from the neural network storage unit 30d. Then, it identifies the first neural network structure and parameters as the neural network to be used during operation of the optical information reading device 1, 1A. This allows the read image to be repaired using the neural network structure and parameters that are optimal for the code conditions, thereby improving reading accuracy.

When training the neural network, it is also possible to train it on defective images and ideal images where the pixel resolution of the modules that make up the code is within a specific range. The pixel resolution of a module can be expressed, for example, as the number of pixels (PPC) in one module that makes up the code, and the neural network can be trained on only defective images and ideal images where the PPC is within a specific range.

The above-mentioned specific range includes pixel resolutions that provide a high level of restoration effect on scanned images, but excludes pixel resolutions that provide little improvement in restoration effect on scanned images. This range can be set assuming the restoration of images containing codes made up of multiple modules. This allows for improved processing speed as a neural network structure specialized for the restoration of images containing codes.

The specific range of pixel counts can be, for example, 4 PPC or more and 6 PPC or less, but it can also be less than 4 PPC or more than 6 PPC. Furthermore, by limiting the specific range to 4 PPC or more and 6 PPC or less and performing machine learning, for example, it is possible to eliminate unnecessary scale variations from the images used for training, and to optimize the trained neural network structure.

(Repair attempt)
As shown in Fig. 5, the processor 20 of the optical information reading device 1, 1A is provided with an image restoration unit 20j configured with an AI chip 23. The image restoration unit 20j reads out the structure and parameters of a neural network stored in the neural network storage unit 30d of the storage unit 30, and configures a neural network in the optical information reading device 1, 1A using the read neural network structure and parameters. The neural network configured in the optical information reading device 1, 1A is the same as the trained neural network trained according to the procedure shown in the flowchart of Fig. 7.

The image restoration unit 20j inputs the scanned image generated by the camera 5 into a neural network, and attempts to restore the scanned image according to the neural network structure and parameters stored in the neural network storage unit 30d. The decoding processing unit 20f then performs a decoding process on the restored scanned image.

The filter processing unit 20c is configured to apply an image processing filter to the scanned image generated by the camera 5 before the image restoration unit 20j inputs the scanned image into the neural network. This allows appropriate image processing to be performed before attempting restoration using the neural network, resulting in more accurate restoration of the scanned image.

The image restoration unit 20j may attempt image restoration on all read images decoded by the decoding processing unit 20f, or may attempt image restoration on only some of the read images decoded by the decoding processing unit 20f. For example, the decoding processing unit 20f performs a decoding process on the read image before attempting restoration and determines whether the decoding was successful. If the decoding is successful, it means that the read image does not require restoration, and the result is output as is. On the other hand, if the decoding fails, the image restoration unit 20j attempts to restore the read image, and the decoding processing unit 20f performs a decoding process on the restored read image.

(Partial image input)
The image input by the image restoration unit 20j to the neural network may be the entire scanned image generated by the camera 5, but the larger the size of the input image, the greater the amount of calculation by the neural network, which increases the calculation load on the processor 20 and may ultimately result in a decrease in processing speed. Specifically, in an FCN (Fully Convolutional Network) type neural network as shown in Figure 9, assuming the structure is the same, the amount of calculation is proportional to the input image size, and since the input image size is proportional to the square of the code size, the amount of calculation is also proportional to the square of the code size.

On the other hand, there are virtually no cases where a code is present throughout the entire scanned image generated by camera 5. During normal operation, a code is present only in a small area of the scanned image, and if only that area can be repaired using a neural network, reading accuracy can be improved.

The image that the image restoration unit 20j inputs into the neural network can be a portion of the scanned image generated by the camera 5, i.e., a partial image containing a code. Specifically, the image restoration unit 20j cuts out a partial image corresponding to the code candidate area extracted by the extraction unit 20g from the scanned image generated by the camera 5. By inputting the cut-out partial image into the neural network, the image restoration unit 20j attempts to restore the partial image according to the structure and parameters stored in the neural network storage unit 30d. The decoding processing unit 20f then performs decoding processing on the restored partial image.

As mentioned above, it is rare for a code to be present in the entire scanned image, so the size of the partial image corresponding to the code candidate area is smaller than the size of the scanned image. This reduces the size of the image input to the neural network, reducing the computational load on the processor 20 and ultimately achieving faster processing speeds. Furthermore, because the partial image corresponds to an area where a code is likely to be present, it is possible to create an image that contains the information necessary for reading, i.e., the entire code. Because this image containing the entire code is input to the neural network, a decrease in reading accuracy is suppressed.

The image restoration unit 20j can determine the input size of the partial image to the neural network based on the code size obtained by the tuning execution unit 20d. As described above, the tuning execution unit 20d can obtain the code size. The image restoration unit 20j increases the input size of the partial image to be input to the neural network by a predetermined amount compared to the code size obtained by the tuning execution unit 20d.

In other words, while inputting a partial image into the neural network can reduce the computational load, if, for example, the code is rotated or the code position cannot be detected precisely, part of the code in the partial image may be missing, which could result in a failed decoding process. In this embodiment, the input size to the neural network is not the same as the code size acquired by the tuning execution unit 20d, but is larger than the code size by a predetermined amount, thereby preventing part of the code from being missing from the partial image. Furthermore, by making the input size larger than the code size by a predetermined amount, an upper limit can be placed on the input size to the neural network, thereby preventing the computational load from becoming too heavy. The "predetermined amount" may mean, for example, that the horizontal (or vertical) length of the partial image input to the neural network is 1.5 times or more, 2.0 times or more, or 3.0 times or more the horizontal (or vertical) length of the code acquired by the tuning execution unit 20d.

When the image restoration unit 20j determines the input size of the partial image to be input to the neural network, it can do so based on the number of pixels in one module that makes up the code acquired by the tuning execution unit 20d and the number of modules arranged vertically or horizontally in the code.

The tuning execution unit 20d can also set the code conditions as described above. The image restoration unit 20j can also set the size of the scanned image to be input to the neural network based on the code conditions set by the tuning execution unit 20d. For example, when the image restoration unit 20j acquires the code size from the code conditions set by the tuning execution unit 20d, it inputs a partial image that is a predetermined amount larger than the acquired code size to the neural network. If the code size is larger, the size of the partial image input to the neural network will be larger, and if the code size is smaller, the size of the partial image input to the neural network will be smaller. In other words, the size of the partial image input to the neural network can be changed depending on the code conditions.

(Zoom in/out function)
In order to fully capture the features of the code and restore the image to a suitable size for decoding using a convolutional neural network such as that shown in Figure 9, it is necessary to extract features within a range covering a certain number of modules. For example, if one feature value obtained from the neural network is a value extracted from a 6x6 module range, and an image is captured with one module consisting of 20 pixels, the neural network must be designed to calculate a feature value aggregated from a 120x120 pixel range. In other words, the wider the pixel range to be covered, the deeper the neural network hierarchy must be. For example, to aggregate feature values from the above-mentioned 120x120 pixel range, six convolutional layers are required.

However, because the code is composed of a random arrangement of white and black modules, the relationship between pixel values over a wide range is weak, and expanding the pixel range covered beyond a certain point hardly improves the restoration effect. In this specification, such features are referred to as narrow-range features.

Furthermore, the arrangement of fixed module patterns, such as finder patterns contained in code, or the rectangular shape of the code as a whole can be considered wide-range features. However, if you only roughly separate the modules from the background, recovery is sufficient using only narrow-range features, without using wide-range features. Wide-range features can be defined as features that have a fixed shape over a wide range.

Therefore, when training the neural network shown in Figure 7, images with code PPCs that fall within a specific range can be used. This can improve processing speed as a neural network structure specialized for repairing images that contain codes, but there is a concern that the repair effect will be reduced for scanned images with PPCs that fall outside the specific range. Note that when training the neural network, images with code PPCs that fall outside the specific range may also be used.

The optical information reading device 1, 1A of this embodiment is equipped with a function for reducing and enlarging the read image so that the PPC of the read image falls within a specific range. As shown in FIG. 5, the processor 20 is configured with a reduction unit 20h and an enlargement unit 20i. The reduction unit 20h is a part that generates a reduced read image so that the pixel resolution of the modules that make up the code in the read image generated by the camera 5 falls within the specific range described above. The enlargement unit 20i is a part that generates an enlarged read image so that the pixel resolution of the modules that make up the code in the read image generated by the camera 5 falls within the specific range described above.

The reduction unit 20h and the enlargement unit 20i determine whether the PPC of the code in the scanned image generated by the camera 5, for example, is outside a specific range (4 PPC to 6 PPC). If it is within the specific range, they do not perform reduction or enlargement, but if it is outside the specific range, they perform reduction or enlargement. By performing reduction or enlargement, the PPC of the code in the partial image input to the neural network by the image restoration unit 20j falls within the specific range.

The image restoration unit 20j inputs the scanned image, enlarged or reduced so that it fits within a specific range, into a neural network configured with the structure and parameters stored in the neural network storage unit 30d, and attempts to restore the scanned image according to the structure and parameters. The decoding processing unit 20f then performs a decoding process on the restored scanned image.

The extraction unit 20g may be configured to search for codes in the scanned image enlarged by the enlargement unit 20i or reduced by the reduction unit 20h, and extract the area containing the searched code as a code candidate area where a code is likely to exist. In this case, the image restoration unit 20j inputs the partial image corresponding to the code candidate area extracted by the extraction unit 20g into a neural network configured with the structure and parameters stored in the neural network storage unit 30d, and attempts to restore the scanned image in accordance with the structure and parameters.

When setting up the optical information reading device 1, 1A, the tuning execution unit 20d can generate multiple read images (enlarged images) with different magnification ratios and multiple read images (reduced images) with different reduction ratios. The tuning execution unit 20d inputs the generated multiple read images (enlarged or reduced images) into a neural network to attempt to repair each read image, and performs a decoding process on each repaired read image to determine the reading margin, which indicates the ease of reading the code. The tuning execution unit 20d identifies the magnification or reduction ratio of the read image for which the determined reading margin is higher than a predetermined value as the magnification or reduction ratio to be used during operation. When the tuning execution unit 20d identifies a magnification or reduction ratio, during operation of the optical information reading device 1, 1A, the reduction unit 20h and the enlargement unit 20i enlarge or reduce the read image at the magnification or reduction ratio identified by the tuning execution unit 20d.

In other words, for example, if the specific range is relatively large, changing the magnification or reduction ratio within that specific range may change the reading margin. The magnification or reduction ratio of the scanned image during operation can be specified so that the reading margin determined by the tuning execution unit 20d is higher than the specified value, thereby improving processing speed and reading accuracy.

After calculating the reading margin for each scanned image, the tuning execution unit 20d can identify the magnification or reduction ratio of the scanned image with the highest reading margin among the multiple reading margins calculated as the magnification or reduction ratio to be used during operation. This can further improve processing speed and reading accuracy.

Furthermore, by utilizing the ability of the reduction unit 20h and the enlargement unit 20i to reduce and enlarge the scanned image, the number of parameter sets can be increased. Multiple parameter sets with different reduction ratios by the reduction unit 20h and multiple parameter sets with different enlargement ratios by the enlargement unit 20i can be generated, and these parameter sets can be stored in the parameter set storage unit 30c of the storage unit 30. For example, if multiple reduction ratios can be set by the reduction unit 20h, such as 1/2, 1/4, and 1/8, the parameter set with a reduction ratio of 1/2, a parameter set with a reduction ratio of 1/4, and a parameter set with a reduction ratio of 1/8 can be stored in the parameter set storage unit 30c. Furthermore, if multiple enlargement ratios can be set by the enlargement unit 20i, such as 2x, 4x, and 8x, the parameter set with a magnification ratio of 2x, a parameter set with a magnification ratio of 4x, and a parameter set with a magnification ratio of 8x can be stored in the parameter set storage unit 30c. Any one of the multiple parameter sets stored in the parameter set storage unit 30c can then be applied.

(Image restoration filter)
The image restoration unit 20j may be a part that executes an image restoration filter that attempts to restore the scanned image using a neural network. The image restoration filter is a filter that attempts to restore the scanned image in accordance with the structure and parameters stored in the neural network storage unit 30d by inputting the scanned image to a neural network configured with the structure and parameters stored in the neural network storage unit 30d, and performs the same function as the image restoration described above.

When an image restoration filter is executable, a setting unit 20e (shown in Figure 5) for setting the image restoration filter can be provided. The setting unit 20e is configured to accept the user's setting of the image restoration filter. For example, when setting up the optical information reading device 1, 1A, the setting unit 20e can be configured to generate a user interface that allows the user to select either "apply image restoration filter" or "do not apply image restoration filter," display this on the display unit 6, and accept the user's selection. Therefore, the image restoration unit 20j attempts to restore the scanned image by applying the image restoration filter set by the setting unit 20e to the scanned image.

The setting unit 20e may be included in the tuning execution unit 20d, or may be configured separately from the tuning execution unit 20d. If the setting unit 20e is included in the tuning execution unit 20d, it becomes possible to set parameters related to the image restoration filter during tuning. Parameters related to the image restoration filter can include "apply image restoration filter" and "not apply image restoration filter." "Apply image restoration filter" means setting the image restoration filter to be executable, and "not apply image restoration filter" means not setting the image restoration filter.

The parameters related to the image restoration filter are also information related to the image restoration filter. In this case, a parameter set including information related to the image restoration filter settings can be stored in the parameter set storage unit 30c. The parameters related to the image restoration filter include "apply image restoration filter" and "not apply image restoration filter." Therefore, the parameter sets stored in the parameter set storage unit 30c include a first parameter set that sets the image restoration filter to be executable, and a second parameter set that does not set the image restoration filter. When the optical information reading device 1, 1A is operated, one of the first and second parameter sets is applied. When the first parameter set is applied, the decoding processing unit 20f performs decoding processing on the read image on which image restoration has been performed using the image restoration filter. Meanwhile, when the second parameter set is applied, the decoding processing unit 20f performs decoding processing on the read image on which image restoration has not been performed.

The parameter sets stored in the parameter set storage unit 30c also include items for setting the code conditions included in the scanned image generated by the camera 5. The items for setting the code conditions include the PPC, code type, code size, etc. acquired by the tuning execution unit 20d. For example, one parameter set may include the PPC, code type, and code size as items for setting the code conditions, the gain of the camera 5, the light intensity and exposure time of the illumination unit 4 as imaging conditions, and the type of image processing filter, parameters of the image processing filter, and application items of the image restoration filter as items for the image processing filter applied by the filter processing unit 20c. The values set by the tuning execution unit 20d may be used as is for each of these items, or the user may change them as desired.

(An example of the tuning process)
An example of the procedure of the tuning process performed by the tuning execution unit 20d when setting up the optical information reading device 1, 1A will be specifically described with reference to the flowchart shown in Fig. 10. In step SB1 after the start of the flowchart shown in Fig. 10, the tuning execution unit 20d controls the illumination unit 4 and camera 5 to cause the camera 5 to generate a read image, and the tuning execution unit 20d acquires the read image. At this time, the presence and type of an image processing filter to be executed before the decoding process and the decoding process parameters for applying image restoration by a neural network are set to arbitrary parameters. Next, proceeding to step SB2, the tuning execution unit 20d causes the decoding processing unit 20f to execute decoding process on the acquired read image.

After the decoding process, the process proceeds to step SB3, where the tuning execution unit 20d determines whether the decoding process of step SB2 was successful. If the determination in step SB3 is NO and the decoding process of step SB2 has failed, i.e., if the code could not be read, the process proceeds to step SB4, where the decoding process parameters are changed to other parameters, and the decoding process is performed again in step SB2. If the decoding process has failed with all decoding process parameters, this flow ends and the user is notified.

On the other hand, if step SB3 returns YES and the decoding process in step SB2 is successful, the process proceeds to step SB5, where the tuning execution unit 20d determines code parameters (PPC, code type, code size, etc.). Once the tuning execution unit 20d determines the code parameters, it also determines the structure and parameters of the neural network stored in the neural network storage unit 30d in association with the code parameters (step SB6). In step SB6, a neural network is constructed using the structure and parameters read from the neural network storage unit 30d.

In step SB7, the image restoration unit 20j attempts to restore the read image by inputting the read image into the neural network configured in step SB6, and the decoding processing unit 20f performs decoding processing on the restored read image. Then, proceeding to step SB8, the tuning execution unit 20d evaluates the reading margin based on the results of the decoding processing in step SB7 and temporarily stores the evaluation.

In step SB9, it is determined whether the decoding process has been completed for all decoding process parameters. If the determination in step SB9 is NO, meaning that the decoding process has not been completed for all decoding process parameters, proceed to step SB10, change the decoding process parameter to another parameter, and then proceed to step SB7.

On the other hand, if step SB9 returns YES and the decoding process is completed for all decoding process parameters, the process proceeds to step SB11. In step SB11, the tuning execution unit 20d selects the decoding process parameter with the highest reading margin from among all the decoding process parameters, and determines the selected decoding process parameter as the parameter to be applied during operation. Note that during the tuning process, imaging conditions, etc. are also set to appropriate conditions.

(Decoding procedure before determining reduction and enlargement ratios)
Next, an example of a decoding process procedure before determining the reduction ratio and enlargement ratio will be specifically described with reference to the flowchart shown in Fig. 11. The process specified in the flowchart shown in Fig. 11 can be executed in step SB2 of the flowchart shown in Fig. 10.

In step SC1 after the start of the flowchart shown in Figure 11, the tuning execution unit 20d selects an arbitrary image processing filter from among multiple image processing filters. This image processing filter is the filter executed by the filter processing unit 20c. Then, in step SC2, the filter processing unit 20c executes the image processing filter selected in step SC1 on the scanned image.

Next, proceed to step SC3 and create an image pyramid. The image pyramid is composed of the original scanned image, an image obtained by reducing the original scanned image by half, an image obtained by reducing the original scanned image by a quarter, an image obtained by reducing the original scanned image by an eighth, and so on. The reduction of the scanned image is performed by the reduction unit 20h. The image pyramid can also be composed of the original scanned image, an image obtained by enlarging the original scanned image by two times, an image obtained by enlarging the original scanned image by four times, an image obtained by enlarging the original scanned image by eight times, and so on. The enlargement of the scanned image is performed by the enlargement unit 20i. Note that either the reduced image or the enlarged image may be omitted.

After creating the image pyramid, the process proceeds to step SC4, where an arbitrary scanned image is selected from the multiple scanned images that make up the image pyramid. The selected images may include the original scanned image that has not been reduced or enlarged. In step SC5, the extraction unit 20g extracts code candidate areas that are likely to contain a code from the scanned images selected in step SC4. The process then proceeds to step SC6, where the image restoration unit 20j inputs the partial image corresponding to the code candidate area extracted in step SC5 into a neural network and attempts to restore the scanned image. The size of the partial image input to the neural network is determined by the code conditions described above.

After the scanned image is restored, the process proceeds to step SC7, where a positioning process is performed on the outline of the code and the modules that make up the code. After the positioning process, the process proceeds to step SC8, where it is determined whether each module that makes up the code is white or black. After determining whether it is black or white, the process proceeds to step SC9, where the decoding processing unit 20f performs a decoding process on the restored scanned image. The decoding process can use a method, for example, to restore a character string from the 0-1 matrix of the modules.

Then, the process proceeds to step SC10, where the tuning execution unit 20d determines whether the decoding process of step SC9 was successful. If the determination in step SC10 is NO and the decoding process of step SC9 failed, the process proceeds to step SC11, where it determines whether all of the scanned images that make up the image pyramid have been selected. If the determination in step SC11 is NO and all of the scanned images that make up the image pyramid have not been selected, the process proceeds to step SC4, where another reduced scanned image or another enlarged scanned image that makes up the image pyramid is selected, and the process proceeds to step SC5.

On the other hand, if step SC11 returns YES and all scanned images that make up the image pyramid have been selected, this flow ends and the conditions for successful decoding are stored.

(Decoding procedure after determining reduction and enlargement ratios)
Next, an example of a decoding process procedure after determining the reduction ratio and enlargement ratio will be specifically described with reference to the flowchart shown in Fig. 12. The process specified in the flowchart shown in Fig. 12 can be executed in step SB7 of the flowchart shown in Fig. 10 and when the optical information reader 1, 1A is in operation.

In step SD1 after the start of the flowchart shown in FIG. 12, the filter processing unit 20c applies the image processing filter selected in step SC1 of the flowchart shown in FIG. 11 to the read image. At this time, if the filter processing unit 20c applies an averaging filter to the read image shown in the upper part of FIG. 13, the image obtained after application of the image processing filter will be as shown in the lower part of FIG. 13.

Then, proceed to step SD2, where the scanned image is reduced or enlarged as necessary. Specifically, if the PPC of the code in the scanned image after the image processing filter is applied is within a specific range, reduction or enlargement is not performed; if it is outside the specific range, reduction or enlargement is performed so that the PPC falls within the specific range. An example of the scanned image after reduction processing is shown at the top of Figure 14.

Next, in step SD3, the extraction unit 20g extracts code candidate areas that are likely to contain a code from the scanned image reduced or enlarged in step SD2. An example of an extracted code candidate area image is shown in the lower part of Figure 14. Note that if the scanned image was not reduced or enlarged in step SD2, the extraction process will be performed on the original scanned image in step SD3.

Then, the process proceeds to step SD4, where the image restoration unit 20j inputs the partial image corresponding to the code candidate area extracted in step SD3 into a neural network and attempts to restore the scanned image. Figure 15 shows examples of the scanned image before and after restoration. After the scanned image is restored, steps SD5 to SD7 are performed, and the flow ends. Steps SD5 to SD7 are the same as steps SC7 to SC9 in the flowchart shown in Figure 11.

(Effects of the embodiment)
As described above, according to this embodiment, when the optical information reading device 1, 1A is in operation, if the read image generated by the camera 5 is input to the neural network, an attempt is made to repair the read image, and a read image after repair attempt is obtained that has been repaired to an image suitable for reading the code. A decoding process is performed on this read image after repair attempt, thereby improving reading accuracy.

The neural network used for repair is constructed with a structure and parameters obtained by prior machine learning of multiple defective images and multiple ideal images. Therefore, the optical information reading device 1, 1A does not need to perform learning processing; only inference processing using the neural network is required, thereby reducing the computational load on the processor 20. As a result, the handheld optical information reading device 1A can be made smaller and lighter, and the processing speed of the stationary optical information reading device 1 can be increased.

Furthermore, when the optical information reading device 1, 1A is in operation, if a code candidate area is extracted from the read image, the partial image corresponding to that code candidate area is input into the neural network, and an attempt is made to repair the partial image. This reduces the size of the image input into the neural network, reducing the computational load on the processor 20 and ultimately achieving faster processing speeds. Furthermore, because the partial image corresponds to an area where a code is likely to exist, it is possible to create an image that includes the information necessary for reading, i.e., the entire code. Because an image including this entire code is input into the neural network, a decrease in reading accuracy is suppressed.

Furthermore, when generating the neural network structure and parameters, the system uses machine learning to learn defective and ideal images in which the pixel resolution of the modules that make up the code is within a specific range, thereby improving processing speed as a neural network structure specialized for repairing images that contain codes. Furthermore, if the pixel resolution of the modules that make up the code in the scanned image is outside the specific range, the scanned image is enlarged or reduced so that it falls within the specific range before being input into the neural network, making it possible to repair images that contain codes with a wide range of pixel resolution.

(Other embodiments)
16 is a diagram illustrating an example of a processing procedure during operation of an optical information reading device 1 according to another embodiment. In this embodiment, in addition to the image restoration unit 20j, the processor 20 includes a pre-processing unit 200, a post-processing unit 201, a first decoding unit 202, and a second decoding unit 203. The pre-processing unit 200, like the filtering unit 20c, is a unit that applies a noise reduction filter, a contrast correction filter, an averaging filter, etc. to the read image generated by the camera 5.

The first decoding processing unit 202 is a unit that performs decoding processing on the scanned image generated by the camera 5, and the decoding processing performed by this first decoding processing unit 202 is also referred to as the first decoding processing. If the processor 20 has multiple cores (N cores), the first decoding processing unit 202 is made up of cores 0 to i. The first decoding processing unit 202 may also be made up of a single core.

The second decoding processing unit 203 is a unit that performs decoding processing on the restored image generated by the image restoration unit 20j, and the decoding processing by this second decoding processing unit 203 is also referred to as the second decoding processing. If the processor 20 has N cores, the second decoding processing unit 203 is made up of cores i+1 to N-1. The second decoding processing unit 203 may also be made up of a single core.

As also shown in Figure 17, when the first decoding processing unit 202 is made up of cores 0 to i, each of cores 0 to i extracts code candidate areas where a code is likely to exist from the scanned image generated by camera 5, then performs positioning processing for the code outline and the modules that make up the code, and then determines whether each module that makes up the code is white or black and performs decoding processing. The processing results are output to the post-processing unit 201.

On the other hand, as shown in Figure 17, when the second decoding processing unit 203 is made up of cores i+1 to N-1, each of cores i+1 to N-1 extracts code candidate areas from the scanned image that are likely to contain code as repair code areas, and sends a trigger signal to the image repair unit 20j to execute inference processing. While Figure 17 shows one task being executed by two cores, one task may be executed by any one or more cores.

Specifically, when cores N-2 and N-1 constituting the second decoding processing unit 203 extract a repair code candidate area, they output a trigger signal (Trigger 1) to the image restoration unit 20j, causing the image restoration unit 20j to execute inference processing. Upon receiving the trigger signal (Trigger 1) sent from cores N-2 and N-1, image restoration unit 20j inputs the partial image corresponding to the repair code candidate area extracted by cores N-2 and N-1 into a neural network and executes inference processing to generate a repaired image by repairing the partial image (Image Restoration 1). The repaired image generated by Image Restoration 1 is sent to cores N-2 and N-1. Based on the repaired image generated by Image Restoration 1, cores N-2 and N-1 determine grid positions indicating the position of each cell of the code and execute decoding processing for the repaired image based on the determined grid positions.

Meanwhile, when cores i+1 and i+2 constituting the second decoding processing unit 203 extract a repair code candidate area, they output a trigger signal (Trigger 2) to the image restoration unit 20j, causing the image restoration unit 20j to execute inference processing. Upon receiving the trigger signal (Trigger 2) sent from cores i+1 and i+2, image restoration unit 20j inputs the partial image corresponding to the repair code candidate area extracted by cores i+1 and i+2 into a neural network and executes inference processing to generate a repaired image by repairing the partial image (Image Restoration 2). The repaired image generated by Image Restoration 2 is sent to cores i+1 and i+2. Based on the repaired image generated by Image Restoration 2, cores i+1 and i+2 determine grid positions indicating the position of each cell of the code and execute decoding processing for the repaired image based on the determined grid positions.

In the same manner, image restoration 3 is executed in response to a trigger signal (trigger 3) sent from cores N-2 and N-1, image restoration 4 is executed in response to a trigger signal (trigger 4) sent from cores i+1 and i+2, image restoration 5 is executed in response to a trigger signal (trigger 5) sent from cores N-2 and N-1, image restoration 6 is executed in response to a trigger signal (trigger 6) sent from cores i+1 and i+2, and image restoration 7 is executed in response to a trigger signal (trigger 7) sent from cores N-2 and N-1.

As shown in this time chart, the inference processing by the image restoration unit 20j and the first decoding processing by the first decoding processing unit 202 are executed in parallel. For example, while the first decoding processing unit 202 continues to perform code region extraction, grid positioning, and decoding processing, the inference processing by the image restoration unit 20j is executed one or more times. Also, while the first decoding processing unit 202 continues to perform code region extraction, grid positioning, and decoding processing, the second decoding processing unit 203 extracts a repair code region, outputs a trigger signal, positions a grid, and executes second decoding processing. In this way, the image restoration unit 20j only needs to specialize in repairing the partial image corresponding to the repair code candidate region, thereby increasing the inference processing speed.

At least a portion of the time during which the second decoding processing unit 20j is performing the second decoding process includes a pause during which the image restoration unit 20j does not perform inference processing. That is, as shown in FIG. 17, after the image restoration unit 20j completes image restoration 1, the second decoding processing unit 203 begins the second decoding process. However, a predetermined period is provided before the image restoration unit 20j begins image restoration 2. This predetermined period is designated as a pause during which the image restoration unit 20j does not perform image restoration. This pause is a period during which the image restoration unit 20j does not perform inference processing. By providing this pause, the load on the image restoration unit 20j is reduced, and heat generation by the AI chip 23 can be suppressed. Note that the processing time for grid positioning may be long or short depending on the image to be restored. While FIG. 17 illustrates an example in which the processing of each core is performed alternately, depending on the processing time for grid positioning, the processing of each core does not necessarily occur alternately. That is, trigger issuance and grid positioning may be performed in order starting with the core with the least processing time.

As described above, the setting unit 20e is configured to allow the user to select either "applying an image restoration filter" or "not applying an image restoration filter." Applying an image restoration filter means performing image restoration, and not applying an image restoration filter means not performing image restoration. Therefore, the setting unit 20e also corresponds to the part that sets whether or not the second decoding processing unit 203 will perform the second decoding process. This setting is performed by the user, and therefore the setting unit 20e is configured to be able to accept user operations. The setting is not limited to applying or not applying an image restoration filter, and may be, for example, applying or not applying inference processing (restoration processing), or applying or not applying AI processing. User operations may include, for example, operating buttons displayed on a user interface screen.

The optical information reading device 1 is operated after a tuning process is performed to determine in advance the settings of imaging conditions such as exposure time and gain, as well as image processing filters. For example, if the workpiece W is a label, the code is printed clearly, so if the appropriate conditions are set through the tuning process, a high reading rate can be achieved. However, for workpieces W made of metal or resin, even if the optimal conditions are set during the tuning process, reading may be difficult due to changes in the read image caused by weak ambient light, or the optical design and algorithm design may make it impossible to read at extremely low contrast.

By performing the image restoration algorithm executed by the image restoration unit 20j in this example in parallel with the decoding process without image restoration, it is possible to improve the reading performance for workpieces W made of metal, resin, etc. while maintaining conventional reading performance, thereby increasing reading stability. Furthermore, by applying the image restoration algorithm, it becomes possible to read workpieces W that cannot be read using conventional optical designs and algorithms.

To give a specific example, in a process at a factory where codes printed on stickers attached to boxes are read, if there are no restrictions on the installation conditions of the optical information reading device 1 and optimal installation is possible, stable operation is possible even with conventional reading performance, so the image restoration filter can be disabled and takt time can be prioritized.

In addition, one factory has a process for reading codes that are clearly printed on metal. Although optimal conditions are determined during the tuning process when setting up the reading, in actual operation, reading can become unstable due to scratches on the metal surface, variations in the reading position and angle, and changes in the brightness of the surrounding environment. In such cases, applying an image restoration filter to improve image quality makes the system more resistant to variations in conditions such as those mentioned above, improving reading performance.

In addition, some workpieces cannot be read using conventional algorithms, but in such cases, applying an image restoration filter can make them readable. For example, even if the code is severely damaged or the contrast is unclear, applying an image restoration filter can restore the contrast and scratches, making it possible to read the code.

The second decoding process performed when an image restoration filter is applied performs decoding on a restored image, and therefore often takes longer than the first decoding process, which performs decoding on a normal scanned image. Therefore, the first decoding processing unit 202 can perform the first decoding process in a shorter time than the time required for the second decoding process by the second decoding processing unit 203. Based on this premise, in this example, the setting unit 20e can change the decoding timeout time, which is the time limit for the decoding process.

Specifically, the setting unit 20e is configured to be able to set a decode timeout time longer than the time required to execute the second decoding process when the second decoding process is set to be executed, and to be able to set a decode timeout time shorter than the time required to execute the second decoding process when the second decoding process is set not to be executed. As a result, during operation, when the second decoding process is executed, a decode timeout time longer than the time required to execute a typical second decoding process can be set, ensuring reliable decoding of the restored image. On the other hand, when the second decoding process is set not to be executed, only the first decoding process, which generally can be decoded in a shorter time than the second decoding process, is executed, so a short decode timeout time corresponding to the first decoding process can be set. By changing the decode timeout time in this way, decode timeout times corresponding to both the first decoding process and the second decoding process can be set.

In addition, while the first decoding processing unit 202 and the second decoding processing unit 203 are separate in Figure 17, as shown in Figure 18, the first decoding processing unit 202 and the second decoding processing unit 203 may be combined into a decoding processing unit 210. The decoding processing unit 210 performs a second decoding processing on the restored image in parallel with the first decoding processing on the scanned image generated by the camera 5. In this way, the parts that perform the decoding processing can be arbitrarily divided or integrated on the hardware. Below, an example in which the parts that perform the decoding processing are integrated into one will be explained, but this is not limiting and the same applies to cases in which the parts are divided into multiple parts.

When the setting unit 20e has set the second decoding process not to be performed, the decoding processing unit 210 is configured to allocate processing resources used for the second decoding process to the first decoding process, thereby speeding up the first decoding process compared to when the second decoding process is set to be performed. Processing resources include, for example, memory usage areas and cores constituting a multi-core CPU. Specifically, when the setting unit 20e has set the second decoding process not to be performed, the number of cores responsible for the first decoding process is increased compared to when the setting unit 20e has set the second decoding process to be performed. Furthermore, when the setting unit 20e has set the second decoding process not to be performed, the memory area used for the first decoding process is expanded compared to when the setting unit 20e has set the second decoding process to be performed. Increasing the number of cores and expanding the memory area may be used together, or either one may be performed alone. This maximizes the performance of the multi-core CPU and speeds up the first decoding process.

Prior to image restoration, the decoding processing unit 210 extracts code candidate regions from the scanned image that are likely to contain a code. Specifically, when using a neural network for image restoration, if the image input to the neural network is large, the computation time increases. Furthermore, from the perspective of speeding up processing, it is desirable to minimize the number of decoding attempts. In response to these issues, this example employs a heat map-based search method rather than a line search-based method to ensure reliable extraction of code candidate regions, even if it takes some time. For example, the decoding processing unit 210 quantifies the features of the code, generates a heat map in which the magnitude of the feature is assigned to each pixel value, and extracts code candidate regions from the heat map that are likely to contain a code. A specific example is a method of extracting feature portions (e.g., finder patterns) of a 2D code from areas that appear relatively hot (large feature amounts) in the heat map. If multiple feature portions are acquired, they can be prioritized and extracted, and stored in RAM 41, storage unit 30, etc.

After the decoding processing unit 210 extracts the code candidate region, it outputs the partial image corresponding to the extracted code candidate region to the image restoration unit 20j. The image restoration unit 20j inputs the partial image corresponding to the code candidate region extracted by the decoding processing unit 210 into a neural network and performs inference processing to generate a restored image by restoring the partial image. This reduces the calculation time.

The decoding processing unit 210 may also be configured to extract a code candidate area from the scanned image that is large enough to contain the code to be read, as determined by the tuning process described above, and that is likely to contain the code. The extracted image is then reduced or enlarged to a predetermined size before being input to the neural network, and the repaired image repaired by the neural network is decoded. By enlarging or reducing the extracted image to a predetermined size and inputting it to the neural network, the PPC of the code image input to the neural network can be kept within a predetermined range, ensuring a stable image repair effect by the neural network. Although the code size to be read may vary, the size of the code ultimately input to the neural network can be fixed, thereby maintaining a consistent balance between high processing speed and ease of decoding.

The flowcharts for the first decoding process and the second decoding process will be explained in detail below with reference to Figure 19. In step SE1 after starting, the processor 20 causes the camera 5 to generate a scanned image and acquires the scanned image. The scanned image acquired in step SE1 is input to the decoding processing unit 210. In step SE2, the first decoding process is executed on the scanned image acquired in step SE1 without performing image restoration. If the first decoding process is successful, the process proceeds to step SE4, where termination processing, i.e., output processing of the decoded result, is executed. If the first decoding process fails in step SE2, the process returns to step SE2, and the first decoding process is executed again. When a predetermined decoding timeout period has elapsed, the first decoding process for the scanned image is stopped.

On the other hand, in the route proceeding from step SE1 to step SE5, the decoding processing unit 210 extracts a repair code area from the read image. Specifically, it extracts a code candidate area that is large enough to contain the code to be read, as set by the tuning process, and that is likely to contain the code within the read image. If multiple repair code areas are extracted at this time, they are temporarily saved as candidate areas R1, R2, .... An example of an extracted repair code area is shown in Figure 20. Then, it proceeds to step SE6, and the value of k is set to 1.

In step SE7, a resizing process is performed on the partial image corresponding to candidate region Rk. The resizing process reduces or enlarges the partial image corresponding to the repair code region extracted in step SE5 to a predetermined size. This resizing process allows the size of the image input to the neural network, described below, to be set to a predetermined size in advance. In the example shown in Figure 20, the size of the partial image input to the neural network is 256 pixels x 256 pixels, but the size of the partial image is not limited to this and can be resized to any size taking into account processing speed, etc.

In step SE8, it is determined whether the black and white inversion setting is set to "both." In other words, image restoration is simply a mapping in mathematical terms, and it can be difficult to achieve white-to-black, black-to-white, black-to-black, and white-to-white mappings using a single neural network. For this reason, black and white inversion processing is performed based on the properties so that the image is printed black on a white background. Depending on how the neural network is trained, the image may also be printed white on a black background. An example of black and white inversion processing is shown in Figure 20. Details of black and white inversion processing will be described later.

If step SE8 returns YES and the black-and-white inversion setting is set to "both," the process proceeds to step SE9, where the resized image is input to the image restoration unit 20j and image restoration is performed. Then, the process proceeds to step SE10, where the decode processing unit 210 performs a second decoding process on the restored image. If the second decoding process is successful in step SE10, the process proceeds to step SE4 and executes termination processing. If the second decoding process fails in step SE10, the process proceeds to step SE11, where the resized image is subjected to black-and-white inversion processing. Then, in step SE11, the image after the black-and-white inversion processing is input to the image restoration unit 20j and image restoration is performed (see FIG. 20). Next, the process proceeds to step SE13, where the decode processing unit 210 performs a second decoding process on the restored image. If the second decoding process is successful in step SE13, the process proceeds to step SE4 and executes termination processing. If the second decoding process fails in step SE13, the process proceeds to step SE14. In step SE14, it is determined whether any candidate areas extracted in step SE5 remain, and if no candidate areas remain, the process proceeds to step SE4 to execute termination processing.

On the other hand, if step SE8 returns NO and the black-and-white inversion setting is not set to "Both," the process proceeds to step SE15, where it is determined whether the black-and-white inversion setting is set to "OFF." If step SE15 returns NO and the black-and-white inversion setting is not set to "OFF," the process proceeds to step SE16, where the resized image is inverted. Then, in step SE17, the image after the black-and-white inversion processing is input to the image restoration unit 20j, where image restoration is performed. Next, the process proceeds to step SE18, where the decoding processing unit 210 performs a second decoding process on the restored image. If the second decoding process is successful in step SE18, the process proceeds to step SE4, where termination processing is performed. If the second decoding process fails in step SE18, the process proceeds to step SE14. In step SE14, it is determined whether or not there are still candidate areas, and if there are no candidate areas, the process proceeds to step SE4, where termination processing is performed.

If it is determined in step SE14 that there are more candidate areas, proceed to step SE19, increment the value of k by 1, and proceed to step SE7. In step SE7, since the value of k has increased by 1 since the previous flow, a similar resizing process is performed on the partial image corresponding to the next candidate area Rk. Steps SE8 and beyond are as described above. In other words, if decoding is successful in either the first decoding process or the second decoding process, the decoded result is output at that point, and decoding processes are not performed on other candidate areas; instead, a scanned image of the next work W is obtained and similar processing is performed.

The post-processing unit 201 shown in FIG. 20 performs a process of combining a partial image that has not been restored after resizing with a partial image that has been restored by the image restoration unit 20j after resizing. In other words, when image restoration using a neural network is performed, even if the code before restoration is visible to the human eye, image restoration may not be successful, or the cells may become round after image restoration. To address these issues, overlapping the partial image that has not been restored with the partial image that has been restored at an appropriate ratio may facilitate subsequent decoding. The overlap ratio between the partial image that has not been restored and the partial image that has been restored may be a predetermined fixed value or may be a non-fixed value. For example, the overlap ratio may be dynamically determined during reading or optimized through pre-evaluation (tuning) based on the stability of restoration by the image restoration unit 20j. For example, in pre-evaluation, the ratio may be adjustable by the user using a scale bar or the like, allowing the user to adjust the ratio while the combined image is presented to the user.

(Bank switching function)
In this embodiment, the parameter set storage unit 30c is configured to store parameter sets, each of which includes, for example, parameters constituting the imaging conditions of the camera 5 and parameters constituting the processing conditions in the decoding processing unit 210. The parameter sets can be called banks. A plurality of banks are provided, each storing different parameters. For example, the parameter set storage unit 30c stores a plurality of imaging conditions and code conditions, including first imaging conditions and code conditions set by the tuning execution unit 20d and second imaging conditions and code conditions, as separate parameter sets.

This optical information reading device 1 is configured to enable switching from one parameter set including first imaging conditions and code conditions to another parameter set including second imaging conditions and code conditions, or vice versa, among the multiple parameter sets stored in the parameter set storage unit 35c. Parameter set switching can be performed by the processor 20, the user, or by a switching signal from an external control device such as the PLC 101. When switching parameter sets, the user simply operates a parameter set switching unit incorporated in the user interface, for example. By enabling the parameter set switching unit, the parameter set of that bank is used when the optical information reading device 1 is in operation, and by disabling the parameter set switching unit, the parameter set of that bank is not used when the optical information reading device 1 is in operation. In other words, the parameter set switching unit is used to switch from one parameter set to another.

In this example, when the decoding processing unit 210 is operating using one parameter set and fails the second decoding process on the restored image, it switches to another parameter set and then performs image restoration and second decoding on a read image captured under different conditions. For example, when the decoding processing unit 210 detects that the second decoding process has failed, it switches to a parameter set other than the current parameter set and then performs image restoration and second decoding. If it detects that the second decoding process has also failed with this parameter set, it switches to a parameter set that has not been used before and then performs image restoration and second decoding. Switching parameter sets may change not only the imaging conditions of the camera 5 but also the code size to be read. In this case, the size used to extract code candidate areas where a code is likely to exist also changes. As a result, the reduction or enlargement ratio of the image input to the neural network also changes, and the results of image restoration also change.

(Black and white inversion supported)
Next, we will explain how to deal with black-and-white inversion of codes with reference to Figures 21 and 22. Figure 21 is a diagram illustrating how to deal with black-and-white inversion of codes using two neural networks, namely, a neural network trained with a code printed in black on a white background and a neural network trained with a code printed in white on a black background. Figure 21A shows a method in which a scanned image (the image at the left end) is input to a neural network trained with a code printed in black on a white background and a neural network trained with a code printed in white on a black background, and the decoded results are output from each neural network. This method is called a parallel execution method. Since the scanned image is an image of a code printed in black on a white background, decoding is successful when the neural network trained with the code printed in black on a white background repairs the image, but decoding is unsuccessful when the neural network trained with the code printed in white on a black background repairs the image.

FIG. 21B shows that the scanned image (the image on the far left) is first input only to a neural network trained on a code printed in white on a black background, and an attempt is made to decode the restored image. Only if the decoding process fails is the scanned image input to a neural network trained on a code printed in black on a white background to generate a restored image. This method uses a different network after a failure.

FIG. 21C shows a method for selecting a neural network to input a scanned image based on user settings. The case shown on the left is when the scanned image is set to be an image of a code printed in black on a white background. In this case, the network selection module selects a neural network trained with a code printed in black on a white background. The scanned image (the image at the left) is then input to the neural network trained with a code printed in black on a white background. On the other hand, the case shown on the right is when the scanned image is set to be an image of a code printed in white on a black background. In this case, the network selection module selects a neural network trained with a code printed in white on a black background. The scanned image (the image at the left) is then input to the neural network trained with a code printed in white on a black background.

Figure 22 is a diagram illustrating how to handle black-and-white inversion of a code using only one neural network. FIG. 22A shows how a black-and-white inverted image is generated by inverting the scanned image (the image on the left). An image without black-and-white inversion and an image with black-and-white inversion are each input to a neural network trained with a code printed in black on a white background. Because the scanned image is an image of a code printed in white on a black background, decoding fails when using a restored image generated by a neural network trained with a code printed in black on a white background. However, decoding is successful when using the image after black-and-white inversion.

In the example shown at the top of FIG. 22B, the scanned image (the image on the left) is an image of a code printed in white on a black background, so when it is input into a neural network trained with a code printed in black on a white background, the decoding process fails. If an image with inverted black and white is then generated and input into a neural network trained with a code printed in black on a white background, the decoding process is successful.

In the example shown at the bottom of FIG. 22B, the scanned image (the image on the left) is an image of a code printed in black on a white background, so when it is input into a neural network trained with a code printed in white on a black background, the decoding process fails. If an image with inverted black and white is then generated and input into a neural network trained with a code printed in white on a black background, the decoding process is successful.

FIG. 22C shows a method for determining whether to perform black-and-white inversion processing based on user settings. The case shown at the top is when the scanned image is set to be an image of a code printed in black on a white background. In this case, the black-and-white inversion processing module does not perform black-and-white inversion processing, but instead inputs the image directly into a neural network trained with a code printed in black on a white background. On the other hand, the case shown at the bottom is when the scanned image is set to be an image of a code printed in white on a black background. In this case, the black-and-white inversion processing module performs black-and-white inversion processing. The inverted image is then input into a neural network trained with a code printed in black on a white background.

(Relationship between contrast and matching level)
The matching level is used, for example, as a score during tuning, or to manage print quality/reading performance during operation. For example, the matching level is defined as a positive integer value between 0 and 100, and a level of 50 or higher can be designed to achieve a 100% read rate when a read rate test is conducted. Also, a matching level of 0 means that the code cannot be read, and a matching level of 1 is the lowest value when the code can be read.

Figure 23 is a graph showing the relationship between contrast and matching level, with the solid line showing the relationship between contrast and matching level of an image (restored image) that has been restored by the image restoration unit 20j, and the dashed line showing the relationship between contrast and matching level of an image (read image) that has not been restored by the image restoration unit 20j. As is clear from this graph, the matching level of the restored image is generally higher than the matching level of the read image, but once the image quality deteriorates beyond a certain level, the matching level drops sharply.

(Matching level calculation process)
A matching level is set for each algorithm, and typically one algorithm is used for each code type. However, in this embodiment, an image restoration function is implemented, and two algorithms (one with image restoration and one without image restoration) are executed for each code type. The result of the algorithm that first successfully decodes the code is output. Therefore, if the two algorithms alternately succeed in decoding, the matching level value may become unstable. Furthermore, because image restoration results in an image that is easier to read, conventional methods of calculating the matching level tend to result in a higher matching level than existing methods. In response to these issues, this embodiment calculates a weighted sum of the matching levels of the two algorithms and adjusts the matching level value. A specific example will now be described based on the flowchart shown in FIG. 24.

After starting, in step SF1, the read image is input to the decoding processing unit 210. In step SF2, the decoding processing unit 210 performs the first decoding process without image restoration. In step SF3, the matching level A (MLA) from the first decoding process is calculated. Meanwhile, in step SF4, the read image is input to the image restoration unit 20j and image restoration process is performed. Then, proceeding to step SF5, the decoding processing unit 210 performs the second decoding process on the restored image. In step SF6, the matching level B (MLB) from the second decoding process is calculated. Note that if decoding is not successful in step SF2, MLA = 0, and if decoding is not successful in step SF5, MLB = 0.

In step SF7, the MLA and MLB values are adjusted using, for example, an averaging function. Furthermore, regardless of whether decoding is successful in step SF2 or SF5, a process is performed to weight the MLA value.

(Example of user interface)
25 is a diagram showing an example of a user interface 300 displayed on the display unit 6. The processor 20 generates the user interface 300 and displays it on the display unit 6. The user interface 300 includes a first image display area 301 in which an image currently captured by the camera 5 (a live view image) is displayed, a second image display area 302 in which a read image is displayed, and a third display area 303 in which a restored image restored by the image restoration unit 20j is displayed. This allows the read image and the restored image to be presented to the user in a form that allows them to compare them. Furthermore, the user can understand what the image to be decoded or what the image will look like after the decoding process has been performed.

(Example of an image sensor with an AI chip)
26A and 26B are diagrams showing examples using an imaging element 5a with an AI chip, and the imaging element 5a of the camera 5 can be configured as shown in each diagram. FIG. 26A shows a packaged imaging element 5a equipped with an AI chip that performs image restoration, whereby the scanned image is restored and then output to the processor 20. FIG. 26B shows a packaged imaging element 5a equipped with an AI chip that extracts a repair code area and performs image restoration, whereby the scanned image is extracted, a partial image corresponding to that area is restored, and then output to the processor 20. FIG. 26C shows a packaged imaging element 5a equipped with an AI chip that extracts a repair code area, whereby the scanned image is extracted, a partial image corresponding to that area is restored, and then output to the processor 20.

In the example shown in FIG. 26A, the scanned image acquired by the image sensor 5a may be output to the processor 20, and the image restoration may be performed by an AI chip, and the restored image may be output to the processor 20. The same applies to the examples shown in FIG. 26B and FIG. 26C.

(Polarizing filter attachment 3)
In this embodiment, as shown in Fig. 2, a first illumination unit 4a and a second illumination unit 4b are provided, each consisting of a plurality of light-emitting diodes as a light source that generates illumination light for illuminating at least the code. The polarizing filter attachment 3 includes a first polarizing plate 3a that transmits light of a first polarization component of the light generated by the light-emitting diodes that constitute the second illumination unit 4b, and a second polarizing plate 3b that transmits light of a second polarization component that is substantially perpendicular to the first polarization component. The first polarizing plate 3a and the second polarizing plate 3b can be provided in the shaded areas in Fig. 2. For example, the first polarizing plate 3a is arranged to cover the second illumination unit 4b, and the second polarizing plate 3b is arranged to cover the optical system 5b of the camera 5 from the light incident side.

The camera 5 therefore receives light that passes through the first polarizer 3a and is reflected from the surface (code) of the workpiece W via the second polarizer 3b. This removes the specular reflection component of the workpiece W, and the camera 5 generates a read image with lower contrast than when the light does not pass through the first polarizer 3a and second polarizer 3b. This low-contrast read image is acquired by the processor 20. The processor 20 inputs the low-contrast read image into a neural network, converting it into a restored image with higher contrast than before input, and performs a decoding process on the restored image.

In other words, by using polarizing plates 3a and 3b, the specular reflection component of the workpiece W is removed, allowing for a read image with reduced influence from the specular reflection component to be obtained. For example, polarizing plates 3a and 3b are suitable for images with a high specular reflection component, such as when capturing an image of a metal workpiece. However, reduced light intensity can cause the read image to become dark and reduce contrast. In such cases, reading performance can be improved by converting the image into a high-contrast restored image using neural network inference processing.

The above-described embodiments are merely illustrative in all respects and should not be interpreted as limiting. Furthermore, all modifications and variations that fall within the scope of the claims are within the scope of the present invention.

As described above, the optical information reading device of the present invention can be used, for example, to read codes attached to workpieces.

1 Stationary optical information reader 1A Handheld optical information reader 2 Housing 2A Handheld housing 2B Grip unit 5 Camera 20 Processor 20c Filter processing unit 20d Tuning execution unit 20e Setting unit 20f Decode processing unit 20g Extraction unit 20h Reduction unit 20i Enlargement unit 20j Image restoration unit 30 Memory unit 30d Neural network memory unit W Work

Claims (16)

An optical information reading device that reads a code attached to a workpiece,
a camera that photographs the code and generates a scanned image;
a storage unit that stores the structure and parameters of a neural network that has been generated in advance by machine learning a plurality of defective images having portions that are inappropriate for reading a code and a plurality of ideal images that respectively correspond to the plurality of defective images;
an optical information reading device comprising: a processor that extracts a code candidate area in which a code is likely to exist from the read image generated by the camera; and inputs a partial image corresponding to the extracted code candidate area into a neural network configured with the structure and parameters stored in the storage unit, thereby attempting to repair the partial image in accordance with the structure and parameters stored in the storage unit; and that executes a first decoding process on the read image and a second decoding process on the repaired partial image. 2. The optical information reading device according to claim 1,
The processor searches for a code in the scanned image generated by the camera based on information for identifying the code, and extracts an area containing the code as the code candidate area. 3. The optical information reading device according to claim 1,
a light irradiation unit for irradiating a visible light of a color different from ambient light into a photographing field of view of the camera to form a mark;
The processor identifies a portion of the scanned image generated by the camera that corresponds to a mark formed by the light irradiation unit, and extracts the identified portion as the code candidate area. 4. The optical information reading device according to claim 3,
The processor identifies a central portion of the scanned image generated by the camera and extracts the central portion as the code candidate area. 5. The optical information reading device according to claim 1,
The processor is configured to accept a user's designation of a specific portion from within the scanned image generated by the camera, and extract the designated portion as the code candidate area. 6. The optical information reading device according to claim 1,
The processor sets various conditions when setting up the optical information reading device so that the various conditions are suitable for the first decoding process , and also acquires the size of the code contained in the read image generated by the camera , and determines the input size of the partial image to the neural network based on the acquired code size. 7. The optical information reading device according to claim 6,
An optical information reading device in which the processor , when setting up the optical information reading device, obtains the size of the code based on an image generated by photographing the code to be decoded with the camera. 8. The optical information reading device according to claim 6 or 7,
The processor increases the input size of the partial image to the neural network by a predetermined amount greater than the size of the code included in the read image . 9. The optical information reading device according to claim 6,
An optical information reading device in which the processor determines the input size of the partial image to the neural network based on the number of pixels in one module that makes up the code contained in the read image and the number of modules arranged vertically or horizontally in the code. 10. The optical information reading device according to claim 1,
An optical information reading device in which an inference process for generating a restored image by repairing the partial image and the first decoding process are executed in parallel by the processor, and at least a portion of the period during which the processor is executing the second decoding process has a pause period during which the inference process is not executed. 10. The optical information reading device according to claim 1,
The processor receives a setting as to whether or not to execute the second decoding process, and switches between executing and not executing the second decoding process in accordance with the setting . 10. The optical information reading device according to claim 1,
The processor inputs a partial image corresponding to the code candidate area into the neural network, performs inference processing to generate a restored image by restoring the partial image, determines grid positions indicating the positions of each cell of the code based on the restored image, and decodes the restored image based on the determined grid positions. 10. The optical information reading device according to claim 1,
The processor repeats the imaging and decoding processes by changing the imaging conditions of the camera and the decoding conditions of the first decoding process and the second decoding process, determines optimal imaging conditions and decoding conditions based on a matching level indicating the ease of reading the code calculated under each imaging condition and decoding condition, and performs a tuning process to set the size of the code to be read, extracts a code candidate area within the read image that is large enough to contain the code to be read set by the tuning process and that is likely to contain the code, reduces or enlarges the partial image corresponding to the extracted code candidate area to a predetermined size and then inputs it to the neural network, and performs the second decoding process on the repaired image repaired by the neural network. 10. The optical information reading device according to claim 1,
a plurality of light sources that generate illumination light for illuminating at least the code;
a first polarizing plate that transmits light of a first polarization component out of the light generated by the light source;
a second polarizing plate that transmits light of a second polarized component that is substantially orthogonal to the first polarized component;
The camera receives light that passes through the first polarizing plate and is reflected from the code via the second polarizing plate, and removes the regular reflection component of the work, thereby generating a read image with lower contrast than when the first polarizing plate and the second polarizing plate are not used,
The processor inputs the low-contrast read image into the neural network, converts it into a restored image with higher contrast than before input, and performs the second decoding process on the restored image. 10. The optical information reading device according to claim 1,
the processor is capable of changing the imaging conditions of the camera, repeating imaging and decoding processes, and setting a plurality of imaging conditions and code conditions for the code to be read based on a matching level that indicates the ease of reading the code calculated under each imaging condition;
The memory unit stores a plurality of imaging conditions and code conditions including first imaging conditions and code conditions and second imaging conditions and code conditions set by the processor , and an optical information reading device inputs a read image generated under the first imaging conditions and code conditions of the plurality of reading conditions stored in the memory unit to the neural network to generate a first restored image, and if the second decoding process on the first restored image fails, inputs a read image generated under the second imaging conditions and code conditions to the neural network to generate a second restored image, and performs the second decoding process on the second restored image. 16. The optical information reader according to claim 1,
The optical information reader is configured such that, when decoding is successful in either the first decoding process or the second decoding process, the processor outputs the result of the decoding.