JP7702211B2 - Meter reading device, meter reading method, and program - Google Patents

Description

Embodiments of the present invention relate to a meter reading device, a meter reading method, and a program.

Meter values must be read periodically to maintain and operate various devices installed in machine rooms in factories, buildings, and facilities, and to calculate usage fees. Traditionally, for this type of work, inspection workers have recorded the results of visual readings on paper forms and tallied the data.

In recent years, services have been provided that use tablets to assist with equipment inspections. For example, there is a known service that assists with the reading of meters, which indicate the status of equipment. In this type of service, the meter value is read using software that photographs the meter, processes the resulting image data, and converts it into a numerical value. By not only photographing the meter value but also reading the value from the photographed image using a computer, human reading errors can be prevented. In addition, by saving the read value as a numerical value, entry errors can be prevented, record accuracy can be ensured, and convenience can be improved, such as by graphing.

Patent No. 6672673

The way meter values are displayed varies from meter to meter. Furthermore, there is a wide variety of meters themselves, such as analog meters, digital meters, square meters, and round meters. In order to accommodate this diversity, there is a demand for improvements in the ease of reading the information displayed on the meter and in the accuracy of reading.

The objective is to provide a meter reading device, a meter reading method, and a program that improves workability and reading accuracy.

According to an embodiment, a meter reading device includes a user interface, a camera that captures an image of a meter having a numerical display frame to acquire multiple image frame data, a memory having a buffer area, and a processor that processes the image frame data. The processor includes an image processing unit, an input control unit, and an output control unit. The image processing unit recognizes characters within the display frame for each of the multiple image frame data and generates a reading result. The input control unit temporarily stores the reading result for each image frame data in the buffer area. The output control unit compares the reading results stored in the buffer area, and outputs a numerical value corresponding to a reading result that has been generated more than a predetermined threshold value to the user interface.

FIG. 1 is a diagram illustrating an example of a cloud system that provides a facility inspection support service according to an embodiment. FIG. 2 is a functional block diagram illustrating an example of the mobile terminal 100. FIG. 3 is a diagram showing an example of a display on the display unit 103 of the mobile terminal 100. As shown in FIG. FIG. 4 is a diagram for explaining the method of counting the number of times a character group completely matches. FIG. 5 is a diagram for explaining the application of the method of counting the number of times a character group completely matches to a number of meters. FIG. 6 is a diagram for explaining a method of selecting the most frequent value for each digit and combining the digits to obtain the read value. FIG. 7 is a flowchart showing an example of a process for reading a numerical value. FIG. 8 is a flowchart showing another example of the process procedure for reading a numerical value. FIG. 9 is a flowchart showing another example of the process procedure for reading a numerical value. FIG. 10 is a flowchart illustrating an example of a processing procedure in the reading process. FIG. 11 is a diagram for explaining reading using a plurality of image frame data. FIG. 12 is a block diagram illustrating an example of a computer capable of implementing the functionality of a meter reading device.

Hereinafter, an embodiment will be described with reference to the drawings.
(composition)
1 is a diagram illustrating an example of a cloud system that provides an equipment inspection support service according to an embodiment. In the following, an example of application to meter reading work used in equipment in a factory or business, or various measuring instruments, etc. will be described.

The cloud system shown in FIG. 1 is a system whose core is a cloud server 13 connected to the Internet 14. A mobile terminal 100 and a site PC 11 can access the cloud server 13 via the Internet 14.

The site PC 11 is a desktop or laptop personal computer. The site PC 11 is installed, for example, in the management department of a meter reading service vendor and is operated by an operator in that department. The site PC 11 has the function of acquiring data files uploaded from the mobile terminal 100 to the cloud server 13, and performing functions such as referencing, viewing, editing, tallying, data conversion/file creation, and report issuing. The acquired data files can also be saved in the external media 12.

The cloud server 13 provides services such as storage, management, and data processing of data uploaded from the mobile terminal 100 and the site PC 11, and data sharing between users. The base station 200 is connected to the Internet 14 via a gateway (GW) 400, and connects the mobile terminal 100 and the Internet 14 so that they can communicate with each other.

The mobile terminal 100 is a smartphone, tablet, or laptop computer, etc., and is carried by an inspection worker who performs meter reading work. The mobile terminal 100 is used with a report creation support app according to the embodiment installed. The report creation support app includes, for example, a meter reading application. The mobile terminal 100 is equipped with a camera and optically reads the measurement values of meters MT1 to MTn installed at the site. The report creation support app creates report data from the read measurement values and transmits it to the cloud server 13.

Meters MT1 to MTn are related to various facilities and measuring instruments, and display in a display frame measurements such as current, voltage, gas or liquid pressure and flow rate, and values indicating the operating status of facilities and equipment. Meters MT1 to MTn are of two types: numeric meters and analog meters, and inspection workers may read a mixture of both types.

Numeric meters are rotary or digital, and visually display measured values, etc. Digital numeric meters, for example, use multiple seven-segment displays showing the numbers from 0 to 9 to show values. Rotary numeric meters, for example, have a mechanism with multiple number wheels with the Arabic numerals from 0 to 9 written on them lined up in a row, and show values by rotating the number wheels.

An analog meter shows values by the position of a pointer that moves across a dial marked with Arabic numerals and scales. In other words, measurements and other numerical values are shown in an analog manner by the relative positions of the pointer and the numbers or scales.

Figure 2 is a functional block diagram showing an example of a mobile terminal 100. The mobile terminal 100, which is an example of a meter reading device, includes a communication unit 101, an input unit 102, a display unit 103, a camera 104, a GNSS (Global Navigation Satellite System) receiving unit 105, storage 106, memory 107, and a processor 110. In other words, the mobile terminal is a computer that includes a processor and a memory.

The processor 110 controls the internal operations of the mobile terminal 100 and also controls communications with external devices of the meter reading device.
The communication unit 101 is a wireless communication interface, which establishes a wireless communication link with the base station 200 and communicates with the cloud server 13 via the Internet 14. The communication unit 101 also performs interface processing required for data transmission and reception, such as adding/removing a header portion of a packet.

The input unit 102 includes input devices such as a touch panel of the display unit 103 and key switches provided on the exterior of the mobile terminal 100, and receives input operations and instructions from an inspection worker.
The display unit 103 is a user interface, and forms a GUI (Graphical User Interface) environment on a display device such as a touch panel to provide visual information to an inspection worker. The touch panel accepts information input by a stylus, a finger, or the like. There are various types of touch panels, such as a capacitive type, a resistive film type, and a projected infrared type. Examples of display devices include a liquid crystal panel, an organic EL (Electro Luminescence) panel, and electronic paper.

The display unit 103 displays, for example, information input fields, soft keys, and various images (photos, CG (computer graphics) images) on a display device. The display of the information input fields and soft keys on the display unit 103 is controlled by the processor 110 so as to correspond to operations on the input unit 102.

Camera 104 is a digital camera that generates digital image data. Camera 104 captures, for example, meters MT1-MTn (Figure 1) to obtain image data including data such as images of displayed numbers. The image data includes multiple image frame data. When the user holds mobile terminal 100 over (closes) meter MT1-MTn that he or she wishes to capture, and a preset time has elapsed with camera 104 facing the meter, processor 110 activates camera 104. This starts the acquisition of image frame data.

The camera 104 comprises an optical system including a lens, an imaging section equipped with an image sensor such as a CMOS (Complementary MOS), and a signal processing section that generates image data in a predetermined format (e.g., JPEG (Joint Photographic Experts Group)) from an imaging signal from the imaging section. The image frame data is processed by the processor 110. The processor 110 may add metadata (e.g., Exif (Exchangeable image file format) data) indicating the date and time of shooting, the location, and the shooting conditions (shutter speed, aperture value, angle of view, latitude, longitude, etc.) to the image data.

When the mobile terminal 100 is brought close to each meter MT1 to MTn, the camera 104 is turned on and imaging begins. The image data acquired by imaging and the results of reading the image data are temporarily stored in the buffer area 5 of the memory 107. The capacity of the buffer area needs to be large enough to hold, for example, the results of reading a maximum of X times, and after X times, the old information is overwritten and updated.

The processor 110 processes the image data captured by the camera 104 and extracts characters such as numbers. For example, AI (Artificial Intelligence) can be used to read the numbers.

The read result data processed by the processor 110 is stored in the buffer 5. The captured image and read result data are stored in the storage 106. Alternatively, they may be stored in the external media 12 or the cloud server 13. When using these storage destinations, the data may be compressed to conserve network resources, or the storage process itself may be performed periodically as a sort of batch process.

The GNSS receiver 105 receives positioning signals transmitted from multiple GNSS satellites, such as GPS (Global Positioning System) satellites, and from terrestrial reference stations, and measures the position of the mobile terminal 100. The positioning signals also contain time information, which can be used to find out the current time.

Storage 106 is a recording device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory such as an SSD (Solid State Drive), or a HDD (Hard Disk Drive). Different types of recording devices may be combined according to the characteristics of the data to be handled.

The storage 106 stores, for example, data generated in accordance with the operation of the processor 110, various parameters, data input by inspection workers, image data acquired by the camera 104 (for example, image data capturing the measurement values of meters MT1 to MTn), data acquired (downloaded) from the cloud server 13, or temporary data for information processing.
In an embodiment, the storage 106 stores report data 106a, setting information data 106b, image data 106c, a trained model 106d, and a program 106e.

The report data 106a is created by aggregating the meter readings of each meter for each specified meter reading period. When new report data 106a is created, it is immediately uploaded to the cloud server 13. The report data 106a is data created for each meter reading site, for example, and includes information such as "meter information," "meter image," "current meter reading value," "difference from previous time," "previous meter reading value," and "difference calculated last time." If multiple meters are installed at the meter reading site, the report data 106a includes the above information for each meter.

Here, "meter information" includes a master code for identifying the meter, identification information indicating what the meter measures (e.g., tenant, parking lot, shared space, etc.), and type (e.g., gas, electricity, water, etc.). "Meter image" is the image that is the basis for the meter reading, i.e., the image captured at the time of meter reading.

"Current meter reading value" is the value read in this meter reading. "Difference from previous" is the difference between the current meter reading value and the previous meter reading value. "Previous meter reading value" is the value read in the previous meter reading. "Difference calculated from previous reading" is the difference between the previous meter reading value and the value before that.

The setting information data 106b is setting information required to aggregate the meter readings of each meter. Like the report data 106a, the setting information data 106b is created for each meter at each meter reading site.

Image data 106c is image data generated by photographing the meter showing the meter reading with camera 104, and is recorded in the above-mentioned report data 106a as a meter image for evidence. At that time, only the necessary part including the numerical value may be selected and cut out from the image immediately after it is photographed.

The trained model 106d is generated by machine learning a neural network such as a deep neural network (DNN) or a convolutional neural network (CNN). For example, the trained model 106d can be generated by repeatedly inputting numeric image data including an image of a numeric value and training data including the gradient of the numeric value shown in the numeric image data into the neural network, and minimizing the error between the output gradient and the correct value using a gradient descent method or the like.

Storage 106's 106e includes an operating system (OS), firmware, application software, etc., and causes the mobile terminal 100 to execute various functions related to the embodiment.

The processor 110 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a chipset, RAM, ROM, etc., and controls each part of the mobile terminal 100. The ROM stores firmware and setting values (various parameters), etc. The processor 110 loads the program 106e from the storage 106 into the RAM, and also uses the RAM as a work area to realize various functions.

The term "processor" used in the embodiments herein refers to, for example, a CPU, a GPU (Graphical Processing Unit), or an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)).

The processor 110 realizes each of the above-mentioned parts by reading and executing the program 106e stored in the storage 106. Note that instead of storing the program in the storage 106, the program may be directly built into the circuitry of the processor 110. In this case, the processor 110 realizes each of the above-mentioned parts by reading and executing the program built into the circuitry.

The processor 110 executes the instructions included in the program 106e to realize at least the following functions. That is, the processor 110 functions as a communication control unit 111, an input control unit 112, a display control unit 113, an image processing unit 114, a location information processing unit 115, and a meter reading data compilation processing unit 116. Here, the program 106e includes instructions for causing the mobile terminal 100, which is a computer, to function as the storage 106, the communication control unit 111, the input control unit 112, the display control unit 113, the image processing unit 114, the location information processing unit 115, and the meter reading data compilation processing unit 116.

The communication control unit 111 controls communication between the mobile terminal 100 and the base station 200, the Internet 14, and the cloud server 13 according to a predetermined communication protocol. The communication control unit 111, for example, downloads files and data from the cloud server 13 and uploads data to the cloud server 13.

The image processing unit 114 performs image processing on the image data captured by the camera 104. That is, the image processing unit 114 recognizes the characters within the display frame for each of multiple image frame data and generates a reading result. The image processing unit 114 has, for example, a function as a recognition engine and a function as an analysis engine.

The recognition engine recognizes various information from images based on image data acquired by the camera 104, for example by applying pattern recognition with AI. The recognition engine has functions such as detecting QR codes (registered trademark), distinguishing (determining) whether a meter shown in an image is a numeric meter or an analog meter, detecting the part of the meter's display panel that displays the measured value, and detecting objects within the display panel (such as the numeric string of measured values on a numeric meter, the pointer or written numbers on an analog meter) and their arrangement (coordinates).

The analysis engine performs processing to analyze the information detected by the recognition engine. For example, the analysis engine performs processing to decode QR codes (registered trademark) on the image of the meter, perform character recognition processing such as optical character recognition (OCR: Optical Character Recognition), or convert the measured value of a numeric meter into text using AI, or detects the value indicated by the pointer from the relative positions of objects marked on the meter part of an analog meter.

Furthermore, if the character spacing of a readable character string within the display frame is equal to or less than a predetermined spacing, the image processing unit 114 recognizes the character string as the same character group. The image processing unit 114 also recognizes the characters within the display frame for each digit and generates a reading result for each digit.

When the character spacing is below a threshold and the characters are recognized as a single character group, for example, when all five digits of 12345 match in the captured image, the number of occurrences is incremented. This method makes it possible to confirm that the image is captured in a stable state without camera shake, and to read the correct value.

In addition, under conditions where the character spacing is below a threshold and the character group is recognized as one character group, for example, the characters with the highest occurrence count or frequency for each character in 12345 are combined to generate a new character group. In this way, it is possible to confirm that the image is in a stable imaging state and to read the correct value.

For character recognition processing, multiple types of analysis engines are prepared in advance. Each analysis engine corresponds to a different type of meter, for example, and converts the displayed measurement value into text using an analysis algorithm that matches the numbers displayed on the corresponding meter. In other words, since meters have different fonts for Arabic numerals and different combinations of numbers and background colors, analysis engines with analysis algorithms that match the display of each meter are prepared.

The input control unit 112 interprets requests from the inspection worker from operations on the input unit 102, accepts input of information (such as character strings), data import, and changes to correspondences between data. The input control unit 112 also temporarily stores the reading results for each image frame data in the buffer area 5 of the memory 107, or temporarily saves the results in the storage 106. Alternatively, in addition to the reading results, the input control unit 112 may temporarily store multiple image frame data in the buffer area 5. The input control unit 112 acquires image data generated by the camera 104, transfers it to the storage 106, and stores it as image data for storage 106c.

The display control unit 113 as an output control unit displays various information including characters and images, various icons, information input fields, soft keys, etc. on the display unit 103. The display control unit 113 also displays a guide indicating the image capture position. Furthermore, the display control unit 113 compares the read results stored in the buffer area 5, and outputs to the display unit 103 a numerical value corresponding to a read result whose generation count exceeds a predetermined threshold value.
Here, the number of times the read result is generated may be interpreted as the generation frequency, that is, the threshold determination may be performed based on the number of times the read result is generated per unit time.

In addition, the display control unit 113 outputs image frame data corresponding to the reading result for which the number of times generated has exceeded the threshold value to the display unit 103. In addition, the display control unit 113 outputs image frame data captured after the number of times generated has exceeded the threshold value to the display unit 103.

The display control unit 113 also notifies a user, such as an inspection worker, from the display unit 103 that the number of times the number of times has been generated has exceeded a threshold. The display control unit 113 also notifies a user, such as an inspection worker, from the display unit 103 that the number of times ...

The display control unit 113 also counts the read results of character strings that are recognized as the same character group by the image processing unit 114, and sets the count value as the number of times the read results have been generated. The display control unit 113 also compares the read results by the image processing unit 114 for each digit, combines the read results for which the number of times they have been generated exceeds a predetermined threshold for each digit, and outputs the resulting numerical value to the display unit 103.

The display control unit 113 reads image frame data corresponding to the read result whose generation count exceeds the threshold from the buffer area 5 or the storage 106, and displays it on the display unit 103.

Furthermore, the display control unit 113 displays on the display unit 103, in the user interface, a field for displaying image data of the display frame, a field for displaying a numerical value corresponding to the read result, and an operation unit for correcting the numerical value displayed in the field for displaying the numerical value corresponding to the read result.

The position information processing unit 115 is equipped with an acceleration sensor, a gyro sensor, etc., and determines the position of the mobile terminal 100 based on the detection results of these sensors, the positioning results from the GNSS receiving unit 105, and/or the positioning information obtained from the base station 200 by the communication control unit 111.
The meter reading data compilation processing unit 116 comprehensively controls various processes relating to the compilation of meter reading data.

Figure 3 is a diagram showing an example of the display of the display unit 103 of the mobile terminal 100. The display unit 103 displays, for example, a meter reading input screen as shown in Figure 3. The meter reading input screen includes a "photo of current month's meter reading" area 61, an "input field for current month's meter reading" area 62, a "voice input result of current month's meter reading" area 63, and a voice input start icon 64.

The "Photo of the current month's meter reading" area 61 corresponds to the field that displays image data of the display frame. The "Input field for the current month's meter reading" area 62 corresponds to the field that displays the numerical value corresponding to the reading result. Furthermore, the "Input field for the current month's meter reading" area 62 also corresponds to an operation unit for correcting the numerical value displayed in the field that displays the numerical value corresponding to the reading result.

Area 61 displays an image of the meter reading obtained by photographing the meter to be read by camera 104. Area 62 is a field for the inspection worker to input the meter reading value. The inspection worker can input the meter reading value in two ways: by looking at the actual meters MT1 to MTn, or by looking at the image in area 61.

Area 62 displays number wheels corresponding to each digit of the numerical value. The inspection worker can rotate the number wheels up and down by operating the GUI. That is, the inspection worker inputs the meter reading value by operating the number wheels for each digit displayed in area 62.

Area 63 is an area where the results of voice input by the inspection worker reading out the meter reading values of meters MT1 to MTn are displayed. The display unit 103 displays the meter reading values resulting from voice input in area 63. The display unit 103 also displays a warning message saying "Please check the meter reading value" in warning display area 65, which is an area that displays warnings against erroneous meter readings. This enables the mobile terminal 100 to urge the inspection worker to check again whether the current meter reading value, which may have been entered incorrectly, is the correct value.

The voice input start icon 64 is an icon that the inspection worker presses when starting voice input. In response to the inspection worker pressing the voice input start icon 64, the input control unit 112 (FIG. 2) enables the voice input unit 232 of the mobile terminal 100.

As shown in FIG. 3, the camera 104 is provided, for example, on the back surface of the mobile terminal 100. In addition, the operation button 53 is provided in the center of the bottom of the display unit 103 of the mobile terminal 100. This operation button 53 may be included in the input unit 102 (FIG. 2).

(effect)
Next, the operation of the above configuration will be described.

[Method of counting the number of times a character group matches exactly]
4 is a diagram for explaining the method of counting the number of times a character group completely matches. A display frame of a numerical value is cut out from image frame data obtained by photographing a meter, and further, each digit of the number is extracted as a character by, for example, character recognition using AI. The image processing unit 114 reads each digit. Here, if the distance between characters (character spacing) is equal to or less than a predetermined threshold spacing, the image processing unit 114 recognizes the characters in the display frame as one identical character group.

In Figure 4, the number of times the number "13868" appears is counted. For example, if the same number is obtained 8 times out of 10, as in (1) to (10) (see part (a) of Figure 4), it is deemed that the threshold has been exceeded and an image for storage is automatically captured. The number "13868" is then recorded as the read value (see part (b) of Figure 4). This is the reading result in numerical units.

The image processing unit 114 extracts characters that can be read from the image and determines the distance between the characters to determine whether the characters belong to the same character string group or whether they should be treated as part of a different character string group. This determination process will be described later.

Figure 5 is a diagram for explaining the application of the method of counting the number of times a group of characters matches perfectly to multiple meters. When multiple meters are within the field of view of the camera 104, the processor 110 sets multiple buffer areas 5 in the memory 107. This is why there are two sets of (1) to (10) in the diagram (see Figures 5(a) and 5(b)). Then, the recognition results for each meter are stored in each buffer area 5.

Figure 6 is a diagram explaining the method of selecting the most frequent value for each digit and combining the digits to obtain the readout value. A numerical display frame is cut out from the image frame data obtained by photographing the meter, and the numbers for each digit are extracted as characters, for example by character recognition using AI. The number of times each digit appears is counted. For example, the value 1 is read from (1) to (10) in the leftmost digit. Therefore, the most frequent value is 1. In the digit next to it on the right, the second digit from the left, the value 3 appears 8 times and the value 8 appears twice. Therefore, the most frequent value for the readout value is 3. In this way, the most frequent values for each digit are combined to obtain "13868".

The image processing unit 114 reads each digit and counts the frequency of occurrence of the numerical value in each digit. The numerical value that has reached a predetermined number of occurrences is regarded as the final result for that digit. For example, in FIG. 6, the most frequent values for each digit are "1", "3", "8", "6", and "8", which are regarded as the values for each digit. The meter reading is the numerical value "13868" obtained by combining the most frequent values for each digit. This is regarded as the reading for each digit. This numerical value "13868" is displayed on the display unit 103.

The timing at which the shutter of the camera 104 starts may be when the user operates the operation button 53, or may be when a certain time has elapsed since the camera 104 was started, as set by a timer. In the explanation, the number of readings is set to 10, but the number can be set to any value.

Figure 7 is a flowchart showing an example of a processing procedure for reading numerical values. In Figure 7, for example, the camera 104 is started from an app installed on the mobile terminal 100 (step S10), and the camera 104 is held over a meter (step S11) until it approaches the target meter so that it can capture the image.

Next, the image processor 114 reads the characters (step S12). Next, it is determined whether or not a predetermined condition has been met (step S13). If the result is Yes, the condition has been met and a final meter image is captured by the camera 104 (step S14), and the meter image is displayed on the display unit 103 together with the read numerical value (step S14).

If the answer is No in step S13, the procedure returns to step S12. The reading process is then looped until the condition is met. Here, the condition in step S13 is, for example, whether the frequency or number of times that the read character string (number string) matches perfectly exceeds a preset threshold. When this condition is met, the reading result is displayed in numerical units or in units of digits with the most frequent value of each digit as the reading value. In the procedure in FIG. 7, the loop is repeated until the condition is met, and a case is shown in which there is no setting such as a timer that sets an upper limit on the time until shooting or shooting with a manual shutter.

Figure 8 is a flow chart showing another example of the processing procedure for reading numerical values. In Figure 8, the camera 104 is started (step S20), held over the meter (step S21), and brought close enough to capture the target meter. The image processing unit 114 reads the characters (step S22).

Next, the processor 110 determines whether the timer setting time has elapsed (step S23). If the answer is No, the processor 110 determines whether the shutter has been manually operated, that is, whether the operation button 53 has been pressed (step S24). If the answer is No in step S24, the processing procedure returns to step S22 and the character reading process is looped until the condition is met (Yes in step S25).

When the timer setting time has elapsed or the manual shutter is pressed, an interrupt causes the loop to be exited. When the loop is exited or the condition of step S25 is met, the camera 104 captures a final meter image (step S26), and the meter image is displayed on the display unit 103 together with the read numerical value (step S27).

In the processing procedure in FIG. 8, "imaging" in step S26 and "numerical output (numerical units)" in step S27 are on a single path with no branching, so the determination is always made based on a perfect match of the target character group. This processing procedure is primarily focused on real-time performance (look-ahead).

Figure 9 is a flow chart showing another example of the processing procedure for reading numerical values. In Figure 9, the camera 104 is started (step S30), held over the meter (step S31), and brought close enough to capture the target meter. The image processing unit 114 reads the characters (step S32).

Next, the processor 110 determines whether the timer setting time has elapsed since the start of reading (step S33). If the answer is No, the processor 110 determines whether the operation button 53 has been pressed (step S34). If the answer is No in step S34, the processing procedure returns to step S32 and the character reading process is looped until the condition is met (Yes in step S35).

In step S34, it is determined whether the user has manually released the camera shutter. This shows the processing procedure to be followed when the user manually releases the shutter before the mobile terminal 100 performs automatic imaging because the timer setting has elapsed or a certain condition has been met.

If the shutter is released in step S34, the answer is Yes, and "image capture" is performed in step S38. If the camera shutter is not released, the answer is No, and a judgment is made in the next step S35. In step S35, if the characters (numbers) read in step S32 exceed a specified number of occurrences or frequency, the threshold is exceeded, and the shooting conditions are met, so the process moves to step S36. If the answer is No in step S35, the process returns to step S32.

When the timer setting time has elapsed or the manual shutter is pressed, an interrupt causes the loop to be exited. When the loop is exited, the processing procedure proceeds to step S38, where the final meter image is captured by the camera 104 (step S38). This is the process performed when the imaging conditions are not met until the timer setting time has elapsed. Next, the numerical value for each digit is output and displayed on the display unit 103 (step S39). In step S39, the reading result of combining the most frequent values for each digit, as shown in [method of selecting the most frequent value for each digit and combining the results of the combination of each digit as the reading value], is output.

On the other hand, even if the condition of step S35 is met, the camera 104 captures a final meter image (step S36). The meter image is displayed on the display unit 103 together with the read numerical value (step S37).

In step S37, similar to the method of counting the number of times the character groups match perfectly, the image captured when the character groups match perfectly is displayed on the display unit 103. In step S37, the "numerical output (numerical units)" is a perfect match, and exceeding the threshold is a difficult condition, including the shooting conditions. In contrast, with the "numerical output (digit units)" in step S39, it is easy to obtain the resulting reading from the threshold value for each digit. For example, if the number of digits in the character groups increases, it is expected that it will take longer to achieve the condition for a perfect match of the "numerical output (numerical units)."

Note that in the flowcharts of Figures 7, 8, and 9, the object to be read is described as a numerical value. This is not limited to this, and other characters may be included in the object to be read, and the same processing procedure is valid.

Figure 10 is a flow chart showing an example of the processing procedure in the reading process. The details of the processing in steps S12, S22, and S32 are shown. When jumping to the reading process in Figure 10, the processor 110 identifies the display frame of the numerical value in the image and starts reading the characters within the frame (step S40). Next, the processor 110 judges whether the display frame in which the numerical value is displayed has been read (step S41). If Yes, the reading has been successful and the process moves to the next step, but if the reading has not been successful, the result in step S41 is No and the processing procedure returns to step S40.

If the answer is Yes in step S41, the processor 110 determines whether the characters were recognized as the first character group or the next character group (step S42). In this step, after reading each character, it is determined whether or not to recognize them as one character group based on the spacing between the characters. If the character spacing is narrower than the threshold, they are treated as one character group and the answer is Yes. If this is the case, the processor 110 temporarily stores the reading results of each character group in memory 107 (step S44) and the process returns to the caller.

On the other hand, if the character spacing is wider than the threshold, it is recognized as a different character group, and the answer to step S42 is No. The processor 110 recognizes it as a different character string group (step S43), and the process returns to the caller.

Figure 11 is a diagram for comparing a conventional reading method with a method using multiple image frame data. The conventional reading method is shown on the left side of the figure (see (a) of Figure 11). After capturing one image, the value is read. The contrast of one image may be changed by the image processing unit 114 to create and process multiple pseudo images. Once the camera 104 is started and characters can be read, acquisition of image frame data and storage of the reading results in the buffer area 5 begin. For example, from 10 comparison images, the digits that appear most frequently are tallied and output, and the numerical reading value is estimated.

Whereas conventional methods read a numerical value from just one image frame, the method of FIG. 11 can significantly increase the amount of image information, improving the reading rate and reading accuracy accordingly. In FIG. 11, a reading value of 13868, which is a combination of the most frequent values, is obtained. Once the reading value is obtained, the processor 110 controls the camera 104 to obtain image data for storage and display on the display unit 103. Alternatively, the display control unit 113 may read image frame data corresponding to the reading value from the buffer area 5 and output it to the display unit 103.

Figure 12 is a block diagram showing an example of a computer capable of implementing the functions of a meter reading device. In Figure 12, information processing device 20 as a general-purpose computer includes a control device such as a CPU 86, storage devices such as a ROM (Read Only Memory) 88, a RAM (Random Access Memory) 90, and a HDD (Hard Disk Drive) 92, an I/F unit 82 that interfaces with various devices, an output unit 80 that outputs various information such as output information, an input unit 94 that accepts operations by the user, and a bus 96 that connects each unit.

In the above configuration, the CPU 86 reads the program from the ROM 88 onto the RAM 90 and executes it, thereby realizing each part shown in FIG. 2 on a general-purpose computer.

The programs for executing the above processes executed by the information processing device 20 of the above embodiment may be stored in the HDD 92. Also, the programs for executing the above processes executed by the information processing device 20 of the above embodiment may be provided in advance in the ROM 88.

The program for executing the above-mentioned processes executed by the information processing device 20 may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disk), or flexible disk (FD) and provided as a computer program product. The program for executing the above-mentioned processes executed by the information processing device 20 of the above-mentioned embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The program for executing the above-mentioned processes executed by the information processing device 20 of the above-mentioned embodiment may be provided or distributed via a network such as the Internet.

(effect)
As described above, according to the embodiment, (1) the mobile terminal 100 is brought close to the meter to be read, and the camera 104 is activated. (2) The camera 104 is held toward the meter. (3) The user presses the operation button 53. (4) The user checks the displayed image and its read value. (5) The user moves to the next meter. By following these steps, the user (inspection worker) can collect meter data.

Existing methods only read numerical values from a single captured image, so accuracy varies greatly depending on the condition of the image. In contrast, in the embodiment, imaging and reading are performed in real time from the moment the camera 104 is held over the meter, making it possible to obtain multiple pieces of reading result information.

In other words, this allows for the following operability: "When camera shake has stopped and the camera is in a stable state where it can take a picture, the user is notified. Upon receiving the notification, the user can release the shutter to take a picture, or take a picture automatically. The image results and value reading results are communicated to the user by image display." As the user becomes accustomed to the system, it is also possible to achieve the following operability: "Starting from the time when it becomes possible to read the meter value from the captured image, the shutter is automatically released and a picture is taken when a set time or number of images has been taken. After that, different images are automatically taken in the same way."

By setting the camera 104 to automatically capture an image when the match rate of the characters read in the image captured by the camera 104 exceeds a threshold value or when a certain amount of time has passed since the camera 104 was started, it is possible to achieve a simple and highly accurate image capture.

That is, according to the embodiment, the reading accuracy of the meter value is improved, and an image that can be read reliably without blurring or insufficient lighting can be saved. In addition, since the camera starts to take and read the meter by simply bringing the mobile terminal close to the meter, it is possible to prevent camera shake caused by releasing the shutter when taking a picture. In addition, the reading accuracy is improved by automatically taking pictures of the same object multiple times. In addition, it is possible to check the brightness, position and angle of the terminal that can be read reliably without blurring, reflection of light on the protective glass in the meter display frame, or reflection of the photographer or the terminal, which is expected to promote the worker's proficiency.
According to the embodiment, a rhythm is created in the work, and efficient photographing and reading can be achieved, while improving both workability and reading accuracy. Furthermore, by repeating photographing at a set timing, a rhythm is created in the work, and efficient photographing and reading can be achieved.

As a result, according to the embodiment, it is possible to provide a meter reading device, a meter reading method, and a program that improves workability and reading accuracy.

Note that this invention is not limited to the above-described embodiment. For example, application examples are not limited to equipment in factories and business establishments, or various measuring instruments. For example, meters MT1 to MTn may indicate other physical quantities or states, not limited to measurements such as current, voltage, pressure or flow rate of gas or liquid, or the operating state of facilities or equipment.

It is also possible to replace the Internet 14 shown in FIG. 1 with a LAN (Local Area Network) and implement the service as an on-premise service. In this case, the base station 200 may be an access point that implements a wireless LAN (IEEE 802.11 series).

Alternatively, if the Internet 14 is a public or local mobile communication network, the base station 200 is a wireless base station for 3G, LTE (registered trademark), 4G, 5G, or the like. Also, the air interface between the mobile terminal 100 and the base station 200 in FIG. 1 is not limited to wireless LAN, 4G, 5G, or the like. In short, any wireless communication method may be applied as long as the mobile terminal 100 can be connected to the Internet 14 via a wireless link.

In addition, it is also possible to integrate some or all of the functions included in the processor 110 in FIG. 2 with other functions, or to divide each function into multiple functional blocks from a different perspective and explain it in a different way. In addition, the neural network that is the basis of the trained model 106d is not limited to CNN or DNN, and neural networks of different formats, such as SVM (Support Vector Machine), can be used as appropriate.

In addition, all or some of the functions of the mobile terminal 100 may be implemented in the site PC 11 or the cloud server 13. For example, an image acquired by the mobile terminal 100 may be transferred to the site PC 11, and image processing, character recognition, numeric recognition, and other processing may be performed by the site PC 11. The extent to which processing is shared between the edge (mobile terminal 100) and the cloud (site PC 11, cloud server 13) can be flexibly determined according to system requirements.

Furthermore, the services provided by the cloud server 13 may be provided on a subscription basis to the mobile terminal 100 or site PC 11 owned (or loaned) by the client receiving the services.

In addition, the method of inputting the meter reading value in the number wheel for each digit displayed in the "Input field for meter reading value for this month" area 62 in FIG. 3 is not limited to the method described above. For example, it may be an input method in which a pull-down menu is displayed in which a number from 0 to 9 can be selected when the number wheel for each digit is touched, or an input method in which a frame in which a number from 0 to 9 can be input is displayed in which a number from 0 to 9 can be input when the number wheel for each digit is touched. The method of inputting the meter reading value in the input field in the "Input field for meter reading value for this month" area 62 may be any of the well-known input methods for inputting numbers.

Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

5...buffer area, 11...site PC, 12...external media, 13...cloud server, 14...internet, 20...information processing device, 53...operation button, 64...voice input start icon, 65...alarm display area, 80...output section, 82...I/F section, 86...CPU, 88...ROM, 90...RAM, 94...input section, 96...bus, 100...mobile terminal, 101...communication section, 102...input section, 103...display section, 104...camera, 105...GNSS receiving section, 105...reception unit, 106...storage, 106a...document data, 106b...setting information data, 106c...image data, 106d...trained model, 106e...program, 107...memory, 110...processor, 111...communication control unit, 112...input control unit, 113...display control unit, 114...image processing unit, 115...location information processing unit, 116...meter reading data aggregation processing unit, 200...base station, 232...voice input unit, 400...gateway, LAN...wireless, MT1 to MTn...meters.

Claims (16)

A user interface;
a camera for capturing an image of a meter having a numerical display frame and acquiring a plurality of image frame data;
a memory having a buffer area;
a processor for processing the image frame data;
The processor,
an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an input control unit that temporarily stores the read result for each image frame data in the buffer area;
an output control unit that compares the read results stored in the buffer area and outputs a numerical value corresponding to a read result whose number of times generated exceeds a predetermined threshold to the user interface ;
The output control unit outputs, to the user interface, image frame data captured after the number of times of generation exceeds the threshold . A user interface;
a camera for capturing an image of a meter having a numerical display frame and acquiring a plurality of image frame data;
a memory having a buffer area;
a processor for processing the image frame data;
The processor,
an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an input control unit that temporarily stores the read result for each image frame data in the buffer area;
an output control unit that compares the read results stored in the buffer area and outputs a numerical value corresponding to a read result whose number of times generated exceeds a predetermined threshold to the user interface;
The output control unit notifies a user through the user interface that the number of times the generation has exceeded the threshold . A user interface;
a camera for capturing an image of a meter having a numerical display frame and acquiring a plurality of image frame data;
a memory having a buffer area;
a processor for processing the image frame data;
The processor,
an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an input control unit that temporarily stores the read result for each image frame data in the buffer area;
an output control unit that compares the read results stored in the buffer area and outputs a numerical value corresponding to a read result whose number of times generated exceeds a predetermined threshold to the user interface;
The output control unit notifies a user through the user interface that the number of times a numerical value recognized from the reading results stored in the buffer area has been generated exceeds a predetermined threshold . The meter reading device according to claim 1 , wherein the output control unit outputs, to the user interface, image frame data corresponding to a reading result for which the number of times generated has exceeded the threshold value . the input control unit temporarily stores the plurality of image frame data in the buffer area;
The meter reading device according to claim 4 , wherein the output control unit reads from the buffer area image frame data corresponding to a reading result for which the number of times generated has exceeded the threshold value, and outputs the image frame data to the user interface . 4. The meter reading device of claim 1, wherein the processor controls the camera to start acquiring the image frame data when a predetermined time has elapsed with the camera facing the meter . The image processing unit includes:
The meter reading device according to claim 1 , wherein if a distance between characters of a readable character string within the display frame is equal to or less than a predetermined threshold distance, the character string is recognized as being the same character group. The meter reading device according to claim 7, wherein the output control unit counts the reading results of character strings that are recognized as the same character group, and sets the number of times the characters are generated. the image processing unit recognizes the characters in the display frame for each digit and generates a reading result for each digit;
The meter reading device according to claim 1 , wherein the output control unit compares the reading results digit by digit, combines the reading results whose generation count exceeds a predetermined threshold digit by digit, and outputs a numerical value obtained by combining the read results digit by digit to the user interface. The output control unit outputs to the user interface:
A field for displaying image data of the display frame;
A field for displaying a numerical value corresponding to the reading result;
The meter reading device according to claim 1 , further comprising an operation section for correcting a numerical value displayed in a field displaying a numerical value corresponding to the read result . A method for reading a meter by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, comprising:
The processor captures an image of a meter having a numerical display frame to acquire a plurality of image frame data;
a step of the processor recognizing characters within the display frame for each of the plurality of image frame data and generating a reading result;
a step of temporarily storing a read result for each of the image frame data in the buffer area by the processor;
the processor comparing the read results stored in the buffer area and outputting to the user interface a numerical value corresponding to the read results whose generation count exceeds a predefined threshold;
and outputting to the user interface image frame data captured after the number of generations exceeds the threshold. A method for reading a meter by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, comprising:
The processor captures an image of a meter having a numerical display frame to acquire a plurality of image frame data;
a step of the processor recognizing characters within the display frame for each of the plurality of image frame data and generating a reading result;
a step of temporarily storing a read result for each of the image frame data in the buffer area by the processor;
the processor comparing the read results stored in the buffer area and outputting to the user interface a numerical value corresponding to the read results whose generation count exceeds a predefined threshold ;
and said processor notifying a user through said user interface that said number of generations has exceeded said threshold . A method for reading a meter by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, comprising:
The processor captures an image of a meter having a numerical display frame to acquire a plurality of image frame data;
a step of the processor recognizing characters within the display frame for each of the plurality of image frame data and generating a reading result;
a step of temporarily storing a read result for each of the image frame data in the buffer area by the processor;
the processor comparing the read results stored in the buffer area and outputting to the user interface a numerical value corresponding to the read results whose generation count exceeds a predefined threshold;
and said processor notifying a user through said user interface that the number of times a recognised numerical value has been generated from the readings stored in said buffer area exceeds a predefined threshold. A program executable by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, the program comprising:
instructions to cause the processor to capture an image of a meter having a numerical display frame with the camera to acquire a plurality of image frame data;
An instruction to cause the processor to function as an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an instruction to cause the processor to function as an input control unit that temporarily stores the read result for each of the image frame data in the buffer area;
an instruction to cause the processor to function as an output control unit that compares the read results stored in the buffer area, outputs a numerical value corresponding to the read result whose generation count exceeds a predetermined threshold to the user interface, and outputs image frame data captured after the generation count exceeds the threshold to the user interface;
Including, the program. A program executable by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, the program comprising:
instructions to cause the processor to capture an image of a meter having a numerical display frame with the camera to acquire a plurality of image frame data;
An instruction to cause the processor to function as an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an instruction to cause the processor to function as an input control unit that temporarily stores the read result for each of the image frame data in the buffer area;
and instructions for causing the processor to function as an output control unit that compares the reading results stored in the buffer area, outputs to the user interface a numerical value corresponding to the reading result whose generation count exceeds a predetermined threshold, and notifies a user from the user interface that the generation count has exceeded the threshold. A program executable by a processor of a computer having a user interface, a camera, a memory having a buffer area, and a processor, the program comprising:
instructions to cause the processor to capture an image of a meter having a numerical display frame with the camera to acquire a plurality of image frame data;
An instruction to cause the processor to function as an image processing unit that recognizes characters within the display frame for each of the plurality of image frame data and generates a read result;
an instruction to cause the processor to function as an input control unit that temporarily stores the read result for each of the image frame data in the buffer area;
and instructions for causing the processor to function as an output control unit that compares the read results stored in the buffer area, outputs to the user interface a numerical value corresponding to a read result whose number of times generated exceeds a predetermined threshold, and notifies a user from the user interface that the number of times a numerical value recognized from the read results stored in the buffer area has been generated exceeds a predetermined threshold.