JP7702208B2 - PROGRAM, INFORMATION PROCESSING APPARATUS, AND INFORMATION PROCESSING METHOD - Google Patents

Description

Embodiments of the present invention relate to a program, an information processing device, and an information processing method.

Recently, services have been provided that use tablets to assist with equipment inspections. For example, there is a known service that assists with the reading of meters, which indicate the status of equipment. In this type of service, the meter value is read using software that photographs the meter, processes the image data obtained, and converts it into a numerical value. AI (Artificial Intelligence), for example, is used to read the numerical value. Meters can be broadly divided into numeric meters, in which the meter value is directly indicated as a numerical value, and analog meters, in which the measurement value is indicated by the position of the pointer.

JP 2007-114828 A JP 2016-212044 A JP 2018-45655 A

There are various types of analog meters, including some in which the numbers on the scale are printed in a roughly concentric pattern. Because the inclination of each number on an analog meter varies, it has traditionally been difficult to read the meter using image recognition, and there has been a demand for technology to improve the accuracy of reading the meter.
Therefore, an object of the present invention is to provide a program, an information processing device, and an information processing method that improve the accuracy of reading numerical values in reading an analog meter.

According to an embodiment, the program causes the computer to function as a storage unit, a tilt information acquisition unit, a correction unit, a number recognition unit, a pointer detection unit, and a measurement value detection unit. The storage unit stores a tilt recognition model generated from learning data related to the tilt of numbers in number image data related to an image of the numbers of an analog meter. The tilt information acquisition unit acquires tilt information of each of the number image data from the tilt recognition model for multiple number image data extracted from image data obtained by photographing an analog meter. The correction unit rotates and corrects the number image data based on the tilt information. The number recognition unit recognizes the value of the number from the rotated and corrected number image data. The pointer detection unit detects the position of the tip of the pointer of the analog meter. The measurement value detection unit detects the measurement value indicated by the pointer from the positions of multiple numbers displayed on the display panel of the analog meter, the values indicated by the numbers, and the position of the tip of the pointer.

FIG. 1 is a diagram illustrating an example of a cloud system that provides a facility inspection support service according to an embodiment. FIG. 2 is an external view showing an example of a square meter. FIG. 3 is a functional block diagram illustrating an example of the mobile terminal 100. FIG. 4 is a diagram for explaining the trained model 600. FIG. 5 is a functional block diagram for explaining the functions of the image processing unit 114. As shown in FIG. FIG. 6 is a diagram for explaining the operations of the classification unit 114a, the correction unit 114b, and the written numeral detection unit 114c. FIG. 7 is a functional block diagram for explaining the functions of the meter reading data compilation processing unit 116. As shown in FIG. FIG. 8 is a functional block diagram for explaining the functions of the analog meter reading unit 116c. FIG. 9 is a flowchart showing an example of a processing procedure of the mobile terminal 100 according to the embodiment. FIG. 10 is a flowchart showing an example of a processing procedure of the mobile terminal 100 according to the embodiment. FIG. 11 is a diagram for explaining the operation of detecting a written number from an analog meter. FIG. 12 is a diagram for explaining the operation of detecting the center point from an analog meter. FIG. 13 is a diagram for explaining the operation of detecting a pointer from an analog meter. FIG. 14 is a diagram for explaining the operation of detecting a pointer from an analog meter. FIG. 15 is a diagram for explaining the operation of detecting a pointer from an analog meter. FIG. 16 is a diagram for explaining the operation of detecting the arrangement of written numbers from an analog meter. FIG. 17 is a diagram for explaining the operation of detecting the value of a notated number from an analog meter. FIG. 18 is a diagram for explaining the operation of detecting the value of a notated number from an analog meter. FIG. 19 is a diagram for explaining the operation of detecting the intersection of the pointer and a circle passing through a notated number on an analog meter. FIG. 20 is a diagram for explaining the operation of detecting the written numbers on the left and right of the intersection from the analog meter. FIG. 21 is a diagram for explaining the operation of detecting the written numbers on the left and right of the intersection from the analog meter. FIG. 22 is a diagram for explaining the operation of detecting the angle between the pointer and the left and right notation numbers on an analog meter. FIG. 23 is a diagram for explaining the operation of detecting the angle between the pointer and the left and right notated numbers on an analog meter. FIG. 24 is a diagram for explaining the operation of detecting the number of decimal places from an analog meter. FIG. 25 is a diagram for explaining the operation of detecting the number of decimal places from an analog meter. FIG. 26 is a diagram showing an example of an interface for correcting the detected index value. FIG. 27 is a diagram showing an example of a GUI for correcting the position of the decimal point of the detected pointer value. FIG. 28 is a diagram for supplementarily explaining the horizontal correction process. FIG. 29 is a diagram for supplementarily explaining the numerical value frame and the pseudo circle.

Hereinafter, an embodiment will be described with reference to the drawings.
(composition)
1 is a diagram illustrating an example of a cloud system that provides an equipment inspection support service according to an embodiment. In the following, an example of application to meter reading work used in equipment in a factory or business, or various measuring instruments, etc. will be described.

In FIG. 1, the cloud system has a server 400 connected to a network 500 at its core. A client terminal 300 can access the server 400 via the network 500. Furthermore, a mobile terminal 100 can access the server 400 via a base station 200 of the network 500.

The mobile terminal 100 is a smartphone, tablet, or laptop computer, and is carried by a meter reader who performs meter reading operations. The mobile terminal 100 is used with a report creation support app according to the embodiment installed. The mobile terminal 100 optically reads the measurement values of meters MT1 to MTn installed at the site, creates report data using the report creation support app, and transmits it to the server 400.

The base station 200 connects the mobile terminal 100 and the network 500 so that they can communicate with each other. If the network 500 is a LAN (Local Area Network), the base station 200 is an access point that implements a wireless LAN (IEEE 802.11 series). If the network 500 is a public or local mobile communication network, the base station 200 is a wireless base station for 3G, LTE (registered trademark), 4G, 5G, etc.

The client terminal 300 is an information processing device such as a desktop or laptop personal computer. The client terminal 300 is installed, for example, in the management department of a meter reading service vendor and is operated by an operator in that department. The client terminal 300 has functions such as acquiring data files uploaded from the mobile terminal 100 to the server 400, and performing functions such as referencing, viewing, editing, tallying, data conversion, file creation, and report issuing.

The server 400 provides services such as storing, managing, and processing data uploaded from the mobile terminal 100 and the client terminal 300, as well as sharing data between users.

Meters MT1 to MTn are related to various facilities and measuring instruments, and display values such as current, voltage, gas or liquid pressure and flow rate, and values that indicate the operating status of facilities and equipment. Meters MT1 to MTn are of two types: numeric meters and analog meters, and the meter readers may read a mixture of both types.

Numeric meters are rotary or digital, and visually display measured values, etc. Digital numeric meters, for example, use multiple seven-segment displays showing the numbers from 0 to 9 to show values. Rotary numeric meters, for example, have a mechanism with multiple number wheels with the Arabic numerals from 0 to 9 written on them lined up in a row, and show values by rotating the number wheels.

An analog meter shows values by the position of a pointer that moves across a dial marked with Arabic numerals and scales. In other words, measurements and other numerical values are shown in an analog manner by the relative positions of the pointer and the numbers or scales.

Figure 2 is an external view showing an example of a square meter. For example, many AC ammeters have the numbers on the scale drawn in concentric circles on the display panel. Many of these types of meters have the numbers on the scale written in concentric circles. In this case, not only are the numbers on the dial a mixture of horizontal and non-horizontal, but each number has a different inclination, making them difficult to read by machine. In addition, some scales have discontinuous or uneven widths. In the embodiment, a technology that can deal with such situations will be described. In the following, the numbers written on the display panel as the meter reading will be referred to as written numbers.

Figure 3 is a functional block diagram showing an example of a mobile terminal 100. The mobile terminal 100, which is an example of an information processing device, includes a communication unit 101, an input unit 102, a display unit 103, a camera 104, a GNSS (Global Navigation Satellite System) receiving unit 105, a storage unit 106, and a processor 110. In other words, the mobile terminal is a computer equipped with a processor and a memory.

The communication unit 101 is a wireless communication interface that establishes a wireless communication link with the base station 200 and communicates with the server 400 via the network 500.

The input unit 102 includes input devices such as a touch panel of the display unit 103 and key switches provided on the exterior of the mobile terminal 100, and receives input operations and instructions from a meter reader.
The display unit 103 includes a display device such as a touch panel, and provides visual information to the meter reader. The touch panel accepts information input by a stylus or a finger. There are various types of touch panels, such as a capacitive type, a resistive film type, and a projected infrared type. Examples of the display device include a liquid crystal panel, an organic EL (Electro Luminescence) panel, and electronic paper.

The display unit 103 displays, for example, information input fields, soft keys, and various images (photos, CG (computer graphics) images) on a display device. The display of the information input fields and soft keys on the display unit 103 is controlled by the processor 110 so as to correspond to operations on the input unit 102.

Camera 104 is a digital camera that generates digital image data. Camera 104 captures, for example, meters MT1 to MTn (Figure 1) to obtain image data including data such as images of notated numbers. That is, camera 104 has an optical system including a lens, an imaging unit equipped with an image sensor such as a CMOS (Complementary MOS), and a signal processing unit that generates image data in a predetermined format (for example, JPEG (Joint Photographic Experts Group)) from an imaging signal from the imaging unit. Processor 110 may add metadata (for example, Exif (Exchangeable image file format) data) indicating the date and time of shooting, the location, and the shooting conditions (shutter speed, aperture value, angle of view, latitude, longitude, etc.) to the image data.

The GNSS receiver 105 receives positioning signals transmitted from multiple GNSS satellites, such as GPS (Global Positioning System) satellites, and from terrestrial reference stations, and measures the position of the mobile terminal 100. The positioning signals also contain time information, which can be used to find out the current time.

The storage unit 106 is a recording device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory such as an SSD (Solid State Drive), or a HDD (Hard Disk Drive). Different types of recording devices may be combined according to the characteristics of the data to be handled.

The memory unit 106 stores, for example, data generated in response to the operation of the processor 110, various parameters, data input by a meter reader, image data acquired by the camera 104 (for example, image data capturing the measurement values of meters MT1 to MTn), data acquired (downloaded) from the server 400, or temporary data for information processing.
In an embodiment, the memory unit 106 stores form data 106a, setting information data 106b, image data 106c, a trained model 600, and a program 700.

The report data 106a is created by tallying up the readings of each meter for each specified meter reading period. When new report data 106a is created, it is immediately uploaded to the server 400. The report data 106a is data created for each meter reading site, for example, and includes information such as "meter information," "meter image," "current meter reading value," "difference from previous time," "previous meter reading value," and "difference calculated last time." If multiple meters are installed at the meter reading site, the report data 106a includes the above information for each meter.

Here, "meter information" includes a master code for identifying the meter, identification information indicating what the meter measures (e.g., tenant, parking lot, shared space, etc.), and type (e.g., gas, electricity, water, etc.). "Meter image" is the image that is the basis for the meter reading, i.e., the image captured at the time of meter reading.

"Current meter reading value" is the value read in this meter reading. "Difference from previous" is the difference between the current meter reading value and the previous meter reading value. "Previous meter reading value" is the value read in the previous meter reading. "Difference calculated from previous reading" is the difference between the previous meter reading value and the value before that.

The setting information data 106b is setting information required to aggregate the meter readings of each meter. Like the report data 106a, the setting information data 106b is created for each meter at each meter reading site.

Image data 106c is image data generated by photographing the meter showing the meter reading with camera 104, and is recorded in the above-mentioned report data 106a as a meter image for evidence. At that time, only the necessary part including the numerical value may be selected and cut out from the image immediately after it is photographed.

The trained model 600 is generated by machine learning a neural network such as a deep neural network (DNN) or a convolutional neural network (CNN). In the embodiment, the trained model 600 is generated by repeatedly inputting numerical image data including an image of a numerical value and training data including the gradient of the numerical value shown in the numerical image data into the neural network, and minimizing the error between the output gradient and the correct value using a gradient descent method or the like.

Here, the number image data is image data that has been cut out from the image data to produce the minimum necessary area. Image data that has not been cut out may be used as the number image data, but it is considered that this will not improve the learning efficiency.

Figure 4 is a diagram for explaining the trained model 600. The trained model 600 as a tilt recognition model is generated, for example, in the server 400. Then, when the form creation support app installed in the mobile terminal 100 is launched for the first time, the trained model 600 is downloaded from the server 400 and stored in the memory unit 106.

As shown in FIG. 4(a), the trained model 600 in FIG. 4(b) is generated by repeatedly providing training data, which is a set of numeric image data including a numerical value and the slope of the numerical value, to an untrained neural network and repeating the procedure of feeding back the error. For example, training data, which is a set of an image including the numerical value "3878" and a slope of 4 degrees, or training data, which is a set of an image including the numerical value "10" and a slope of 18 degrees, is prepared in advance and repeatedly provided to a CNN or DNN for training. When image data including a numerical value is separately provided to the trained model 600 obtained by such machine learning, the slope (slope information) of the image numerical value is output instead of the numerical value itself. Here, the training data may be created, for example, by the server 400 or another computer using an existing framework, or image data collected by a meter reader on-site may be used as training data.

Returning to FIG. 3, the explanation will be continued. The program 700 in the storage unit 106 includes an operating system (OS), firmware, application software, etc., and causes the mobile terminal 100 to execute various functions related to the embodiment.

The processor 110 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a chipset, RAM, ROM, etc., and controls each part of the mobile terminal 100. The ROM stores firmware and setting values (various parameters), etc. The processor 110 loads the program 700 from the storage unit 106 into the RAM, and also uses the RAM as a work area to realize various functions.

The processor 110 executes the instructions included in the program 700 to realize at least the following functions. That is, the processor 110 functions as a communication control unit 111, an input control unit 112, a display control unit 113, an image processing unit 114, a location information processing unit 115, and a meter reading data compilation processing unit 116. Here, the program 700 includes instructions for causing the mobile terminal 100, which is a computer, to function as the memory unit 106, the communication control unit 111, the input control unit 112, the display control unit 113, the image processing unit 114, the location information processing unit 115, and the meter reading data compilation processing unit 116.

The communication control unit 111 controls communication between the mobile terminal 100 and the base station 200, the network 500, and the server 400 in accordance with a predetermined communication protocol. For example, the communication control unit 111 downloads files and data from the server 400 and uploads data to the server 400.

The input control unit 112 interprets requests from meter readers based on operations on the input unit 102, accepts input of information (such as character strings), data import, and changes to correspondences between data. The input control unit 112 also acquires image data generated by the camera 104, transfers it to the storage unit 106, and stores it as image data 106c.

The display control unit 113 displays various information including text and images, various icons, information input fields, soft keys, and other GUIs (Graphical User Interfaces) on the display unit 103. The display control unit 113 also displays guides such as the image capture position.

The image processing unit 114 performs image processing on the image data captured by the camera 104. The image processing unit 114 has, for example, a function as a recognition engine and a function as an analysis engine.

The recognition engine recognizes various information from an image based on image data acquired by the camera 104, for example by pattern recognition using AI (Artificial Intelligence). The recognition engine has functions such as detecting QR codes (registered trademark), distinguishing (determining) whether a meter shown in an image is a numeric meter or an analog meter, detecting the part of the meter's display panel that displays the measured value, and detecting objects within the display panel (such as the numeric string of measured values on a numeric meter, the pointer or written numbers on an analog meter, etc.) and their arrangement (coordinates).

The analysis engine performs processing to analyze the information detected by the recognition engine. For example, the analysis engine converts the measured value of a numeric meter into text using character recognition processing such as deciphering a QR code (registered trademark) on the image of the meter or optical character recognition (OCR: Optical Character Recognition), or detects the value indicated by the pointer from the relative positions of objects marked on the meter portion of an analog meter.

For character recognition processing, multiple types of analysis engines are prepared in advance. Each analysis engine corresponds to a different type of meter, for example, and converts the displayed measurement value into text using an analysis algorithm that matches the numbers displayed on the corresponding meter. In other words, since meters have different fonts for Arabic numerals and different combinations of numbers and background colors, analysis engines with analysis algorithms that match the display of each meter are prepared.

The position information processing unit 115 is equipped with an acceleration sensor, a gyro sensor, etc., and determines the position of the mobile terminal 100 based on the detection results of these sensors, the positioning results from the GNSS receiving unit 105, and/or the positioning information obtained from the base station 200 by the communication control unit 111.
The meter reading data compilation processing unit 116 comprehensively controls various processes relating to the compilation of meter reading data.

Figure 5 is a functional block diagram for explaining the functions of the image processing unit 114. The image processing unit 114 includes a classification unit 114a, a correction unit 114b, and a written numeral detection unit 114c. That is, the program 700 includes instructions for causing the mobile terminal 100 to function as the classification unit 114a, the correction unit 114b, and the written numeral detection unit 114c.

The classification unit 114a provides image data 106c including data of written numbers to the trained model 600 and obtains gradient information of the image data 106c.
The correction unit 114b performs rotation correction on the image data 106c based on the inclination information.
The written numeral detection unit 114c detects the value and position of the written numeral based on the rotation-corrected image data 106c.

Figure 6 is a diagram for explaining the functions of the classification unit 114a, the correction unit 114b, and the written numeral detection unit 114c. The classification unit 114a provides image data 106c to the trained model 600, and obtains tilt information for each image data. For example, tilt information is obtained such that the tilt of image data including the written numeral "0.04" is 0 degrees, the tilt of image data including "45" is 52 degrees, and the tilt of image data including "0" is -45 degrees. This tilt information is input to the correction unit 114b together with each image data.

The correction unit 114b performs rotation correction on each image data 106c so that the inclination information is cancelled and the image is upright, i.e., the angle is set to 0. The image rotation correction can be performed by existing technology, for example, using open source software or libraries.

The written number detection unit 114c detects the value and position of the written number based on the rotation-corrected image data 106c. In FIG. 6, the values obtained are "0.04", "45", and "0", respectively.

Figure 7 is a functional block diagram for explaining the functions of the meter reading data compilation processing unit 116. The meter reading data compilation processing unit 116 includes a report creation unit 116a, a numeric meter reading unit 116b, an analog meter reading unit 116c, and a correction processing unit 116d.

The report creation unit 116a creates report data based on the meter reading value read by the numeric meter reading unit 116b or the analog meter reading unit 116c, or the pointer value corrected by the correction processing unit 116d.

The numeric meter reading unit 116b uses the recognition engine and analysis engine of the image processing unit 114 as necessary to read the numeric string of the measurement value displayed on the numeric meter as the pointer value based on a photographed image of the numeric meter.

The analog meter reading unit 116c uses the recognition engine and analysis engine of the image processing unit 114 as necessary to read the value indicated by the pointer on the analog meter as the pointer value based on a captured image of the analog meter.

The correction processing unit 116d accepts an instruction to correct the indicator value in response to an operation by the meter reader via the input unit 102, and corrects the indicator value read by the numeric meter reading unit 116b or the analog meter reading unit 116c.

Figure 8 is a functional block diagram for explaining the functions of the analog meter reading unit 116c. The analog meter reading unit 116c includes a needle detection unit 6a, a circle detection unit 6b, an intersection detection unit 6c, an intersection number detection unit 6d, and a measurement value detection unit 6e. That is, the program 700 includes instructions for causing the mobile terminal 100 to function as the needle detection unit 6a, the circle detection unit 6b, the intersection detection unit 6c, the intersection number detection unit 6d, and the measurement value detection unit 6e.

The needle detection unit 6a detects the position of the tip of the needle based on image data 106c obtained by photographing the analog meter.
The circle detection unit 6b detects a circle that passes through the position of the written numeral detected by the written numeral detection unit 114c and the position of the center of this circle.

The intersection detection unit 6c determines the position of the intersection between a line segment connecting the tip of the pointer detected by the pointer detection unit 6a and the center of the circle detected by the circle detection unit 6b, and the circumference of the circle detected by the circle detection unit 6b.
The intersection number detection unit 6d detects a first written number located on one circumference of the intersection and a second written number located on the other circumference of the intersection from among the written numbers detected by the written number detection unit 114c.
The measurement value detection unit 6e detects the measurement value indicated by the hand based on the position of the first written numeral, the position of the second written numeral, the position of the center of the circle, the position of the intersection, the value of the first written numeral, and the value of the second written numeral.

(effect)
Next, the operation of the cloud system having the above configuration will be described.
9 and 10 are flowcharts showing an example of a processing procedure of the mobile terminal 100 according to the embodiment. An analog meter reading process for reading a meter reading value from an analog meter will be described with reference to FIGS.

In step S301 of FIG. 9, the analog meter reading unit 116c of the mobile terminal 100 reads the setting information data 106b from the storage unit 106 and proceeds to step S302. The read setting information data 106b is used as initial setting parameters (initial setting values) in subsequent processing.

Examples of the setting information data 106b include information that records the minimum and maximum values displayed on the display panel for various analog meters in association with the meter's identification information, or learning data that has been trained by AI (Artificial Intelligence) to identify the meter.

The learning data is generated through machine learning based on preset information in an image of an analog meter, such as the center point of the meter, the start and end points of the pointer's range of motion, the lowest point of the display dial, the range of the pointer (minimum and maximum values), and threshold values.

In step S302, the analog meter reading unit 116c activates the camera 104 to capture an image of the analog meter MT (e.g., any one of meters MT1 to MTn) to obtain image data, and the process proceeds to step S303.

Specifically, in step S302, the camera 104 is started, and the image captured by the camera 104 is displayed on the display unit 103. The meter reader checks this display, adjusts the composition of the shot, and operates the shutter so that the entire analog meter MT is captured. This provides image data that captures the entire analog meter MT, and the captured analog meter MT is displayed on the display unit 103.

In step S303, the analog meter reading unit 116c displays the photographed analog meter MT on the display unit 103 based on the image data obtained in step S302, and detects the numerals and pointers on the display panel of the analog meter MT that appear in the image data. Note that the recognition engine of the image processing unit 114 is used for this detection.

In addition, in step S303, the analog meter reading unit 116c displays the detected written numbers and pointer on the displayed analog meter MT by enclosing them in a rectangular frame (numerical frame), and then proceeds to step S304.

Figure 11 shows an example of a frame that surrounds each of the notated numbers. These displays are performed through the display control unit 113, and the same applies to the displays described below. Note that the notated numbers and pointers on the analog meter MT can be detected, for example, by using AI.

That is, based on the learning data and image data contained in the setting information data 106b, the analog meter reading unit 116c identifies the type of analog meter MT, and further detects the numbers and pointer on the display panel from the learning data for this identified analog meter MT.

In step S304, the analog meter reading unit 116c detects the center position (coordinates) of the frame surrounding the notated number displayed in step S303, and proceeds to step S305. The coordinates detected here are referred to as notated number coordinates in the following description. Notated number coordinates are based on two-dimensional coordinate axes set on the displayed analog meter MT, and the coordinates in the following description are also based on the above coordinate axes.

In step S305, the analog meter reading unit 116c finds an approximate circle (pseudo circle) that passes through the center of the frame line based on the notated numeric coordinates detected in step S304. That is, the analog meter reading unit 116c finds a pseudo circle on whose circumference the notated numeric coordinates exist. Furthermore, the analog meter reading unit 116c calculates the coordinates of the center point O of this circle and its radius R, and proceeds to step S306.

Figure 12 is a diagram showing an example of a pseudo circle and its center point O. In step S306, the analog meter reading unit 116c detects the coordinate Ac of the tip A of the pointer based on the frame line surrounding the pointer and the center point O of the pseudo circle, and the processing procedure proceeds to step S307.

Fig. 13 is a diagram showing an example of a frame line surrounding a pointer and a center point O of the circle. Fig. 14 shows an example of an image including a dial that is tilted toward the back. In the example of Fig. 14, of the four vertices of the rectangular frame line surrounding the pointer, the vertex diagonal to the vertex closest to the center point O of the circle is detected as the coordinate Ac of the tip A of the pointer.

However, depending on the angle of the pointer, as shown in Figure 15, it is possible that an inappropriate vertex b2 will be detected instead of vertex b1, which is the closest point to the center O of the circle. For this reason, it is also possible to detect vertex b1 by taking into consideration the distance between the center O and the midpoints c1 to c4 of each side of the frame line, and then detect the coordinates of vertex a, which is diagonal to vertex b1 and corresponds to the tip A of the pointer.

In step S307, the analog meter reading unit 116c arranges the notated numeric coordinates detected in step S304 in order, and the processing proceeds to step S308.

In step S307, the analog meter reading unit 116c specifically arranges the numerical coordinates on the circumference in a clockwise direction, starting from the starting point S on the circumference below the center point O of the circle, as shown by the arrow in FIG. 16. This is in consideration of the fact that the readings of a typical analog meter change (become larger) in the clockwise direction.

Next, in step S501, the image processing unit 114 cuts out a numerical image of the frame line (numerical frame), and in step S502, recognizes the rotation angle of the numerical value in the cut-out numerical image. The rotation angle is output as tilt information from the trained model 600. Using this tilt information, in step S503, the image processing unit 114 performs horizontal correction processing on the cut-out numerical image to generate image data of the written numerals without tilt. This generates image data in which all the numerical values on the dial are horizontal, as shown in FIG. 17.

In step S308, the analog meter reading unit 116c reads the numerical value of the number displayed within the frame enclosed in step S303, as shown in FIG. 17, and the processing procedure proceeds to step S309.

Reading (detecting) the numerical value of the notated digits can be achieved, for example, by using AI. That is, the analog meter reading unit 116c can identify the type of analog meter MT, etc., based on the learning data and image data contained in the setting information data 106b. Furthermore, the numerical value of the notated digits on the display panel can be detected from the learning data for this identified analog meter MT.

In step S309, the analog meter reading unit 116c determines whether the numerical value read in step S308 includes a "0". If the numerical value read in step S308 includes a "0", the process proceeds to step S310, and if the numerical value does not include a "0", the process proceeds to step S312.

In step S310, the analog meter reading unit 116c determines whether or not a number exists to the left of the numeric value "0" from the center point O, based on the arrangement performed in step S307 and the numeric value read in step S308. If a numeric value exists to the left, the process proceeds to step S311, and if no numeric value exists to the left, the process proceeds to step S314.

In step S311, the analog meter reading unit 116c converts any numerical values read in step S308 that are to the left of the numerical value "0" to a negative value of the same magnitude, and proceeds to step S314. Note that numerical values to the right of the numerical value remain positive.

In the case of an analog meter as shown in FIG. 18, the numbers "0.05" and "0.1" exist to the left of the number "0." Therefore, the analog meter reading unit 116c converts these numbers to "-0.05" and "-0.1," respectively.

In step S312, the analog meter reading unit 116c compares two adjacent values based on the arrangement performed in step S307 and the values read in step S308, and determines whether or not a value exists that is equal to or greater than the value on the right. In other words, it determines whether or not a value exists on the left side of the center point O that is equal to or greater than the value on the right side. If a value exists that is equal to or greater than the value on the right side, the process proceeds to step S311, and if no value exists that is equal to or greater than the value on the right side, the process proceeds to step S314.

In step S313, the analog meter reading unit 116c inverts the sign of any numerical value read in step S308 that is equal to or greater than the numerical value to the right, and the process proceeds to step S314.

In the case of an analog meter as shown in FIG. 18, the value "0.05" exists to the right of the value "0.1" on the far left. Therefore, the analog meter reading unit 116c converts this value "0.1" to "-0.1". Similarly, the value "0" exists to the right of the value "0.05" on the left, so the analog meter reading unit 116c converts this value "0.05" to "-0.05".

In step S314 in FIG. 9, the analog meter reading unit 116c detects the coordinate Xc of the intersection X between the line segment connecting the coordinate Ac of the tip A of the pointer detected in step S306 and the coordinate of the center point O of the circle detected in step S305, and the circumference of the circle O determined in step S305, as shown in FIG. 19, for example. After that, regardless of the presence or absence of the intersection X, the processing procedure proceeds to step S401 in FIG. 10.

In step S401 of FIG. 10, the analog meter reading unit 116c determines whether or not the coordinate Xc of the intersection X was detected in step S314. If the coordinate Xc of the intersection X was detected (i.e., the intersection X exists), the process proceeds to step S403. If the coordinate Xc of the intersection X cannot be detected, for example, because the intersection X does not exist, the process proceeds to step S402.

In step S402, the analog meter reading unit 116c extends the pointer, creates an intersection X between the extended pointer and the circumference, and detects the coordinate Xc of the intersection X. The processing procedure then proceeds to step S402. Note that the detection of the coordinate Xc of the intersection X can be achieved, for example, by extending the pointer from the center point O in the direction of the coordinate Ac based on the coordinate Ac of the tip A of the pointer detected in step S306 and the center point O.

In step S403, the analog meter reading unit 116c searches for written numbers on the left and right circumferences of the coordinate Xc of the intersection point X detected in step S314 or step S402, and determines whether or not those written numbers are present.

In the example of FIG. 20, the notation number NL is "20" on the left circumference of the coordinate Xc of the intersection point X, and the notation number NR is "40" on the right circumference.

If notational numbers can be detected on both the left and right, the process proceeds to step S404. If notational numbers are present only on one side, the process proceeds to step S409, assuming that the minimum value (when only NR is detected) or maximum value (when only NL is detected) has been detected as the pointer value.

In FIG. 21(a), as in FIG. 20, the notation numeral NL is "20" on the left circumference of the coordinate Xc of the intersection point X, and the notation numeral NR is "40" on the right circumference. Therefore, the processing procedure proceeds to step S404.

On the other hand, as shown in FIG. 21(b), if there are no written numbers to the left of intersection point X (up to the starting point of the circumference) as viewed from center point O, the minimum value based on the initial setting value is determined as the pointer value, and the processing procedure proceeds to step S409.

If there are no written numbers to the right of intersection point X (up to the end point of the circumference), the pointer value is determined to be the maximum value based on the initial setting value, and the processing proceeds to step S409.

Also, as shown in FIG. 21(c), if the hand overlaps with the written numeral "40" and the written numeral "40" is not detected in step S303 or step S304, the written numeral "40" is not detected. Therefore, "40" is not used in the judgment of step S403. However, since the other written numerals "20" and "60" on the left and right are detected as NL and NR, respectively, in step S303 and step S304, the processing procedure proceeds to step S404 after the presence of these written numerals is detected in step S403.

Furthermore, as shown in FIG. 21(d), if the hand overlaps with the numeral "40" and for some reason the numeral "60" is not detected, the numerals "40" and "60" are not used in the determination of step S403. However, since the numerals "20" and "80" on the left and right sides have been detected in steps S303 and S304, these numerals are detected as NL and NR, respectively, in step S403, and the processing procedure proceeds to step S404.

As shown in FIG. 22, in step S404, the analog meter reading unit 116c first detects the angle S1 formed by the line segment connecting the coordinates of the notated number NL on the left side of the intersection point X and the coordinates of the center point O, out of the two notated numbers NL and NR detected in step S403, and the line segment connecting the coordinates of the intersection point X and the coordinates of the center point O.

Next, the analog meter reading unit 116c detects the angle S2 between the line segment connecting the coordinates of the notated number NR on the right side of the intersection point X and the coordinates of the center point O, out of the two notated numbers NL and NR detected in step S403, and the line segment connecting the coordinates of the intersection point X and the coordinates of the center point O, and proceeds to step S405.

As shown in FIG. 23, in step S405, the analog meter reading unit 116c calculates the pointer value VD corresponding to the intersection point X based on the angles S1 and S2, the radius R, the value VL of the left-side notation number NL, and the value VR of the right-side notation number NR. After this, the processing procedure ends and proceeds to step S406.

Here, the indicator value VD can be calculated, for example, as VD = (S1 * VR + S2 * VL) / (S1 + S2).

In step S406, the analog meter reading unit 116c detects the maximum number of decimal places based on the numerical value of the notated digits read in step S308, and the processing procedure proceeds to step S402.

For an analog meter like that shown in Figure 24, the read value is an integer value, so the maximum number of digits after the decimal point is detected as 0. For an analog meter like that shown in Figure 25, the read value includes "0.1" and "0.05", and based on "0.05", the maximum number of digits is detected as 2.

In step S407, the analog meter reading unit 116c calculates the maximum number of digits detected in step S406 plus 1 as the number of significant decimal places. The analog meter reading unit 116c then determines the value up to the number of significant decimal places from the calculation result in step S405 as the pointer value, and the process proceeds to step S408.

In an analog meter such as that shown in FIG. 24, if the pointer value is calculated as "30.04567..." in step S405, the pointer value is determined to be "30.0" up to the first decimal place because the number of significant decimal places is 1. The first decimal place value is determined by rounding off the value to the second decimal place.

In step S408, the analog meter reading unit 116c generates data of a cropped image (hereinafter referred to as cropped image data) obtained by cropping the needle and the numerical values displayed in its vicinity, and the range including the intersection point X from the captured image of the analog meter MT based on the image data obtained in step S302, and then proceeds to step S409.

In step S409, the correction processing unit 116d displays the trimmed image on the display unit 103 based on the trimmed image data generated in step S408, instead of the image being displayed, and displays the pointer value determined in step S403 or the pointer value determined in step S407 on the display unit 103. The processing procedure then proceeds to step S410.

Figure 26 shows an example of an interface for correcting the detected pointer value. That is, a cropped image in which the range including the intersection point X in the image data is enlarged is displayed, and a drum roll and a slide bar are displayed on the display unit 103 as a GUI that functions as soft keys.

The drum roll display is an input interface for correcting the pointer value, and graphically displays a virtual drum roll (number wheel) that corresponds to each digit of the pointer value. The drum roll can be operated via a touch panel (input unit 102) placed on the display unit 103.

For example, the input unit 102 can detect a flick input that rotates each drum roll up and down, and in response rotate and display the image of the drum roll, and change the pointer value to any value displayed accordingly.

The slide bar display is an input interface for setting a decimal point between drum rolls, and corresponds to the position of the drum roll in the vertical direction. By sliding the pointer P left and right, a decimal point can be set between digits (drum rolls).

Figure 27 shows an example of a GUI for correcting the position of the decimal point of the detected indicator value. Figures 27(a) to (e) show how the position of the decimal point of the indicator value changes depending on the position of the pointer P. In Figure 27, the position of the decimal point of the indicator value and the position of the pointer P are set to match in the vertical direction.

The position of the pointer P and the position of the decimal point in the number string represented by the drum roll do not necessarily have to coincide vertically. For example, the range that can be displayed on the display unit 103 may be maximized for inputting the decimal point, such as by displaying a slide bar with a range wider than the display range of the number string of the pointer value. In that case, the position of the decimal point is variable depending on the relative position of the pointer P within its movable range.

Returning to FIG. 10 for further explanation, in step S410, the correction processing unit 116d requests the meter reader to input confirmation of the indicator value displayed on the display unit 103. If an OK input operation is performed on the input unit 102, the displayed indicator value is determined to be the meter reading value, and the processing procedure proceeds to step S412. On the other hand, if an NG input operation is performed on the input unit 102, the processing procedure proceeds to step S411.

In step S411, the correction processing unit 116d detects an operation on the drum roll display and the slide bar display through the input unit 102. The correction processing unit 116d accepts an instruction to correct the meter value from the meter reader, determines the corrected (displayed) meter value as the meter reading value, and the processing procedure proceeds to step S412.

In step S412, the report creation unit 116a stores the image data obtained in step S302, the cropped image data generated in step S408, and the meter reading value determined in step S410 or step S411 as report data 106a in the storage unit 106 in association with the identification information of the analog meter to be read. After this, the processing procedure proceeds to step S413.

In step S413, the analog meter reading unit 116c determines whether or not an instruction to end work has been given by the meter reader via the input unit 102. If an instruction to end work is detected, the process ends. If an instruction to continue work is given, the process proceeds to step S302, and meter reading of the analog meter MT is started anew.

(effect)
As described above, according to the embodiment, the trained model 600 that classifies image data into "tilt" rather than "numerical values" shown in the image is created by machine learning and stored in the mobile terminal 100. The mobile terminal 100 provides the image data 106c including notated numbers to the trained model 600 and acquires tilt information.

As shown in FIG. 28, the mobile terminal 100 cuts out the image data using the recognized numerical frame to generate numeric image data, and inputs the numeric image data into the AI-based trained model 600. The AI then recognizes the rotation angle of each numerical value. Then, the numeric image data is rotated around the center of the numerical frame as the axis by the angle obtained by the rotation angle recognition, and horizontal correction processing is performed.

Here, since the numerical frame in FIG. 28 is a rectangle, for example, the intersection of the diagonals of the four vertices can be extracted as the center point of the numerical frame. In the embodiment, a pseudo-circle that passes through the center point of the numerical frame is detected, and this pseudo-circle is used in the horizontal correction process. The pseudo-circle is found along the arc drawn by the center point of the numeric image data.

Figure 29 is a diagram for providing additional explanation of the numerical frame and the pseudo circle. As shown in Figure 29 (a), the coordinates of the center point of the numerical frame and the coordinates of the center (center point) of the pseudo circle are detected based on the recognition results of the numerical frame and the pointer. Furthermore, a line segment connecting the center of the pseudo circle and the center point of the numerical frame is detected. Then, the angle between this line segment and one of the four sides of the numerical frame that intersects with the line segment (for example, the bottom side of the numerical frame) is detected, and by performing a rotation correction to cancel this angle, it is possible to obtain image data in which the numerical value is horizontal, as shown in Figure 29 (b). Then, the measurement value indicated by the pointer can be calculated from the division ratio of the pseudo circle by the meter pointer in the pseudo circle.

Three points are geometrically necessary and sufficient to define a pseudo-circle. If the numerical box has three center points, a pseudo-circle that passes through these three points is uniquely determined. On the other hand, if the numerical box has four or more center points, a unique pseudo-circle can be calculated by applying, for example, the least squares method using the coordinates of each center point.

In this way, according to the embodiment, the diagonal numbers in the captured image data are corrected so that they are horizontal. Therefore, since it is only necessary to read the numbers in a horizontal state, the accuracy of reading numbers can be dramatically improved compared to directly subjecting image data containing numbers in a diagonal state to a number recognition process.

In other words, in the embodiment, instead of directly classifying image data into numerical values, the image data is first classified into tilt information and then the image data is corrected to be horizontal. Then, the horizontal image data is used to perform image recognition processing to read the numerical values. Therefore, according to the embodiment, it is possible to provide a program, information processing device, and information processing method that improve the accuracy of reading numerical values when reading analog meters. Ultimately, it is possible to reduce the human burden involved in meter reading work.

Note that this invention is not limited to the above-described embodiment. For example, application examples are not limited to equipment in factories or business establishments, or various measuring instruments. For example, meters MT1 to MTn may indicate other physical quantities or states, not limited to measurements such as current, voltage, pressure or flow rate of gas or liquid, or the operating state of equipment or devices.

In addition, the air interface between the mobile terminal 100 and the base station 200 in FIG. 1 is not limited to wireless LAN, 4G, 5G, etc. In other words, as long as the mobile terminal 100 can be connected to the network 500 via a wireless link, any wireless communication method may be applied.

In addition, it is also possible to integrate some or all of the functions included in the processor 110 in FIG. 2 with other functions, or to divide each function into multiple functional blocks from a different perspective and explain it in a different way. In addition, the neural network in FIG. 4 is not limited to CNN or DNN, and neural networks of different formats, such as SVM (Support Vector Machine), can be used as appropriate.

The meter reader may also indicate via the input unit 102 whether to read a numeric meter or an analog meter. The camera 104 may also capture an image of the meter to be read, and the recognition engine may distinguish (determine) the type of meter captured in the image as either a numeric meter or an analog meter.

In the above embodiment, the analog meter is photographed, the measurement value of the analog meter is detected, and the document data 106a that records the detected value is created by the mobile terminal 100. However, the present invention is not limited to this. For example, only the analog meter is photographed by the mobile terminal 100, and the measurement value of the analog meter is detected and the document data 106a that records the detected value is created by information processing in the client terminal 300 or the server 400.

In other words, all or some of the functions of the mobile terminal 100 may be implemented in the client terminal 300 or the server 400. For example, an image acquired by the mobile terminal 100 may be transferred to the client terminal 300, and image processing, angle recognition, rotation correction, numerical recognition, and other processing may be performed by the client terminal 300. The extent to which processing is shared between the edge (mobile terminal 100) and the cloud (client terminal 300, server 400) can be flexibly determined according to system requirements.

Furthermore, the services provided by the server 400 may be provided on a subscription basis to the mobile terminal 100 or client terminal 300 owned (or loaned) by the client receiving the services.

Furthermore, as described with reference to FIG. 29, it is possible to generate image data in a horizontal state by detecting the angle between the center of the pseudo circle and the numerical frame through image processing and performing rotation correction to cancel this angle. According to such a procedure, it is possible to perform horizontal correction processing without relying on the trained model 600.

Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

6a... needle detection unit, 6b... circle detection unit, 6c... intersection detection unit, 6d... intersection number detection unit, 6e... measurement value detection unit, 100... mobile terminal, 101... communication unit, 102... input unit, 103... display unit, 104... camera, 105... GNSS receiving unit, 106... storage unit, 106a... form data, 106b... setting information data, 106c... image data, 110... processor, 111... communication control unit, 112... input control unit, 113... display control unit, 114... Image processing unit, 114a...classification unit, 114b...correction unit, 114c...notation number detection unit, 115...location information processing unit, 116...meter reading data compilation processing unit, 116a...document creation unit, 116b...numeric meter reading unit, 116c...analog meter reading unit, 116d...correction processing unit, 200...base station, 300...client terminal, 400...server, 500...network, 600...trained model, 700...program, MT1 to MTn...meters.

Claims (6)

Computer,
A storage unit that stores a slant recognition model generated from learning data related to the slant of the numbers in the number image data related to an image of the numbers of the analog meter;
a tilt information acquisition unit that acquires tilt information of each of the plurality of number image data extracted from image data obtained by photographing the analog meter from the tilt recognition model;
a correction unit that extracts a center point of the number image data based on the inclination information, and rotates the number image data around the center point to correct the inclination of the number image data;
a number recognition unit for recognizing a number value from the rotation-corrected number image data;
a pointer detection unit for detecting the position of the tip of a pointer of the analog meter;
a measurement value detection unit that detects a measurement value indicated by the pointer based on the positions of a plurality of numbers displayed on a display panel of the analog meter, values indicated by the numbers, and the position of the tip of the pointer ;
a circle detection unit that detects a pseudo-circle passing through a position of a written number of the number image data and a center position of the pseudo-circle;
A program to make it function as such. The circle detection unit
Detecting a rotation center point of a pointer of the analog meter and the center points of the plurality of numeric image data;
The pseudo circle is determined so that the center point of the numerical image data is on an arc having the rotation center point of the pointer as a center;
The measurement value detection unit
2. The program according to claim 1 , further comprising: a measuring device for measuring a measurement value indicated by said pointer based on a division ratio of said pseudo circle by said pointer . An information processing device having a storage unit and a processor,
The storage unit is
storing a slant recognition model generated from learning data related to the slant of the numbers in the number image data related to an image of the numbers of the analog meter;
The processor,
a tilt information acquisition unit that acquires tilt information of each of the plurality of number image data extracted from image data obtained by photographing the analog meter from the tilt recognition model;
a correction unit that extracts a center point of the number image data based on the inclination information, and rotates the number image data around the center point to correct the inclination of the number image data;
a number recognition unit for recognizing a number value from the rotation-corrected number image data;
a pointer detection unit for detecting the position of the tip of a pointer of the analog meter;
a measurement value detection unit that detects a measurement value indicated by the pointer based on the positions of a plurality of numbers displayed on a display panel of the analog meter, values indicated by the numbers, and the position of the tip of the pointer;
a circle detection unit that detects a pseudo-circle passing through a position of a written number of the number image data and a center position of the pseudo-circle;
An information processing device comprising: The circle detection unit
Detecting a rotation center point of a pointer of the analog meter and the center points of the plurality of numeric image data;
The pseudo circle is determined so that the center point of the numerical image data is on an arc having the rotation center point of the pointer as a center;
The measurement value detection unit
4. The information processing apparatus according to claim 3, further comprising: a measuring device for measuring a measurement value indicated by said pointer based on a division ratio of said pseudo circle by said pointer . The computer
A classification process for acquiring inclination information of each of a plurality of number image data related to images of numbers extracted from image data obtained by photographing an analog meter, from an inclination recognition model generated from learning data related to the inclination of the numbers in the number image data;
a correction step of extracting a center point of the number image data based on the inclination information, and rotating the number image data around the center point to correct the inclination of the number image data;
a number recognition step of recognizing a number value from the rotation-corrected number image data;
a pointer detection step of detecting the position of the tip of a pointer of the analog meter;
a measurement value detection step of detecting a measurement value indicated by the pointer based on the positions of a plurality of numbers displayed on a display panel of the analog meter, values indicated by the numbers, and the position of the tip of the pointer;
a circle detection step of detecting a pseudo circle passing through the position of the written number of the number image data and the position of the center of the pseudo circle;
An information processing method comprising: The circle detection step includes:
The computer
Detecting a rotation center point of a pointer of the analog meter and the center points of the plurality of numeric image data;
A step of determining the pseudo circle having a rotation center point of the pointer as a center and a center point of the numerical image data on an arc;
The measurement value detection step includes:
6. An information processing method according to claim 5, further comprising the step of: calculating, by a computer, a measurement value indicated by said pointer from a division ratio of said pseudo circle by said pointer.