JP7664897B2 - Recognition processing device, recognition processing program, recognition processing method, and recognition processing system - Google Patents

Description

This invention relates to a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system, and can be applied, for example, to online handwritten character recognition processing.

Conventionally, a method has been proposed for online handwritten character recognition processing in which features are acquired from the strokes (handwriting) made when characters are input, machine learning is performed based on the acquired features to acquire a learning model, and the acquired learning model is used to recognize the input characters.

One technique for performing character recognition processing by performing machine learning on the features of strokes like those described above is described in Patent Document 1.

In the technology described in Patent Document 1, the positions of strokes over time (hereinafter referred to as "input patterns") are obtained as samples for each character, feature points (positions that make up the strokes) are associated between the sampled input patterns and a standard pattern (an input pattern when input in standard handwriting), and the difference between the associated feature points is obtained as a feature value to be used in character recognition processing.

The technology described in Patent Document 1 performs machine learning using the feature values acquired for learning as training data to acquire a learning model. In the technology described in Patent Document 1, during character recognition processing, the differences between all standard patterns and feature points of the sample input pattern are calculated and acquired as feature amounts, and the feature amounts with all the acquired standard patterns are input into the learning model to perform character recognition processing.

JP 2018-112521 A

However, with the technology described in Patent Document 1, a suitable standard pattern must be prepared for each character in order to obtain a learning model, which means the cost of creating the learning model is high. In addition, with the technology described in Patent Document 1, the feature amounts between the sample input pattern and all standard patterns must be acquired and then recognized during character recognition processing, which means an extremely high processing load.

In the technology described in Patent Document 1, the difference from a standard pattern is used as the feature value for the character input in order to secure a larger amount of information, but as described above, the technology described in Patent Document 1 requires a large amount of resources (e.g., work costs, hardware resources, etc.) for creating a learning model and for recognition processing.

Therefore, there is a demand for a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system that can obtain features of online character input more efficiently and perform machine learning.

A first recognition processing device of the present invention has a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character, the input stroke data including position information of feature points in chronological order of the input pattern, and performing a normalization process for normalizing the acquired input stroke data to an input pattern with a fixed number of samples to acquire normalized stroke data, an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by feature amounts of the fixed number of samples, and a character recognition processing means for performing a character recognition process on the input vector data acquired by the input vector data acquisition means using a learning model trained by machine learning using the input vector data acquired by the input vector data acquisition means, and the normalization process performed by the normalization means includes a first normalization processing step for acquiring intermediate stroke data by thinning out feature points for the input stroke data such that an interval between all adjacent feature points in chronological order is equal to or greater than a predetermined threshold, and a second normalization processing step for acquiring data obtained by interpolating feature points for the intermediate stroke data such that an interval between all adjacent feature points in chronological order is equal to or less than the threshold .

A second recognition processing program of the present invention causes a computer to function as: a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character, the input pattern including position information of feature points in chronological order, and performing a normalization process for normalizing the acquired input stroke data into an input pattern with a fixed number of samples to acquire normalized stroke data; an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by feature amounts of the fixed number of samples; and a character recognition processing means for performing character recognition processing on the input vector data acquired by the input vector data acquisition means, using a learning model trained by machine learning using the input vector data acquired by the input vector data acquisition means, wherein the normalization process performed by the normalization means includes a first normalization processing step for acquiring intermediate stroke data by thinning out feature points for the input stroke data such that an interval between all adjacent feature points in chronological order is equal to or greater than a predetermined threshold; and a second normalization processing step for acquiring data for the intermediate stroke data by interpolating feature points such that an interval between all adjacent feature points in chronological order is equal to or less than the threshold .

A third aspect of the present invention is a recognition processing method performed by a recognition processing device, the recognition processing device having a normalization means, an input vector data acquisition means, a character recognition processing means, and a character recognition result output means, the normalization means acquires input stroke data indicating an input pattern of strokes made by an electronic pen in chronological order for each input character and including position information of feature points in chronological order of the input pattern, and acquires normalized stroke data by performing a normalization process for normalizing the acquired input stroke data into an input pattern with a fixed number of samples, and the input vector data acquisition means acquires input vector data expressed by feature amounts of the fixed number of samples from the normalized stroke data normalized by the normalization means. the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means, and the normalization processing performed by the normalization means includes a first normalization processing step of acquiring intermediate stroke data by thinning out feature points from the input stroke data such that intervals between all feature points adjacent in chronological order are equal to or greater than a predetermined threshold, and a second normalization processing step of acquiring data by interpolating feature points from the intermediate stroke data such that intervals between all feature points adjacent in chronological order are equal to or less than the threshold.

The fourth invention is a recognition system having a pen tablet capable of receiving input using an electronic pen and a front electronic pen, and a recognition processing device that recognizes characters written on the pen tablet by a user with the electronic pen, characterized in that the recognition processing device of the first invention is applied as the recognition processing device.

The present invention makes it possible to obtain features of online character input more efficiently and perform machine learning.

1 is a block diagram showing an overall configuration according to a first embodiment. 4 is a diagram showing an example of the configuration of input vector data according to the first embodiment; FIG. FIG. 10 is a diagram showing a specific example (part 1) of input vector data according to the first embodiment. FIG. 11 is a diagram showing a specific example (part 2) of input vector data according to the first embodiment. FIG. 2 is a diagram showing an example of the configuration of a character input screen according to the first embodiment; 5A to 5C are diagrams illustrating an example of the configuration of input stroke data acquired by a stroke data processing unit according to the first embodiment; 5 is a flowchart showing a process in which a stroke data processing unit according to the first embodiment generates input vector data for online character recognition processing from input stroke data. 5A to 5C are diagrams illustrating a process in which the stroke data processing unit according to the first embodiment normalizes input vector data for online character recognition processing to obtain first normalized stroke data. 6A to 6C are diagrams illustrating processing performed by the stroke data processing unit according to the first embodiment to thin out feature points of first normalized stroke data; FIG. 11 is a diagram illustrating an example in which the stroke data processing unit according to the first embodiment acquires second normalized stroke data by performing feature point interpolation processing for filling intervals between feature points of first normalized stroke data to a predetermined interval or less. FIG. 11 is a diagram illustrating an example of third normalized stroke data obtained by extracting N+1 feature points from second normalized stroke data by the stroke data processing unit according to the first embodiment; 13A to 13C are diagrams illustrating an image based on third normalized stroke data acquired by the stroke data processing unit according to the first embodiment. 13 illustrates an example of input vector data acquired by the stroke data processing unit according to the first embodiment based on the third normalized stroke data. 10 is a flowchart showing a process in which a stroke data processing unit according to the first embodiment generates input image data for offline character recognition processing from input stroke data. 13A to 13C are diagrams illustrating normalization processing in the process in which the stroke data processing unit according to the first embodiment generates input image data for offline AI processing. FIG. 13 is a diagram showing input image data acquired by the stroke data processing unit according to the first embodiment by drawing each feature point of the fifth normalized stroke data in a normalized area of 64 pixels by 64 pixels. 10 is a flowchart showing a learning process when the character recognition processing unit according to the first embodiment operates in a learning mode. 5 is a flowchart showing a recognition process when the character recognition processing unit according to the first embodiment operates in a recognition process mode. 13 is a timing chart (part 1) showing the state of the pen tip over time when a user inputs characters to a pen tablet using an electronic pen in the second embodiment; 13 is a timing chart (part 2) showing the pen tip state over time when a user inputs characters to a pen tablet using an electronic pen in the second embodiment. The results of counting the pen tip states at each characteristic point shown in the timing chart of FIG. 19 are shown. FIG. 20 is a diagram showing each sample (characteristic point) shown in the timing chart of FIG. 19 . FIG. 20 is a diagram showing an example of input stroke data when a second normalization method is applied to each feature point shown in the timing chart of FIG. 19 . FIG. 20 is a diagram showing an example of input stroke data when the third normalization method is applied to each feature point shown in the timing chart of FIG. 19 . FIG. 20 is a diagram showing an example of input stroke data when a fourth normalization method is applied to each feature point shown in the timing chart of FIG. 19 . FIG. 20 is a diagram showing an example of input stroke data when the fifth normalization method is applied to each feature point shown in the timing chart of FIG. 19 .

(A) First embodiment Hereinafter, a first embodiment of a recognition processing device, a recognition processing program, a recognition processing method, and a recognition processing system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which an information processing terminal is configured as the recognition processing device of the present invention will be described.

(A-1) Configuration of the First Embodiment Fig. 1 is a block diagram showing the overall configuration of a recognition processing system 1 according to the first embodiment. Note that the reference characters in parentheses in Fig. 1 are reference characters used only in the second embodiment described later.

The recognition processing system 1 has an information processing terminal 10, a pen tablet 20, and an electronic pen 30. The pen tablet 20 is a device capable of receiving input using the electronic pen 30.

The pen tablet 20 is a device that detects the pen tip 31 of the electronic pen 30 on the surface of the display panel 21 (a device that functions as a so-called "pointing device"). The display panel 21 can also output an image based on a video signal supplied from the information processing terminal 10. The recognition processing system 1 is capable of accepting character input operations from the user by displaying the trajectory of the pen tip 31 on the display panel 21.

The information processing terminal 10 has a control unit 11, a video IF 12, and a USB port 13.

The information processing terminal 10 can be configured by installing a program (including the recognition processing program according to the embodiment) on various computers (e.g., a PC, etc.).

The control unit 11 has a content processing unit 111, a display driver 112, a pen tablet driver 113, and a character recognition processing unit 114.

The content processing unit 111 is an application program that accesses the pen tablet 20 via the display driver 112 and the pen tablet driver 113, and processes various contents using the pen tablet 20 and the electronic pen 30 (for example, contents including various operation screens involving character input using the pen tablet 20 and the electronic pen 30).

The content processing unit 111 displays an operation screen (GUI) involving character input via the display driver 112, and acquires the contents input using the electronic pen 30 on the liquid crystal tablet via the pen tablet driver 113 (e.g., information on the coordinates of the part touched by the electronic pen 30, etc.). The content processing unit 111 acquires stroke data during character input (data including the coordinates of the electronic pen 30 for each time series; hereafter referred to as "input stroke data") via the pen tablet driver 113. In this embodiment, the content processing unit 111 supplies the input stroke data to the character recognition processing unit 114.

The character recognition processing unit 114 performs character recognition processing based on input stroke data, and includes a stroke data processing unit 1141, an online AI processing unit 1142, an offline AI processing unit 1143, and a character recognition result output unit 1144. The character recognition processing unit 114 is responsible for the functions corresponding to the recognition processing program according to this embodiment.

The stroke data processing unit 1141 performs processing to generate, from the input stroke data, vector data (input vector data) that can be applied to processing by the online AI processing unit 1142, and image data (hereinafter referred to as "input image data") that can be applied to processing by the offline AI processing unit 1143.

In this embodiment, the input stroke data and input image data are described as being supplied from the content processing unit 111 to the character recognition processing unit 114, but the source that supplies the input stroke data and input image data to the character recognition processing unit 114 is not limited to this. For example, input stroke data and input image data created externally may be supplied to the character recognition processing unit 114 for processing.

When input vector data is supplied, the online AI processing unit 1142 performs character recognition processing or learning processing based on the input vector data. When the character recognition processing unit 114 operates in the learning mode, the online AI processing unit 1142 performs machine learning processing using a set of input vector data and correct answer labels (teacher labels) to update the learning model. When input vector data is supplied, the online AI processing unit 1142 performs character recognition processing or learning processing based on the input vector data. When the character recognition processing unit 114 operates in the learning mode, the online AI processing unit 1142 performs machine learning processing using a set of input vector data and correct answer labels. When the character recognition processing unit 114 operates in the recognition processing mode, the online AI processing unit 1142 performs character recognition processing using a learned learning model based on the input vector data, and outputs a character recognition result (hereinafter also referred to as an "online character recognition result"). In this embodiment, the online AI processing unit 1142 adds reliability data to the online character recognition result.

When input image data is supplied, the offline AI processing unit 1143 performs character recognition processing or learning processing based on the input image data. When the character recognition processing unit 114 operates in learning mode, the offline AI processing unit 1143 performs machine learning processing using a set of input image data and correct answer labels to update the learning model. When input image data is supplied, the offline AI processing unit 1143 performs character recognition processing or learning processing based on the input image data. When the character recognition processing unit 114 operates in learning mode, the offline AI processing unit 1143 performs machine learning processing using a set of input image data and correct answer labels. When the character recognition processing unit 114 operates in recognition processing mode, the offline AI processing unit 1143 performs character recognition processing using a learned learning model based on the input image data, and outputs a character recognition result (hereinafter also referred to as an "offline character recognition result"). In this embodiment, the offline AI processing unit 1143 adds reliability data to the offline character recognition result.

In this embodiment, the online AI processing unit 1142 and the offline AI processing unit 1143 can be configured using various machine learning engines (AI platforms). Therefore, in this embodiment, details of the machine learning method (method of creating a learning model) in the online AI processing unit 1142 and the offline AI processing unit 1143 and the recognition processing (determination processing) using the created learning model will not be described.

When the character recognition processing unit 114 operates in the recognition processing mode, the character recognition result output unit 1144 outputs a final character recognition result (hereinafter also referred to as an "output character recognition result") based on the online character recognition result and the offline character recognition result. The character recognition result output unit 1144 may evaluate the online character recognition result and the offline character recognition result, and output one of the character recognition results based on the evaluation result. For example, the character recognition result output unit 1144 may adopt the online character recognition result or the offline character recognition result with the higher added evaluation value, and output it as the output character recognition result.

In this embodiment, when the content processing unit 111 operates the character recognition processing unit 114 in the learning mode, it supplies the character recognition processing unit 114 with the input stroke data as well as a correct label (an identifier of the correct character corresponding to the input stroke data) corresponding to the input stroke data. Also, in this embodiment, when the content processing unit 111 operates the character recognition processing unit 114 in the recognition processing mode, it supplies the input stroke data to the character recognition processing unit 114 and obtains the output character recognition result.

Next, we will explain an example of the configuration of input stroke data acquired by the stroke data processing unit 1141.

The input stroke data includes information regarding the position of the tip 31 of the electronic pen 30 for each time series as recognized by the pen tablet 20 (display panel 21).

Hereinafter, the direction parallel to the surface of the display panel 21 that comes into contact with the pen tip 31 of the electronic pen 30 (hereinafter referred to as the "display contact surface") will be referred to as the "horizontal direction", and the direction perpendicular to the display contact surface will be referred to as the "height direction". In addition, hereinafter, the state (status) related to the height direction between the pen tip 31 and the display contact surface will be referred to as the "pen tip state". In the first embodiment, the term "pen tip state" will be described as a status indicating either a state in which the pen tip 31 is in contact with the display contact surface (hereinafter referred to as the "contact state") or a state in which the pen tip 31 is not in contact with the display contact surface (hereinafter referred to as the "non-contact state").

In this embodiment, the input stroke data is described as including information on the lateral position of the pen tip 31 recognized by the pen tablet 20 (display panel 21) (hereinafter referred to as the "sample" or "sample position") and information on the pen tip state (the vertical position of the pen tip 31) recognized by the pen tablet 20 (display panel 21).

In the following, as shown in FIG. 1, the horizontal direction (left-right direction) of the pen tablet 20 (display panel 21) as the "screen/touch panel/pointing device" is taken as the X axis, and the vertical direction (up-down direction) of the pen tablet 20 (display panel 21) as the "screen/touch panel/pointing device" is taken as the Y axis, and the coordinate system of the sample position in the input stroke data is expressed. In this embodiment, the coordinate system of the sample position has the upper left point of the pen tablet 20 (display panel 21) as the origin (x=0, y=0), the downward direction is the "direction in which the Y coordinate increases (+Y direction)", and the right direction is the "direction in which the X coordinate increases (+X direction)". Therefore, in the following, the upward direction is the "direction in which the Y coordinate decreases (-Y direction)", and the leftward direction is the "direction in which the X coordinate decreases (-X direction)". In addition, in the following, the positions along the trajectory of the sample positions (samples) for each time series (including positions on the normalized area described later) are collectively referred to as "feature points".

Next, we will provide an overview of the character recognition process performed by the online AI processing unit 1142.

First, we will explain an example of the configuration of input vector data processed by the online AI processing unit 1142.

In this embodiment, the input vector data is data composed of M-dimensional vectors corresponding to each of the N feature points for each time series per character (i.e., data represented by a matrix of M columns by N rows).

In this embodiment, the input vector data includes, for each feature point, the coordinates of the current feature point, the amount of movement (vector) to the next feature point in the time series, and information indicating the pen tip state.

In this embodiment, the input vector data is assumed to be vector data represented by seven-dimensional parameters (M=7) as shown in FIG. 2.

In this embodiment, the input vector data is described as including the X coordinate of the feature point (hereinafter, represented as "VX"), the Y coordinate of the feature point (hereinafter, represented as "VY"), the rightward movement amount from the feature point to the feature point of the next time series (hereinafter, represented as "VR"), the upward movement amount from the feature point to the feature point of the next time series (hereinafter, represented as "VU"), the leftward movement amount from the feature point to the feature point of the next time series (hereinafter, represented as "VL"), the downward movement amount from the feature point to the feature point of the next time series (hereinafter, represented as "VD"), and a value indicating the pen tip state in the section between the feature point and the next time series (hereinafter, represented as "VT"). Note that VT is set to either "1" which indicates a state in which the pen tip 31 of the electronic pen 30 is in contact with the display contact surface (hereinafter, referred to as "contact state"), or "0" which indicates a state in which the pen tip 31 is not in contact (hereinafter, referred to as "non-contact state"). Therefore, if the feature amount of a time series t (t is any integer from 1 to N) is V(t), then V(t) can be expressed as in the following equation (1). If the input vector data for one character is Z, then Z can be expressed by a determinant such as the following equation (2).
(t)={VX(t), VY(t), VR(t), VU(t), VL(t), VD(t), VT(t)} …(1)

As described above, among the feature quantity V(t), VR(t), VU(t), VL(t), and VD(t) represent the amount of movement to the feature point of the next time series t+1. In this embodiment, in the coordinate system representing the input vector data, the upper left corner is the origin (X=0, Y=0), and the X value increases to the right and the Y value increases to the bottom. In this case, the direction in which Y increases on the Y axis (+Y direction) is the "downward direction" and the direction in which Y decreases (-Y direction) is the "upward direction". Also, the direction in which X increases on the X axis (+X direction) is the "rightward direction" and the direction in which X decreases (-X direction) is the "leftward direction". In this case, VR(t) represents the amount of movement in the +X direction, so if VX(t+1)>VX(t), then VR(t)=VX(t+1)-VX(t), and if VX(t+1)≦VX(t), then VR(t)=0. Furthermore, since VL(t) represents the amount of movement in the -X direction, if VX(t+1)<VX(t), then VL(t)=VX(t)-VX(t+1), and if VX(t+1)≧VX(t), then VL(t)=0. Furthermore, since VU(t) represents the amount of movement in the -Y direction, if VY(t+1)<VY(t), then VU(t)=VY(t)-VY(t+1), and if VY(t+1)≧VY(t), then VU(t)=0. Furthermore, since VD(t) represents the amount of movement in the +Y direction, if VY(t+1)>VY(t), then VD(t)=VY(t+1)-VY(t), and if VY(t+1)≦VY(t), then VD(t)=0.

In this embodiment, the amount of movement (amount of movement) in the input vector data is expressed in four dimensions (VR(t), VU(t), VL(t), VD(t)), but it may be expressed in two dimensions (horizontal and vertical) if negative values are acceptable. By expressing the amount of movement in the input vector data in four dimensions (VR(t), VU(t), VL(t), VD(t)), even changes on the same axis can be expressed as features of separate items, which can affect the accuracy of machine learning (i.e., adjustment of the judgment process by AI). For example, in this embodiment, the amount of movement of the input vector data can be expressed in either two dimensions (x(t), Y(t)) or four dimensions (VR(t), VU(t), VL(t), VD(t)), and the processing using the one with better recognition accuracy can be performed by the operator's operation, etc.

Next, a specific example of the feature quantity V(t) at time t that constitutes the input vector data will be described with reference to Figures 3 and 4.

Figure 3 shows the feature quantity V(1) when the feature points at time t = 1 and t = 2 are P1 and P2, respectively. In Figure 3, the coordinates of P1 are (X, Y) = (1, 2), and the coordinates of P2 are (X, Y) = (2, 1).

Then, the feature amount V(1) can be expressed as the following equation (3).
V(1)={VX(1), VY(1), VR(1), VU(1), VL(1), VD(1), VT(1)}
={1,2,1,1,0,0,1}...(3)

Figure 4 shows feature quantities V(1), V(2), and V(3) when feature points at times t=1, t=2, t=3, and t=4 are P1, P2, P3, and P4, respectively. In addition, in Figure 4, the coordinates of P1 are (X,Y)=(1,3), the coordinates of P2 are (X,Y)=(3,1), the coordinates of P3 are (X,Y)=(1,1), and the coordinates of P4 are (X,Y)=(3,3). In Figure 4, the section between P2 and P3 is considered to be a section in a non-contact state.

Then, the feature amounts V(1), V(2), and V(3) can be expressed by the following equations (4) to (6), respectively.
V(1)={VX(1), VY(1), VR(1), VU(1), VL(1), VD(1), VT(1)}
={1,3,2,1,0,0,1}...(4)
V(2)={VX(2), VY(2), VR(2), VU(2), VL(2), VD(2), VT(2)}
={3,1,0,0,2,0,0}...(5)
V(3)={VX(3), VY(3), VR(3), VU(3), VL(3), VD(3), VT(3)}
={1,1,2,0,0,2,1}...(6)

(A-2) Operation of the First Embodiment Next, the operation of the recognition processing system 1 of this embodiment having the above-mentioned configuration (each procedure of the recognition processing method according to the embodiment) will be described.

First, we will explain an example of a process in which the recognition processing system 1 accepts character input from a user using the pen tablet 20 (display panel 21) and electronic pen 30.

As described above, in the recognition processing system 1 of this embodiment, regardless of whether the system is operating in the learning mode or the recognition processing mode, the content processing unit 111 displays an operation screen (hereinafter referred to as the "character input screen") on the pen tablet 20 (display panel 21) for accepting character input from the user (character input using the electronic pen 30).

Operation screens of various configurations can be used as the character input screen, but for example, an operation screen such as that shown in FIG. 5 may be used.

In the character input screen shown in FIG. 5, rectangular areas (hereinafter referred to as "character input fields") that can accept character input are arranged. In FIG. 5, one character input field F101 is arranged. In FIG. 5, characters that request the user to input characters (characters that function as correct labels in learning processing and judgment processing) are added to the character input field. For example, the character "10" (kanji) is added to each character input field F101. As a result, the recognition processing system 1 can accept input of the added character (input using the electronic pen 30) from the user in the frame of each character input field. In FIG. 5, the character "10" is handwritten with the electronic pen 30 in the frame of the character input field F101. Note that, in FIG. 5, an example in which one character input field is arranged on one screen is shown for the sake of simplicity of explanation, but the layout of such a character input field is not limited, and it is natural that multiple character input fields may be arranged.

For ease of explanation, the character input screen of this embodiment will be described using an example in which one area (character input field) is set for one character input, but the recognition processing system 1 may also accept multiple character inputs in one area and perform character-by-character extraction.

In this embodiment, the content processing unit 111 can present a character input screen such as that shown in FIG. 5 to the user, accept writing input from the user using the electronic pen 30, and obtain input stroke data corresponding to the input character based on the input.

In this embodiment, the content processing unit 111 is described as acquiring, as sample positions, the coordinates when the pen tip state changes from a non-contact state to a contact state, the coordinates for each predetermined period (e.g., about 0.1 seconds) while the pen tip state is in the contact state, and the coordinates when the pen tip state changes from a contact state to a non-contact state.

Figure 6 shows an example of the configuration of input stroke data acquired by the stroke data processing unit 1141 when the character "10" is handwritten with the electronic pen 30 within the frame of the character input field F101 as shown in Figure 5.

Figure 6(a) is a diagram depicting sample points when the character "10" is handwritten with the electronic pen 30. In the coordinate system shown in Figure 6(a), the X range is 0 to 1000, and the Y range is 0 to 10000. In other words, the image of the input stroke data shown in Figure 6 is an image of 1000 pixels x 1000 pixels.

Figure 6(b) is a diagram showing the values for each time series number of the input stroke data shown in Figure 6(b). The time series number is a value that indicates the order of the time series, with smaller values representing earlier time series (times). As shown in Figure 6(b), the input stroke data records information on the X and Y coordinate values and pen tip state value for each time series number.

At this time, the stroke data processing unit 1141 manages the data of each sample (feature point) of the input stroke data by dividing it into strokes (strokes). For example, by using data with a pen tip state of "0" as a boundary in the input stroke data, a list of sample positions for each stroke (stroke) can be obtained. For example, in FIG. 6(a), the 14th sample position from the top has a pen tip state of "0", so the sample positions from 1 to 13 in the time series are the first stroke, and the sample positions from 14 to 23 in the time series are the second stroke.

Next, the process in which the stroke data processing unit 1141 generates input vector data for online character recognition processing from input stroke data will be explained using the flowchart in FIG. 7.

First, input stroke data for one character is supplied from the content processing unit 111 to the stroke data processing unit 1141 and stored (S101).

Next, the stroke data processing unit 1141 acquires data (hereinafter referred to as "first normalized stroke data") that has been normalized so that the input stroke data fits exactly within an area of a predetermined resolution (hereinafter referred to as "normalized area") (S102).

Figure 8 shows the process in which the stroke data processing unit 1141 normalizes input vector data for online character recognition processing to obtain first normalized stroke data.

Figure 8 (a) is an image that has been cut out from the input stroke data shown in Figure 6, showing only the area where the sample positions (feature points) are drawn.

In the input stroke data shown in FIG. 6, the maximum X value is 635 and the minimum X value is 427. Also, in the input stroke data shown in FIG. 6, the maximum Y value is 658 and the minimum Y value is 388. Therefore, the image in FIG. 8(a) (the cropped image) is an image of 208 pixels x 270 pixels (the number of pixels in the X direction is 208, and the number of pixels in the Y direction is 270).

Figure 8 (b) shows the image of Figure 8 (a) converted into a normalized region of 100 pixels x 100 pixels (a region with an aspect ratio of 1:1).

Figure 8 (c) is a diagram showing data (first normalized stroke data) corresponding to each feature point (each pixel) of the image in the normalized area of Figure 8 (b).

As shown in FIG. 8B and FIG. 8C, when the stroke data processing unit 1141 normalizes the input stroke data to a normalized area of 100 pixels x 100 pixels, it provides a margin of two pixels at the top, bottom, left, and right edges. That is, the stroke data processing unit 1141 essentially performs a process of normalizing the input stroke data to an area of 96 pixels x 96 pixels. In the example of FIG. 8, the stroke data processing unit 1141 obtains the coordinates of each feature point after a resolution conversion process that converts the image of the input stroke data (an image of 208 pixels x 270 pixels) into an image of 96 pixels x 96 pixels (an image with an aspect ratio of 1:1), thereby obtaining the first normalized stroke data shown in FIG. 8C. At this time, various image processing methods can be applied to the specific method of the resolution conversion process performed by the stroke data processing unit 1141, so a description of the specific process will be omitted.

Next, the stroke data processing unit 1141 performs a process of thinning out feature points from the first normalized stroke data so that the distance between each feature point and adjacent feature points is equal to or greater than a predetermined distance (S103).

For example, the following formula (7) may be calculated based on the feature points of time series t and the feature points of the next time series t+1, and if the formula holds, it may be determined that the distance between the two feature points is equal to or smaller than a predetermined value.

In equation (7), the x-coordinate of a feature point in time series t is x(t) and the y-coordinate is y(t), and the x-coordinate of a feature point in the time series t+1 that follows time series t is x(t+1) and y(t+1).
In addition, in formula (7), SIZE is the horizontal and/or vertical resolution of the entire image (here, 100). Here, a description will be given of applying SIZE/100=100/10=10 as the threshold value (right side of formula (7)) for determining whether or not to thin out feature points, but this threshold value may be set to an arbitrarily designed value (for example, a suitable value may be searched for and set through experiments, etc.).

Here, the stroke data processing unit 1141 applies the following formula (7) to the feature points of each time series to perform calculations, and if the formula holds, thins out the feature points of the next time series. When the stroke data processing unit 1141 has thinned out the feature points of time series t+1, it may also calculate the feature points of the next time series as time series t+1 by applying the following formula (7) again, and repeat the thinning process if the formula holds.

The stroke data processing unit 1141 may also perform the thinning process for each stroke. In other words, the stroke data processing unit 1141 may repeat the thinning process for each stroke until the following formula (7) is not satisfied for all feature points in time series (because the distances between all feature points are equal to or greater than a predetermined value). For example, the stroke data processing unit 1141 may extract the data of the first stroke (data of time series numbers 1 to 13) from the first normalized stroke data shown in FIG. 8C and perform the above-mentioned thinning process, and then extract the data of the second stroke (data of time series numbers 14 to 22) and perform the above-mentioned thinning process.
{x(t+1)-x(t)} ² +{y(t+1)-y(t)} ² >SIZE/10...(7)

Figure 9 shows the state after feature points have been thinned out from the first normalized stroke data shown in Figures 8(b) and 8(c) using the above process.

When inputting characters using the electronic pen 30, when the non-contact state changes to a contact state, the pen tip 31 of the electronic pen 30 slides on the display panel 21, generating dense feature points. However, since these dense feature points do not form the shape of a character, they become noise in the data for character recognition. Therefore, the stroke data processing unit 1141 performs the above-mentioned thinning process on the input stroke data to remove noise and improve learning accuracy and recognition accuracy.

Figure 9(a) is an image showing the first normalized stroke data after the thinning process. Figure 9(b) shows the first normalized stroke data after the thinning process.

In the first normalized stroke data shown in FIG. 9, the feature points have been thinned from 22 before the thinning process to 16. As shown in FIG. 9, when the stroke data processing unit 1141 thins out feature points from the first normalized stroke data, it renumbers the time series numbers so that there are no gaps.

In the normalized stroke data shown in FIG. 9, the feature points with time series numbers 1 to 9 are the feature points of the first stroke, and the feature points with time series numbers 10 to 16 are the feature points of the second stroke. The stroke data processing unit 1141 manages a list of the feature points of each stroke before and after thinning.

Next, the stroke data processing unit 1141 generates data (hereinafter referred to as "second normalized stroke data") by performing a process of interpolating feature points (hereinafter referred to as "feature point interpolation process") based on the thinned-out first normalized stroke data so that the distance between each feature point is equal to or less than a predetermined value, including non-contact sections (sections between each stroke) (S104).

For example, the following formula (8) may be calculated based on a feature point of a time series t and a feature point of the next time series t+1. If the formula is satisfied, a new feature point (a feature point in a time series between two feature points) may be interpolated (added) to a position between the two feature points (e.g., an intermediate position).
{x(t+1)-x(t)} ² +{y(t+1)-y(t)} ² >SIZE/10...(8)

Here, we will explain how to apply SIZE/100 = 100/10 = 10 as the threshold value (right side of equation (8)) for determining whether or not to thin out sample positions, but this threshold value may be set to an arbitrarily designed value (for example, by searching for and setting a suitable value through experiments, etc.).

Here, the stroke data processing unit 1141 applies equation (8) to the sample positions of each time series to perform calculations, and if the equation holds, it interpolates a new feature point between the sample positions and the next time series.

In this case, the x coordinate of the newly added feature point may be set to "{x(t+1)+x(t)}/2" (i.e., the average value of x(t+1) and x(t)), and the y coordinate may be set to "{y(t+1)+y(t)}/2" (i.e., the average value of y(t+1) and y(t)).

The stroke data processing unit 1141 recursively performs feature point interpolation processing for each stroke until equation (8) is no longer satisfied for all sample positions in the time series. For example, when the stroke data processing unit 1141 interpolates a new feature point between a feature point in time series t and a feature point in time series t+1, it may perform calculations by applying equation (8) to the time series of the added feature point as t+1 again, and if equation (8) is satisfied, repeat the process of interpolating the new feature point.

For example, the stroke data processing unit 1141 extracts the first stroke data (data with time series numbers 1 to 13) from the first normalized stroke data shown in FIG. 8(c) and performs the above-mentioned feature point interpolation process, and further extracts the second stroke data (data with time series numbers 14 to 22) and performs the above-mentioned feature point interpolation process.

The stroke data processing unit 1141 also performs the above-mentioned feature point interpolation process between strokes, filling in the spaces between strokes with feature points at a predetermined interval. For example, the stroke data processing unit 1141 performs the above-mentioned feature point interpolation process between the last time-series feature point of the first stroke and the first time-series feature point of the second stroke, thereby filling in the space between the end of the first stroke and the beginning of the second stroke with feature points at a predetermined interval. In other words, the stroke data processing unit 1141 performs feature point addition process on the space between the end of the first stroke and the beginning of the second stroke as one stroke.

The stroke data processing unit 1141 further assigns information on the pen tip state to each feature point. Specifically, the stroke data processing unit 1141 assigns "1" representing a contact state to the pen tip information of feature points in a contact state (feature points belonging to each stroke), and assigns "0" representing a non-contact state to pen tip information of feature points in a non-contact state (feature points in the section between strokes).

Figure 10 shows the process of performing the above-mentioned feature point interpolation process on the thinned first normalized stroke data shown in Figure 9 to obtain second normalized stroke data.

In FIG. 10, feature points with time series numbers 1 to 40 are set as the first stroke, and feature points with time series numbers 63 to 102 are set as the second stroke. In FIG. 10, feature points with time series numbers 41 to 62 between the first and second strokes are set as the non-contact state section.

Next, the stroke data processing unit 1141 generates data (hereinafter referred to as "third normalized stroke data") by extracting N+1 feature points from the second normalized stroke data (S105).

Here, the number of feature points in the second normalized stroke data is represented as "C".

If C>N+1, the stroke data processing unit 1141 extracts (selects) N+1 feature points from the second normalized stroke data to generate third normalized stroke data.
Furthermore, when N+1>C, the stroke data processing unit 1141 selects some or all of the feature points of the second normalized stroke data multiple times to extract data of N+1 feature points and generate third normalized stroke data. It is desirable that the third normalized stroke data have as little bias in the distances between the feature points as possible. The stroke data processing unit 1141 is not limited to a particular method for extracting N+1 feature points from the second normalized stroke data, but it is possible to reduce bias in the distances between the feature points in the third normalized stroke data by performing the following process, for example.

Here, the stroke data processing unit 1141 selects the D(i)th time-series feature point of the second normalized stroke data as the i-th feature point (i is an integer from 1 to N) of the third normalized stroke data. For example, the following formula (9) can be applied as D(i). In other words, D(i) is an integer obtained by rounding down the decimal point from C/(N+1) multiplied by i.
D(i)=[{C/(N+1)}*(i-1)]+1...(9)

Figure 11 shows an example of third normalized stroke data obtained by extracting N+1 feature points from the second normalized stroke data shown in Figure 10.

Figure 12 shows the third normalized stroke data shown in Figure 11 in the form of an image.

For example, the second normalized stroke data shown in FIG. 10 is composed of 102 feature points, so 101 are extracted from it using formula (9). For example, D(1) = 1, D(2) = 2, ..., D(99) = 99, D(100) = 100, D(101) = 102, so only the 101st feature point of the second normalized stroke data is not selected (extracted) (skipped).

Next, we will explain an example where N+1<C. If C=30, then D(1)=1, D(2)=1, D(2)=1, D(3)=1, D(4)=2, ..., D(99)=29, D(100)=29, D(101)=30.

By using equation (9) as described above, it is possible to efficiently pick up the i-th feature point of the third normalized stroke data from the second normalized stroke data.

Next, the stroke data processing unit 1141 converts the second normalized stroke data into input vector data and obtains it (S105).

The stroke data processing unit 1141 sets the X coordinate, Y coordinate, and pen tip state of each time series constituting the second normalized stroke data to VX, VY, and VT of each time series, respectively. Then, as described above, the stroke data processing unit 1141 can obtain VR(t), VU(t), VL(t), and VD(t) based on VX(t), VX(t+1), VY(t), and VY(t+1). This allows the stroke data processing unit 1141 to obtain V(1) to V(100).

Figure 13 shows an example of input vector data obtained based on the second normalized stroke data shown in Figure 11.

Next, the process in which the stroke data processing unit 1141 generates input image data for offline character recognition processing from input stroke data will be explained using the flowchart in FIG. 14.

First, the stroke data processing unit 1141 is assumed to hold input stroke data for one character (S201).

Next, the stroke data processing unit 1141 acquires data (hereinafter referred to as "fourth normalized stroke data") that is normalized so that the input stroke data fits exactly within the normalized area of a specified resolution (S202).

Figure 15 shows the normalization process performed by the stroke data processing unit 1141 when generating input vector data for online character recognition processing.

Figure 15 (a) is an image that has been cut out from the input stroke data shown in Figure 6, showing only the area where the feature points (sample positions) are drawn.

Figure 15(b) shows the image of Figure 15(a) converted into a normalized region of 64 pixels x 64 pixels (region with an aspect ratio of 1:1).

Figure 15(c) is a diagram showing normalized stroke data (fourth normalized stroke data) corresponding to each feature point (each pixel) of the image in the normalized area of Figure 15(b).

As shown in FIG. 15(b) and FIG. 15(c), when the stroke data processing unit 1141 normalizes the input stroke data to a normalized area of 64 pixels x 64 pixels, it provides a margin of two pixels at the top, bottom, left, and right edges. That is, the stroke data processing unit 1141 essentially performs a process of normalizing the input stroke data to an area of 60 pixels x 60 pixels. In the example of FIG. 15, the stroke data processing unit 1141 obtains the coordinates of each feature point after a resolution conversion process that converts the image of the input stroke data (an image of 208 pixels x 270 pixels) into an image of 60 pixels x 60 pixels (an image with an aspect ratio of 1:1), thereby obtaining the fourth normalized stroke data shown in FIG. 15(c). At this time, various image processing methods can be applied to the specific method of the resolution conversion process performed by the stroke data processing unit 1141, so a description of the specific process will be omitted.

Next, the stroke data processing unit 1141 performs a process of thinning out feature points from the fourth normalized stroke data so that the distance between each feature point and adjacent feature points is equal to or greater than a predetermined distance (S203).

The stroke data processing unit 1141 may apply a process for thinning out feature points from the fourth normalized stroke data in a manner similar to the process for thinning out feature points from the first normalized stroke data described above. For example, the stroke data processing unit 1141 may repeat the thinning process for each stroke until equation (7) is no longer satisfied for all time-series feature points (because the distance between all feature points is equal to or greater than a predetermined value). At this time, it is desirable for the stroke data processing unit 1141 to set SIZE when applying equation (7) to 64, which is the same as the resolution of the fourth normalized stroke data.

Next, the stroke data processing unit 1141 acquires input image data based on the fourth normalized stroke data after thinning out the feature points (S204).

For example, the stroke data processing unit 1141 may obtain data on the feature points of each stroke from the fourth normalized stroke data after thinning processing in an image area of 64 pixels x 64 pixels, and obtain input image data by drawing lines connecting the feature points of each stroke in the above image area.

Figure 16 shows an image of the input image data obtained based on the fourth normalized stroke data shown in Figure 15.

Next, the processing performed when the character recognition processing unit 114 operates in learning mode will be explained using FIG. 17.

Here, it is assumed that when the character recognition processing unit 114 is operating in the learning mode, the content processing unit 111 supplies the character recognition processing unit 114 with input stroke data for learning (input stroke data for one character) and a set of correct labels corresponding to the characters in the input stroke data.

First, the stroke data processing unit 1141 generates input vector data and input image data based on the supplied input stroke data, and supplies them to the online AI processing unit 1142 and the offline AI processing unit 1143, respectively (S301).

In the online AI processing unit 1142 of the character recognition processing unit 114 operating in learning mode, learning processing is performed based on the supplied input vector data and correct answer labels (S302).

In addition, in the offline AI processing unit 1143 of the character recognition processing unit 114 operating in the learning mode, learning processing is performed based on the supplied input image data and correct answer labels (S303).

As described above, in the character recognition processing unit 114, each time learning data is supplied from the content processing unit 111, a learning process is performed using the learning data set.

Next, the processing performed when the character recognition processing unit 114 operates in the recognition processing mode will be explained using FIG. 18.

Here, it is assumed that when the character recognition processing unit 114 is operating in character recognition mode, input stroke data for learning (input stroke data for one character) is supplied from the content processing unit 111 to the character recognition processing unit 114.

First, the stroke data processing unit 1141 generates input vector data and input image data based on the supplied input stroke data, and supplies them to the online AI processing unit 1142 and the offline AI processing unit 1143, respectively (S401).

The online AI processing unit 1142 of the character recognition processing unit 114 operating in character recognition mode performs character determination processing using the stored learning model based on the supplied input vector data, and supplies the determination result (online determination result) together with the reliability to the character recognition result output unit 1144 (S402).

The offline AI processing unit 1143 of the character recognition processing unit 114 operating in character recognition mode performs character determination processing using the stored learning model based on the supplied input image data, and supplies the determination result (offline determination result) together with the reliability to the character recognition result output unit 1144 (S403).

Next, the character recognition result output unit 1144 compares the reliability of the online judgment result and the offline judgment result, selects the judgment result with the greater reliability (S404), and outputs it (S405).

As described above, the character recognition processing unit 114 outputs a determination result each time input stroke data is supplied from the content processing unit 111.

(A-3) Advantages of the First Embodiment According to this embodiment, the following advantages can be achieved.

(A-3-1) First, the inventor actually constructed the character recognition processing unit 114 of the recognition processing system 1 and conducted an experiment (hereinafter referred to as "this experiment") on the character recognition accuracy (correctness rate of the judgment result) when learning processing and character recognition processing were performed. The contents and results of this experiment are described below.

In this experiment, learning and recognition processes were performed using "katakana," "hiragana," and "JIS level 1 kanji" characters (a total of 3,107 types of characters) as sample typefaces (hereinafter referred to as "sample typefaces"). In this experiment, 160 samples of input stroke data (input stroke data input by a human into the pen tablet 20 using the electronic pen 30) and correct answer labels were prepared for each character in the sample typeface, and supplied to the character recognition processing unit 114 operating in learning mode. As a result, the processing of the flowchart in Figure 17 above is performed for each sample of each sample typeface, and a learning model is obtained in the online AI processing unit 1142 and the offline AI processing unit 1143, respectively.

In this experiment, in order to confirm the recognition accuracy after the learning process is completed in the character recognition processing unit 114, 160 samples of input stroke data and correct answer labels for each character of the sample font were prepared and supplied to the character recognition processing unit 114 operating in the recognition processing mode. As a result, the recognition process of the flowchart in FIG. 18 was performed for each sample of each sample font. At this time, the inventor confirmed the accuracy rate of each of the online judgment result and the offline judgment result, and the accuracy rate of the judgment result output from the character recognition result output unit 1144 (a result of comprehensively judging the online judgment result and the offline judgment result). As a result of this experiment, the accuracy rate of the online judgment result alone was about 94%, and the accuracy rate of the offline judgment result alone was about 90%. And the judgment result output from the character recognition result output unit 1144 was about 98%. In other words, it was found that the recognition accuracy is higher when the final judgment result is output taking into account both the online judgment result and the offline judgment result.

(A-3-2) In the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, input vector data used in the online character recognition learning process and character recognition process is normalized and processed so that all of the input vector data has N feature points. This allows the information processing terminal 10 (character recognition processing unit 114) of the first embodiment to generate and process fixed-length input vector data for all characters, regardless of the number of strokes, etc. This is because, in general, in machine learning processes using neural networks, it is desirable to input fixed-length data.
It is possible to use an AI engine that supports variable-length input layers, or to configure a neural network with a fixed-length input layer that matches the longest data length (effectively using a fixed-length input layer with variable length), but there is a risk that processing efficiency and recognition accuracy will be unstable compared to when the process is completed with fixed-length input and output.

(A-3-3) In the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, not only coordinates (VX, VY) but also parameters related to the motion vector (VR, VU, VL, VD) and the state of the tip of the electronic pen 30 (contact state or non-contact state) are introduced as feature quantities constituting the input vector data. As a result, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, information on strokes while the tip of the electronic pen 30 is in a non-contact state is also acquired as feature quantities. In addition, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, it is not necessary to prepare a standard pattern for each character in advance as in the technology described in Patent Document 1, or to perform processing such as matching feature points with all standard patterns during character recognition. As described above, in the information processing terminal 10 (character recognition processing unit 114) of the first embodiment, efficient character recognition processing can be performed while increasing the amount of information acquired about strokes during character input.

(B-1) Second embodiment Hereinafter, a second embodiment of the recognition processing device, the recognition processing program, the recognition processing method, and the recognition processing system according to the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which an information processing terminal is configured as the recognition processing device of the present invention will be described.

The characters according to the second embodiment can also be shown using FIG. 1. Note that the symbols in parentheses in FIG. 1 are symbols used only in the second embodiment.

The following explains the differences between the characters in the second embodiment and the first embodiment.

The recognition processing system 1A of the second embodiment differs from the first embodiment in that the information processing terminal 10 is replaced with an information processing terminal 10A. Also, in the information processing terminal 10A of the second embodiment, the control unit 11 is replaced with a control unit 11A. Furthermore, in the control unit 11A of the second embodiment, the content processing unit 111 and the character recognition processing unit 114 are replaced with a content processing unit 111A and a character recognition processing unit 114A, respectively, which is different from the first embodiment. Furthermore, in the character recognition processing unit 114A of the second embodiment, the stroke data processing unit 1141 is replaced with a stroke data processing unit 1141A, which is different from the first embodiment.

In the first embodiment, the pen tip state of the electronic pen 30 is described as being either a contact state in which the pen tip 31 of the electronic pen 30 is in contact with the display panel 21 of the pen tablet 20, or a non-contact state in which the pen tip 31 of the electronic pen 30 is not in contact with the display panel 21 of the pen tablet 20. However, depending on the combination of devices applied to the pen tablet 20 and the electronic pen 30, there are some that are capable of tracking the lateral position of the pen tip 31 of the electronic pen 30 even in a non-contact state. For example, when applying a combination of a pen tablet and a stylus pen manufactured by Wacom (registered trademark), even if the stylus pen is in a non-contact state, the position (lateral position) of the pen tip can be tracked on the pen tablet as long as the height of the pen tip is equal to or less than a predetermined value.

Therefore, in this embodiment, the pen tablet 20 is described as being configured to be able to detect the lateral position (coordinates) of the pen tip 31 even when the pen tip 31 of the electronic pen 30 is in a non-contact state, as long as the height of the pen tip 31 is equal to or less than a predetermined value. In the second embodiment, when the pen tip state of the electronic pen 30 is in a non-contact state and the pen tablet 20 is able to track the lateral position of the pen tip 31, this state (pen tip state) is referred to as a "hover state." Also, in the second embodiment, when the pen tip state of the electronic pen 30 is in a non-contact state and the pen tablet 20 is unable to track the lateral position of the pen tip 31, this state (pen tip state) is referred to as a "loss state."

Figures 19 and 20 are timing charts showing the height of the pen tip 31 and the state of the pen tip over time when a user uses the electronic pen 30 to draw a Chinese character with two strokes (such as "八") on the pen tablet 20.

In Figure 19, the horizontal axis represents time t, and the vertical axis represents the height of the pen tip 31 of the electronic pen 30 (height over time). In Figure 19, the position of the pen tip 31 at each of times t0 to t21 is represented by the position of the tip of a wedge-shaped symbol (a wedge-shaped symbol with the tip facing downward). Here, times t0 to t21 are described as indicating the timing at which the pen tablet 20 detects (samples) the coordinates, etc., of the electronic pen 30 (pen tip 31).

In addition, in FIG. 19, the symbols of the times when the pen tip 31 is in contact are colored black, the symbols of the times when the pen tip 31 is in hover are hatched (diagonal lines), and the outlines of the symbols of the times when the pen tip 31 is in loss are dashed.

In FIG. 19, times t0 to t4 indicate drawing the first stroke, and the pen tip 31 of the electronic pen 30 is in a contact state. During the contact state, the coordinates and writing pressure of the electronic pen 30 (pen tip 31) are acquired by a sensor in the pen tablet 20.

In FIG. 19, from time t5 to t7, the first stroke is drawn and the pen tip 31 of the electronic pen 30 is in a hover state. As described above, with a Wacom pen tablet or the like, it is possible to acquire coordinates and detect the presence of a stylus pen even when the stylus pen is floating a certain distance above the panel of the pen tablet. In the case of a hover state, a pen pressure value of "0" (in other words, a value indicating a hover) is acquired with a Wacom pen tablet.

In FIG. 19, from time t8 to time t11, the user further removes the pen tip 31 of the electronic pen 30 from the pen tablet 20, and the electronic pen 30 (pen tip 31) is in a loss state.
During the loss state, the pen tablet 20 cannot detect the coordinates of the electronic pen 30 (pen tip 31 ).

In FIG. 19, from time t12 to t13, the user again brings the electronic pen 30 (pen tip 31) close to the pen tablet 20 in order to draw the second stroke, resulting in a hover state. Then, from the following time t14 to t19, the user starts drawing the second stroke, so the electronic pen 30 (pen tip 31) comes into contact with the pen tablet 20, resulting in a contact state.

The example in Figure 20 is the same as the example in Figure 19, except that timings t8 to t12 are in a hover state rather than a loss state.

Figure 21 shows the tally of the pen tip state at each sample (characteristic point) shown in the timing chart of Figure 19. As shown in Figure 21, in the example of Figure 19, 16 samples (coordinates of the electronic pen 30) are obtained over the course of 20 samples, excluding the period of the loss state (times t8 to t11). Also, as shown in Figure 21, in the example of Figure 19, of the 16 samples obtained, 11 are samples in the contact state and 5 are samples in the hover state.

Figure 22 shows each sample (feature point) shown in the timing chart of Figure 19. Figure 22 shows the case where the user draws the kanji character "八".

In Figure 22, feature points in the contact state are indicated by circular (○) symbols, feature points in the hover state are indicated by triangular (△) symbols, and positions in the loss state are indicated by rectangular (□) symbols. Note that in the case of a loss state, the coordinates of feature points cannot be acquired by the pen tablet 20, but in Figure 22, the positions in the case where the coordinates (horizontal position of the pen tip 31) could be acquired are shown by rectangular symbols. In the following, the x coordinates of each feature point from t0 to t20 are represented as x0 to x20, and the y coordinates are represented as y0 to y20.

As described above, the pen tablet 20 of the second embodiment is capable of detecting the contact state, hover state, or loss state of the electronic pen 30.

In the information processing terminal 10 (control unit 11) of the first embodiment, input stroke data consisting only of contact state samples (coordinates) was normalized to generate input vector data for the online AI processing unit 1142. In contrast, the information processing terminal 10A (control unit 11A) of the second embodiment differs from the second embodiment in that the input stroke data for the online AI processing unit 1142 can reflect three statuses: contact state, hover state, and loss state. Note that in the second embodiment, the configuration and normalization process of the input stroke data for the offline AI processing unit 1143 can be applied in the same manner as in the first embodiment, so a description thereof will be omitted here.

Next, examples of methods for configuring and normalizing input stroke data for the online AI processing unit 1142 in the second embodiment will be described. In the second embodiment, the following five methods can be given as examples of methods for configuring and normalizing input stroke data for the online AI processing unit 1142.

[First normalization method]
In the first normalization method, input stroke data is constructed using only the feature points (coordinates) of the contact state, and normalized to N+1 feature points. That is, in the example of Fig. 19 and Fig. 21, the input stroke data is expressed using a total of nine sample feature points from t0 to t4 and t14 to t19. In this case, assuming that normalization is performed with N=100, the feature points of the input stroke data are upsampled by about 11 times. In the first normalization method, as in the first embodiment, the input stroke data is processed (thinning out and interpolating feature points) for each stroke (section in which the pen tip state is in contact) and normalized to obtain input vector data.

[Second normalization method]
In the second normalization method, input stroke data is constructed by extracting and adding a portion (e.g., only one sample) of the hover state samples to the contact state samples (feature points), and normalized. In the second normalization method, for example, from the hover state samples, a sample at either the timing before or after the period of the loss state (e.g., the sample at time t7 or t12 in the example of FIG. 19) may be extracted and added to the input stroke data, or samples both before and after the period of the loss state (e.g., the samples at times t7 and t12 in the example of FIG. 19) may be extracted and added to the input stroke data.

FIG. 23 is a diagram showing an example of input stroke data when the second normalization method is applied to each sample (feature point) shown in the timing chart of FIG.
23 shows an example of input stroke data constructed by adding samples from both before and after the period of the loss state (samples at times t7 and t12) to the contact state sample. In FIG. 23, the hover state is represented by "0" in the pen tip state item.

When the second normalization method is applied, the stroke data processing unit 1141A may perform normalization processing on the section between strokes (between contact state sections) (a section consisting of a hover state and a loss state) in the same way as for strokes (contact state sections) to obtain input vector data. For example, in the case of input stroke data as shown in FIG. 23, the stroke data processing unit 1141A may perform normalization processing on the section between the last feature point of the first stroke and the first feature point of the second stroke in the same way as for one stroke.

When the stroke data processing unit 1141A applies the second normalization method, if feature points are interpolated during a non-contact state (hover state and loss state period) or immediately before or after the non-contact state period, the pen tip state of those feature points in the input vector data is set to "0".

[Third normalization method]
In the third normalization method, the input stroke data is normalized by constructing the input stroke data by adding all the hover state samples to the contact state samples. For example, in the examples of Figures 19 and 21, 16 samples of the contact state or the hover state may be included in the input stroke data.

When the stroke data processing unit 1141A applies the third normalization method, the section between strokes (between contact state sections) (a section consisting of a hover state and a loss state) may also be normalized in the same way as the strokes (contact state sections) to obtain input vector data.

Figure 24 shows an example of input stroke data when the third normalization method is applied to each feature point shown in the timing chart of Figure 19. In Figure 24, the pen tip state in the hover state is set to "0".

When the second normalization method is applied, the stroke data processing unit 1141A may perform normalization processing on the section between strokes (between contact state sections) (a section consisting of a hover state and a loss state) in the same way as for strokes (contact state sections) to obtain input vector data. For example, in the case of input stroke data as shown in FIG. 23, the stroke data processing unit 1141A may perform normalization processing on the section between the last feature point of the first stroke and the first feature point of the second stroke in the same way as for one stroke.

In the third normalization method, when feature points are interpolated during a hover state period and before and after a hover state period, the pen tip state of those feature points is set to "0" in the input vector data. In the stroke data processing unit 1141A, when the third normalization method is applied, when feature points are interpolated during a non-contact state period (hover state and loss state periods) or immediately before or after the non-contact state period, the pen tip state of those feature points is set to "0" in the input vector data.

[Fourth normalization method]
In the fourth normalization method, the input stroke data is normalized by forming all samples (contact state, hover state, and loss state). For example, in the examples of Figures 19 and 21, all 20 samples may be included in the input stroke data.

Figure 25 shows an example of input stroke data when the fourth normalization method is applied to each feature point shown in the timing chart of Figure 19.

In FIG. 25, the pen tip state in the hover state is designated as "0", and the pen tip state in the loss state is designated as "2". In FIG. 25, the x coordinates of times t8 to t11 in the loss state are designated as c_x8 to c_x11, and the y coordinates are designated as c_y8 to c_y11. In FIG. 25, the coordinates (X and Y coordinates) of times t8 to t11 in the loss state may be set to coordinates of positions interpolated (linearly interpolated) between the coordinates of the feature points of the preceding and succeeding hover periods. In the example of FIG. 25, the coordinates of t8 to t11 may be set at equal intervals on a line connecting the coordinates (x7, y7) of t7 immediately before the loss state period and the coordinates (x12, y12) immediately after.

When the fourth normalization method is applied, the stroke data processing unit 1141A performs normalization processing (feature point thinning and interpolation processing) on the hover state section and the loss state section in the same way as the image (contact state section) to acquire input vector data. For example, in the case of input stroke data as shown in FIG. 25, the stroke data processing unit 1141A may perform normalization processing (feature point thinning and interpolation processing) on the hover state section from time t5 to t7, the loss state section from time t8 to t11, and the hover section from time t12 to t14 in the same way as the image section (contact state section) to connect them.

In addition, in the fourth normalization method, when feature points are interpolated during a hover state period or a loss state period, the pen tip state value on the input vector data may be set according to the following rules. In the fourth normalization method, when feature points are interpolated during a loss state period and immediately before or immediately after a loss state period, the pen tip state of those feature points may be set to "2" (loss state) in the input vector data. Furthermore, in the fourth normalization method, when feature points are interpolated during a hover state period immediately before a loss state period and immediately before the hover state period, the pen tip state of those feature points may be set to "0" (hover state) in the input vector data. In addition, in the fourth normalization method, when feature points are interpolated during a hover state period immediately after a loss state period and immediately after the hover state period, the pen tip state of those feature points may be set to "0" (hover state) in the input vector data.

As described above, when the fourth normalization method is applied, in the example of the timing chart in FIG. 19, input stroke data is generated using all samples (20 samples), including those in the loss state, so that normalization requires only about five times the interpolation (dilution) of feature points, allowing more accurate information (more information volume) to be included in the input vector data. In other words, when the fourth normalization method is applied, in the example of the timing chart in FIG. 19, information distinguishing between the hover state and the loss state can be included in the input vector data. As a result, with the fourth normalization method, the amount of information in the input vector data can be increased, which can improve recognition accuracy depending on the learning environment and recognition environment.

[Fifth normalization method]
In the fifth normalization method, input stroke data is constructed and normalized for all samples (contact state, hover state, and loss state) in the same way as in the fourth normalization method. However, the fifth normalization method differs from the fourth normalization method in that the loss state and the hover state are treated uniformly. Here, the fifth normalization method will be described assuming that the loss state and the hover state are all treated as the hover state.

Figure 26 shows an example of input stroke data when the fifth normalization method is applied to each feature point shown in the timing chart of Figure 19.

Figure 26 differs from Figure 25 (fourth normalization method) in that the pen tip state during the loss state is also set to "0" like the hover state.

In the fifth normalization method, when feature points are interpolated during a period consisting of a hover state and a loss period (in FIG. 19, the period from time t5 to t13) and immediately before or after that period, the pen tip state of those feature points in the input vector data is set to "0".

As a result, in the fifth normalization method, even if the height (detection range) at which coordinates can be obtained in the hover state varies due to differences in the capabilities of the sensor (sensor that detects the electronic pen 30) of the pen tablet 20, and the state of Figure 19 and the state of Figure 20 are mixed, the contents of the input vector data will be approximately the same, making it possible to perform stable learning and recognition processes.

(C) Other Embodiments The present invention is not limited to the above-described embodiments, and modified embodiments such as those exemplified below can also be mentioned.

(C-1) In each of the above embodiments, the information processing terminal 10 and the pen tablet 20 (display panel 21) are configured as separate devices, but they may be configured as an integrated device. For example, the information processing terminal 10 may be configured using a computer (e.g., a tablet terminal or a smartphone) equipped with a touch panel display.

(C-2) In each of the above embodiments, the character recognition processing unit 114 has both an online AI processing unit 1142 and an offline AI processing unit 1143, but it may also be configured to have only the online AI processing unit 1142.

1...recognition processing system, 10...information processing terminal, 11...control unit, 111...content processing unit, 112...display driver, 113...pen tablet driver, 114...character recognition processing unit, 1141...stroke data processing unit, 1142...online AI processing unit, 1143...offline AI processing unit, 1144...character recognition result output unit, 12...video IF, 13...USB port, 20...pen tablet, 21...display panel, 30...electronic pen, 31...pen tip.

Claims (8)

a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character, the input stroke data including position information of feature points in the time series order of the input pattern, and performing a normalization process for normalizing the acquired input stroke data into an input pattern having a fixed number of samples to acquire normalized stroke data;
an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples;
a character recognition processing means for performing character recognition processing on the input vector data acquired by the input vector data acquisition means, using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means,
The normalization process performed by the normalization means includes the following steps:
a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data;
and a second normalization processing step of acquiring data obtained by interpolating feature points of the intermediate stroke data such that intervals between all feature points adjacent in chronological order are equal to or smaller than the threshold value. The recognition processing device according to claim 1, characterized in that each feature constituting the input vector data includes a coordinate parameter indicating a coordinate corresponding to the time series of the feature, a motion vector parameter indicating a motion vector from the time series immediately preceding the feature, and a pen tip state parameter indicating the state of the pen tip of the electronic pen corresponding to the time series of the feature. The recognition processing device according to claim 2, characterized in that each feature constituting the input vector data includes a pen tip state parameter indicating the state of the pen tip of the electronic pen corresponding to the time series of the feature. The input vector data acquisition means further generates input image data drawn based on the input stroke data,
the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means using a first learning model previously trained by machine learning using input vector data to acquire a first character recognition result, and further performs character recognition processing on the input image data acquired by the input vector data acquisition means using a second learning model previously trained by machine learning using an input image to acquire a second character recognition result;
4. The recognition processing device according to claim 1, further comprising a character recognition result output means for selecting either the first character recognition result or the second character recognition result by the character recognition processing means and outputting the selected result as a final character recognition processing result. the character recognition processing means acquires a reliability of the first character recognition result or the second character recognition result when acquiring the first character recognition result or the second character recognition result;
5. The recognition processing device according to claim 4, wherein the character recognition result output means selects one of the first character recognition result and the second character recognition result by the character recognition processing means, which has a higher reliability, and outputs the selected result as a final character recognition processing result. Computer,
a normalization means for acquiring input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character, the input stroke data including position information of feature points in the time series order of the input pattern, and performing a normalization process for normalizing the acquired input stroke data into an input pattern having a fixed number of samples to acquire normalized stroke data;
an input vector data acquisition means for converting the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples;
a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means, the learning model functioning as a character recognition processing means that performs character recognition processing on the input vector data acquired by the input vector data acquisition means;
The normalization process performed by the normalization means includes the following steps:
a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data;
and a second normalization process step of acquiring data obtained by interpolating feature points of the intermediate stroke data so that intervals between all feature points adjacent in chronological order are equal to or less than the threshold value. A recognition processing method performed by a recognition processing device,
The recognition processing device includes a normalization means, an input vector data acquisition means, a character recognition processing means, and a character recognition result output means,
the normalization means acquires input stroke data indicating an input pattern of strokes made by an electronic pen in a time series order for each input character and including position information of feature points in the time series order of the input pattern, and acquires normalized stroke data by performing a normalization process for normalizing the acquired input stroke data into an input pattern with a fixed number of samples;
the input vector data acquisition means converts the normalized stroke data normalized by the normalization means into input vector data expressed by the feature amount of the fixed number of samples;
the character recognition processing means performs character recognition processing on the input vector data acquired by the input vector data acquisition means, using a learning model that has been machine-learned using the input vector data acquired by the input vector data acquisition means;
The normalization process performed by the normalization means includes the following steps:
a first normalization process for thinning out feature points from the input stroke data so that intervals between all feature points adjacent in time series are equal to or greater than a predetermined threshold value, thereby acquiring intermediate stroke data;
and a second normalization process step of acquiring data by interpolating feature points of the intermediate stroke data so that intervals between all feature points adjacent in chronological order are equal to or less than the threshold value. A recognition processing system having a pen tablet capable of receiving input using an electronic pen and a front electronic pen, and a recognition processing device that recognizes characters written on the pen tablet by a user with the electronic pen, characterized in that the recognition processing device is a recognition processing device according to any one of claims 1 to 5.

Images