JP6437208B2 - Handwritten music symbol recognition apparatus and handwritten music symbol recognition program - Google Patents

Description

  The present invention relates to a handwritten music symbol recognition apparatus and a handwritten music symbol recognition program, and in particular, recognizes a handwritten symbol when creating a score by handwriting (including using a pen) on a tablet terminal (including a touch-compatible PC). The present invention relates to a technique suitable for use.

  A tablet terminal corresponding to handwriting input detects trajectory information obtained by scanning a touch panel by an input means such as a finger, and recognizes a symbol or a figure based on the detected information.

As a method for recognizing handwritten input characters and symbols, there are a method for comparing stroke coordinates of characters with dictionary information, a method for comparing the writing direction of characters with a dictionary, and the like.
However, the above-described method has a problem that erroneous recognition is required if a large storage capacity is required or the writing order is different from the normal one.

  In addition, when a handwritten character is written in a continuous character, an irregular bending point occurs depending on how the continuous character is written, and the bending position of the character cannot be detected correctly. There was a problem that the accuracy of character recognition was lowered.

  In order to solve the above-described problems, Patent Document 1 discloses that a starting point and an ending point of a writing stroke are connected with a virtual line segment, a feature amount of a bent portion generated in that case is detected, and this is stored in a dictionary. A character recognition method that performs character recognition in comparison with information has been proposed.

JP 2004-133951 A

  In the character recognition method proposed in Patent Document 1, the bent portion of the written character corresponds to the bent portion of the recognition candidate character, and the character can be recognized with high accuracy even if the user continuously writes the character. However, it is necessary to register in advance a specific symbol or figure for converting characters. For this reason, since it is necessary to retain data for the number of recognizable characters, symbols, or figures, a storage medium having a large storage capacity is required.

In addition, since it is necessary to correlate the input characters with the character candidates stored in a large amount in the storage medium on a one-to-one basis, there is a problem that a complicated and large program is required.
SUMMARY OF THE INVENTION In view of the above-described problems, an object of the present invention is to enable accurate recognition only by performing a simpler form of recognition processing without holding a huge amount of data.

The handwritten music symbol recognition apparatus according to the present invention includes a touch information acquisition unit that acquires position information every time the input unit moves until the input unit touches the screen and leaves, and a position obtained from the touch information acquisition unit. From the information , to indicate the movement of the input means , point, down → up, bottom left → top right, left → right, top left → bottom right, top → bottom, top right → bottom left, right → left, bottom right → top left, Vector information classified into nine types is created, and stroke information storage means for storing a series of vectors as one stroke information from when the input means touches the screen until it completely leaves, and created by the stroke information storage means from nine vector information classification that is, connected connection types each other adjacent vector, determined acute angle, a right angle, an obtuse angle, or other of which one is stored in the connection type sequence type Storing means, dividing means for dividing the vector before and after the acute angle point in the connection type obtained by the connection type storage means, and stroke shape recognition means for recognizing the shape of the stroke of the vector array divided by the dividing means And a symbol determination means for determining a symbol based on the stroke shape recognized by the stroke shape recognition means and its positional relationship.
The handwritten music symbol recognition program according to the present invention includes a touch information acquisition process for acquiring position information every time the input unit moves until the input unit touches the screen and then leaves, and a position obtained from the touch information acquisition process. From the information , to indicate the movement of the input means , point, down → up, bottom left → top right, left → right, top left → bottom right, top → bottom, top right → bottom left, right → left, bottom right → top left, Creates vector information classified into nine types, stroke information storage processing for storing a series of vectors from the time when the input means touches the screen until it completely leaves as one stroke information, and the stroke information storage processing connections from nine vector information classification that is, the connection type of each other adjacent vector, determined acute angle, a right angle, an obtuse angle, or other of which one is stored in the connection type sequence Type storage processing, division processing for dividing a vector before and after an acute angle point in the connection type obtained by the connection type storage processing, and stroke shape recognition processing for recognizing the stroke shape of the vector array divided by the division processing And a symbol determination process for determining a symbol based on the stroke shape recognized by the stroke shape recognition process and its positional relationship.

  According to the present invention, the stroke is divided at the point where the vector angle of the handwritten character or symbol changes acutely, and each divided portion is recognized, so that the handwritten input symbol is complicated. However, it is possible to accurately recognize by performing a simpler form of recognition processing and processing for associating the positional relationship without holding enormous data.

It is a block diagram which shows the structural example of the handwritten music symbol recognition apparatus in embodiment of this invention. It is a figure which shows the example which divides | segments and recognizes one stroke into several strokes. It is a flowchart explaining the procedure of a music symbol recognition process. It is a figure which shows the example which classify | categorizes a vector into nine types. It is a figure which shows the example of the inscribed rectangle of S character vector, and a center coordinate. It is a figure which shows the example of a change of the rotation angle from the said vector to the following vector. It is a figure which shows how to determine a connection type and shows the relationship between the ID of the vector and the ID of the next vector. It is a figure which shows how to determine a rotation direction, and shows the relationship between ID of the said vector, and ID of the next vector.

Hereinafter, an entire handwritten music symbol recognition apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a schematic configuration of a handwritten music symbol recognition apparatus 10, and the handwritten music symbol recognition apparatus 10 includes a tablet terminal (including a touch-compatible PC).

  In FIG. 1, 11 is a bus, 12 is ROM, 13 is RAM, 14 is CPU, 15 is flash memory, 16 is a network interface, 17 is a touch display, 18 is an audio output device, and 19 is a MIDI output device.

A program for controlling the overall operation of the handwritten music symbol recognition apparatus 10 is stored in the flash memory 15, and is read out, expanded in the RAM 13, and executed by the CPU 14.
The touch information acquisition unit includes the touch display 17, the RAM 13, the CPU 14, the flash memory 15, and the like, and acquires the position information every time the touch position moves.

The stroke information storage means and the object information storage means are constituted by the RAM 13.
The stroke feature amount calculating means, stroke recognizing means, and object recognizing means are composed of a RAM 13, a CPU 14, and a flash memory 15. The stroke information storage means creates vector information indicating the movement of the input means such as a finger and a touch pen from the position information obtained from the touch information acquisition means, and a series of vectors from when the input means touches the screen until it completely leaves Is stored as one stroke information.
The stroke recognizing means compares the stroke feature quantity with the feature of the symbol and recognizes what symbol the stroke is.

The network interface 16 stores the recognized music symbol information as a file in other hardware, or reads a stored music symbol information file.
The audio output device 18 is provided in order to create music performance information from the recognized music symbols and output the sound.
The MIDI output device 19 is provided in order to create MIDI information from the recognized music symbols and output the MIDI information.

A method for assigning a vector ID in this embodiment will be described.
In this embodiment, in order to simplify the processing, the vector is down → up, bottom left → top right, left → right, top left → bottom right, top → bottom, top right → bottom left, right → left, bottom right → top left, Classify into 9 types of points.
As shown in FIG. 4, the slope a is obtained from the starting point (x1, y1) and the ending point (x2, y2) of the vector using the equation (1).
Then, the vector ID is determined in consideration of the positional relationship of x1, y1, x2, and y2. The lower side of the y-axis is “+”.

| a |> 2
y2 <y1: Lower → Up y2> y1: Lower → Up-2 ≦ a <−0.5
x2 <x1: lower left → upper right x2> x1: upper right → lower left
| a | ≦ 0.5
x2> x1: Left → Right x2 <x1: Right → Left 0.5 <a ≦ 2
x2> x1: upper left → lower right x2 <x1: lower right → upper left x1 = x2
y2 <y1: Bottom → Up y2> y1: Bottom → Up x1 = x2 and y1 = y2
It becomes a point.

Each feature amount will be described.
(Start point end point distance)
When this distance is smaller than the specified value, it is considered as a closed figure like a circle, and when it is larger, it is considered as a figure not closed like an arc line.
If the start point coordinates of the first vector are (x1, y1) and the end point coordinates of the final vector are (x2, y2), the start point / end point distance D _se is obtained by the following equation (2).

(The coordinates of the four points of the inscribed rectangle and the center coordinates)
This value is used to obtain the positional relationship between the strokes.
FIG. 5 shows an example of the inscribed rectangle and center coordinates of the S-shaped vector.
Rectangular coordinate left is the leftmost X coordinate of the start or end points of all vectors.
Rectangular coordinate right is the rightmost X coordinate of the start or end points of all vectors.
On the rectangular coordinates, it is the highest Y coordinate among the start points or end points of all vectors.
Below the rectangular coordinates is the lowest Y coordinate of the start or end points of all vectors.
The center coordinate is represented by an X coordinate intermediate between the rectangular coordinate left and the rectangular coordinate right, and an intermediate Y coordinate above the rectangular coordinate and below the rectangular coordinate.

(Connection type)
It is used to judge whether the symbol is smooth or has corners.
It is determined from the ID of the vector and the ID of the next vector (see FIG. 7). In the present embodiment, it is determined that the angle is not an obtuse angle, a right angle, or an acute angle.

(Direction of rotation)
The point where the rotation direction has changed is important as a clue for identifying the symbol.
For example, in the first stroke of “f”, vectors are smoothly continuous like “S”, the start point and the end point are separated, but “S” has one point where the rotation direction changes. So you can distinguish. This is also determined from the ID of the vector and the ID of the next vector (see FIG. 8). When the number of changes in the rotation direction is 0, the total rotation angle is obtained, and symbol determination is performed thereby.

Rotation angle ≤ 180 °
The arc line is determined under the following conditions.
・ There is “right” in the left / right movement direction and there is no “left”.
・ The highest point is other than the start point and end point → upper arc line ・ The highest point is the start point or end point → lower arc line

-There is no "right" in the left / right movement direction and there is "left".
・ The highest point is other than the start point and end point → upper arc line ・ The highest point is the start point or end point → lower arc line

・ There is “Down” in the vertical movement direction, and there is no “Up”.
・ Leftmost point is other than start point and end point → left arc line ・ Leftmost point is start point or end point → right arc line

・ There is no “down” in the vertical movement direction and “up”.
・ Left-most point is other than start point and end point → Left arc line ・ Left-most point is start point or end point → Right arc line ・ Other than above → No applicable symbol.

Rotation angle> 360 ° → multiple circles

The determination of other circles and ellipses is performed according to the following procedure.
1. Find the equation of the straight line (the major axis) connecting the leftmost point and the rightmost point.
2. From each point, the perpendicular foot drawn on the straight line is obtained.
3. The point where the length of each point and the leg of the perpendicular is the longest is defined as the half minor axis of the ellipse.
4). The ratio of the major axis of the semi-minor axis × 2 (assuming the minor axis) is obtained.
・ The calculated ratio is almost 1: 1 → circle ・ others → ellipse

(rotation angle)
FIG. 6 shows a change example of the rotation angle from the vector to the next vector. From this total, it can be determined whether the symbol is a circle or a multiple circle.
If the start point of the vector is (x11, y11), the end point is (x12, y12), the start point of the next vector is (x21, y21), and the end point is (x22, y22), the angle Deg ( °) is obtained by the following formula.

(Left / Right direction and Up / Down direction)
It is determined whether the symbol is simply a line segment drawn from top to bottom and from left to right.
Obtained from the vector ID. If the vector ID is down → up, there is no left / right movement direction, and the up / down movement direction is up. If the vector ID is lower right → upper left, the left / right moving direction is left and the up / down moving direction is up.

(Leftmost point, rightmost point, topmost point, lowest point index)
For example, if the leftmost point Index is 0 (start point of the first vector) and the rightmost point Index is N (start point of the final vector), if the highest point Index is near N / 2, the bottom is an open mountain Can be determined.

<First Embodiment>
Hereinafter, a first embodiment of the handwritten music symbol recognition apparatus of the present invention will be described.
A tablet terminal that supports handwriting has an API that acquires position information whenever a finger moves until the finger touches the screen and then leaves. A series of changes in position information is stored as a plurality of vectors, the features of finger movement are extracted from the vectors, and the drawn symbols are recognized.

One of the features of finger movement is a connection type of a vector and the next vector. As described in the connection type of the feature amount described above, the connection type is divided into an acute angle, a right angle, and an obtuse angle.
In this embodiment, one stroke is divided into a plurality of strokes at an acute angle, and by recognizing each of the divided strokes, symbols can be recognized easily and accurately by recognizing simpler shapes and their positional relationships. it can.

A specific example is shown in FIG.
In the character “2” shown in FIG. 2A, the connection type from the first vector a1 to the fifth vector a5 is an obtuse angle, and the connection type of the fifth vector a5 and the sixth vector a6 is an acute angle. It is.
Therefore, if the first to fifth vectors a1 to a5 and the sixth vector a6 are recognized as separate strokes, as shown in FIG. 2B, arc lines b1 and straight lines b2 are obtained, and only simple shapes are recognized. I understand that

Next, the processing procedure of this embodiment will be described with reference to the flowchart of FIG.
In S31, a connection type storage process is performed in which connection types between adjacent vectors are obtained and stored in a connection type array. In the case of the character “2”, an obtuse angle, an obtuse angle, an obtuse angle, an obtuse angle, and an acute angle are stored in order.

In S32, a vector array is newly created before and after the acute angle point based on the connection type array. A vector array is created by +1 acute angle point. In the case of the character “2”, two vector arrays are created.
Vector array 1 stores vectors a1 to a5, and vector array 2 stores vector a6.

In S33, a stroke shape recognition process for recognizing the shape of each vector array (= stroke) is performed.
About vector array 1 ・ Check whether the change in the rotation direction is either clockwise or counterclockwise. In this case, the rotation direction is constant clockwise.
・ Check the total rotation angle. In this case, since it is less than 180 ° and the rotation direction is constant, it can be determined as an arc line.

In addition:-Number of vectors going to the right-Number of vectors going to the left-Number of vectors going down-Number of vectors going up-And-Index of the top point-Index of the bottom point-Leftmost point The index of the rightmost point is obtained and the arc line is determined from them.

In this case ・ The number of vectors from top to bottom is 1 or more and the number of vectors from bottom to top is 0
・ The index of the leftmost point is not 0, and the index of the leftmost point is not the last point. From the above conditions, it can be seen that the arc is open to the left.

Regarding vector array 2-The number of vectors is 1.-The vector ID obtained from the relationship between the slope of the vector, the start point, and the end point is from left to right.

  In S34, object generation processing for generating one object is performed from the shape of each vector and its positional relationship. In the case of FIG. 2B, since the horizontal straight line b2 is below the arc line b1 opened to the right, the character “2” is generated.

  In the above-described embodiment, an example of recognizing the character “2” has been described. However, one stroke is divided at an acute change point, and various characters and symbols that can be recognized are recognized. Is possible.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

10 Handwritten Music Symbol Recognition Device 11 Bus 12 ROM
13 RAM
14 CPU
15 Flash memory 16 Network interface 17 Touch display 18 Audio output device 19 MIDI output device

Claims (6)

Touch information acquisition means for acquiring position information each time the input means moves until the input means touches the screen and then leaves,
The position information obtained from the touch information acquiring means, to indicate the movement of the input means, the points, the lower → upper, lower left → upper right, left → right, upper left → lower right, upper → lower, upper right → lower left, right Stroke information storage means that creates vector information classified into nine types of left, lower right, and upper left, and stores a series of vectors as one stroke information from when the input means touches the screen until it completely leaves When,
A connection type that determines whether the connection type between adjacent vectors is an acute angle, a right angle, an obtuse angle, or the like, from the nine types of vector information created by the stroke information storage means, and stores it in the connection type array Storage means;
Dividing means for dividing the vector before and after the acute angle point in the connection type determined by the connection type storage means;
Stroke shape recognition means for recognizing the shape of the stroke of the vector array divided by the dividing means;
A handwritten music symbol recognition apparatus, comprising: a stroke shape recognized by the stroke shape recognition means; and a symbol determination means for determining a symbol based on the positional relationship. Rotation direction determination means for determining whether the rotation direction from the vector to the next vector is clockwise or counterclockwise based on nine types of vector information created by the stroke information storage means. Further comprising
2. The handwritten music symbol recognition apparatus according to claim 1, wherein the stroke shape recognition unit recognizes the shape of a stroke of the vector array divided by the division unit based on the rotation direction. First calculating means for calculating a distance from the start point to the end point;
Second calculation means for calculating the coordinates and center coordinates of four points of a rectangle inscribed by the stroke;
Third calculating means for calculating a rotation angle from the vector to the next vector;
Based on the vector information, a fourth calculating means for calculating a moving direction to the left and right and a moving direction to the up and down;
A fifth calculating means for calculating an index corresponding to the leftmost point, rightmost point, uppermost point, and lowermost point;
The stroke shape recognizing unit recognizes the shape of the stroke of the vector array divided by the dividing unit based on the calculation results of the first to fifth calculating units. Handwritten music symbol recognition device. The stroke shape recognition means includes
When the number of changes in the rotation direction is 0,
When the total rotation angle of the vector is 180 ° or less, the upper arc line or the lower arc line is recognized according to the left / right moving direction and the index of the uppermost point, and the upper / lower moving direction and the leftmost line are recognized. Depending on the index of the point, recognize the left or right arc,
If the sum of the rotation angles of the vectors is greater than 360 °, a multiple circle is recognized,
If the total rotation angle of the vector is greater than 180 ° and less than or equal to 360 °, an equation of a straight line connecting the leftmost point and the rightmost point is obtained, and a perpendicular foot drawn from each point to the straight line is obtained. The longest length from each point to the foot of the perpendicular is the semi-minor axis, and when the ratio of the semi-minor axis × 2 and the straight line is approximately 1: 1, a circle is recognized, and the ratio 4. An apparatus for recognizing handwritten music symbols according to claim 3, wherein an ellipse is recognized when is not approximately 1: 1. The stroke shape recognition means includes
In the stroke of the first vector array divided by the dividing means, the number of upper → lower vector information is 1 or more, the number of lower → upper vector information is 0, and the index of the leftmost point If is not 0 and the leftmost index is not the last point, it recognizes the left arc,
In the stroke of the second vector array divided by the dividing means, when the number of vector information is 1 and the vector information is from left to right, a horizontal straight line is recognized,
The symbol determination means recognizes the character “2” when the stroke of the first vector array is a left arc line and the stroke of the second vector array is a horizontal straight line. The handwritten music symbol recognition apparatus according to claim 4. Touch information acquisition processing for acquiring the position information every time the input means moves until the input means touches the screen and then leaves,
The position information obtained from the touch information acquisition process, in order to show the movement of the input means, the points, the lower → upper, lower left → upper right, left → right, upper left → lower right, upper → lower, upper right → lower left, right Stroke information storage processing for creating vector information classified into nine types of left, lower right, upper left, and storing a series of vectors as one stroke information from when the input means touches the screen until it completely leaves When,
A connection type that determines whether the connection type between adjacent vectors is an acute angle, a right angle, an obtuse angle, or the like from the nine types of vector information created by the stroke information storage process, and stores it in the connection type array Amnestics,
A dividing process of dividing the vector before and after the acute angle point in the connection type obtained by the connection type storing process;
Stroke shape recognition processing for recognizing the shape of the stroke of the vector array divided by the division processing;
A handwritten music symbol recognition program that causes a computer to execute a symbol determination process for determining a symbol based on a stroke shape recognized by the stroke shape recognition process and its positional relationship.

Images