JP2604402B2 - Automatic music transcription method and device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION The present invention relates to an automatic music transcription method and apparatus for creating musical score data from acoustic signals such as singing voices, humming voices, and instrument sounds, and more particularly, to a method and apparatus based on power information. The present invention relates to a segmentation process for dividing an audio signal into sections that can be regarded as having the same pitch.

[Prior Art] In an automatic transcription system for converting an acoustic signal such as a singing voice, a humming voice, or a musical instrument sound into musical score data, a sound length, a pitch, a key, a time signature, and the like, which are basic information as a musical score from an acoustic signal. Detecting the tempo.

By the way, an acoustic signal is only a signal that continuously includes a repetitive waveform of a basic waveform, and the above-described information cannot be obtained immediately.

Therefore, in the conventional automatic transcription system, first, a basic waveform representing a pitch of an acoustic signal is repeatedly extracted (hereinafter, referred to as pitch information) and power information is extracted for each analysis cycle. Based on the information, the sound signal is divided into sections (segments) that can be regarded as the same pitch (this processing is called segmentation). Then, from the pitch information of the segment, the pitch of the sound signal of each segment is identified as a pitch along the absolute pitch axis. Each information is obtained in the order of determining the tone of the acoustic signal based on the distribution of the pitch information around the pitch axis, and further determining the beat and tempo of the acoustic signal based on the segment.

Therefore, the interval, time signature, tempo, and the like are determined based on the segment (tone length), and thus the segmentation processing is particularly important in creating musical score data.

[Problems to be Solved by the Invention] As described above, the segmentation is an important element in creating the score data, and if the accuracy of the segmentation is low, the degree of the score data finally obtained is also extremely low. The accuracy of the segmentation process itself from the power information should be improved both when performing the final segmentation from both the segmentation result based on the pitch information and the segmentation result based on the power information, or when performing the final segmentation based on the power information. Is desired.

The present invention has been made in view of the above points, and an object of the present invention is to provide an automatic music transcription method and apparatus that can perform segmentation based on power information satisfactorily and improve the accuracy of music score data. Is what you do.

[Means for Solving the Problem] In order to solve the problem, according to the first aspect of the present invention, a process of extracting power information from an input audio signal, and the audio signal can be regarded as the same pitch based on the power information. A segmentation process for converting an audio signal into musical score data, wherein the segmentation process extracts a rising change point of the power information, and classifies the audio signal at the extracted rising change point. Then, a process in which a section shorter than a predetermined length in the divided sections is connected to one of the preceding and following sections is performed.

In the second aspect of the present invention, the power extraction means for extracting power information from the input audio signal and the segmentation means for dividing the audio signal from the extracted power information into sections that can be regarded as having the same pitch are partially included. In an automatic transcription apparatus for converting an audio signal into musical score data, a segmentation unit includes a change point extraction unit that extracts a rising change point of power information, and a division unit that separates an audio signal by the extracted rising change point. And a section correction unit for making a section shorter than a predetermined length of the divided sections continuous with any of the preceding and following sections.

[Operation] In the first aspect of the present invention, when the sound information is divided into sections that can be regarded as having the same pitch based on the power information, the sound signal has the maximum power at the start point of a new sound regardless of the source of the sound signal. Paying attention to the fact that the power information gradually attenuates thereafter, a rising change point of the power information is extracted, and the audio signal is classified based on the extracted change point.
Further, in order to eliminate a segmentation error due to noise, power fluctuation, or the like, a section shorter than a predetermined length is not a single section but is connected to one of the preceding and following sections to form one section.

Further, according to the second aspect of the present invention, when the power information is divided into sections which can be regarded as the same pitch by the segmentation means, the rising change point of the power information is similarly extracted by the change point extracting unit by paying attention to the way of changing the power information. Then, the sound signal is divided at the extracted change point by the dividing section. Further, in order to eliminate a segmentation error due to noise, power fluctuation, or the like, the section shorter than the predetermined length is not divided into a single section by the section correction unit, but is connected to any of the preceding and following sections to form one section.

[Embodiment] One embodiment of the present invention will be described below in detail with reference to the drawings.

Automatic transcription system First, an automatic transcription system to which the present invention is applied will be described.

In FIG. 4, a central processing unit (CPU) 1 controls the whole of the apparatus, and performs a transcription process shown in FIG. 5 stored in a main storage device 3 connected via a bus 2. Execute the program. In addition to the CPU 1 and the main storage device 3, a keyboard 4 as an input device, a display device 5 as an output device, an auxiliary storage device 6 used as a working memory, and an analog / digital converter 7 are connected to the bus 2. I have.

An audio signal input device 8 including, for example, a microphone is connected to the analog / digital converter 7. The acoustic signal input device 8 captures an acoustic signal such as singing or humming uttered by a user or a musical tone generated from a musical instrument and converts the signal into an electric signal, and converts the electric signal into an analog / digital converter. 7 is output.

When a process is instructed by the keyboard input device 4, the CPU 1 starts the transcription process, executes a program stored in the main storage device 3, and converts the sound converted into a digital signal by the analog / digital converter 7. The signals are temporarily stored in the auxiliary storage device 6, and thereafter, these sound signals are converted into musical score data by executing the above-described program and output to the display device 5 as necessary.

Next, the music transcription process performed by the CPU 1 after capturing the audio signal will be described in detail with reference to the flowchart shown in the functional level of FIG.

First, the CPU 1 performs an autocorrelation analysis of the acoustic signal to extract pitch information of the acoustic signal at each analysis cycle, and also performs a sum-of-squares process to extract power information at each analysis cycle, and then performs noise removal and smoothing processing. And other post-processing (steps SP1, SP
2). Thereafter, for the pitch information, the CPU 1 calculates a shift amount of the pitch axis of the acoustic signal with respect to the absolute pitch axis, and executes a tuning process of shifting the obtained pitch information according to the shift amount (step SP3). . That is, the pitch information is corrected so that the difference between the pitch axis of the singer or the musical instrument that generated the acoustic signal and the absolute pitch axis becomes smaller.

Next, the CPU 1 obtains a continuous period of pitch information in which the obtained pitch information is considered to indicate the same pitch,
A segmentation is performed to divide the acoustic signal into segments for each sound, and a segmentation is performed based on the obtained change in the power information (step SP
4, SP5). Based on these two pieces of segment information obtained, the CPU 1 calculates a reference length corresponding to a time length of a quarter note, an eighth note, etc., and executes the segmentation again based on this reference length (step SP6). .

The CPU 1 identifies the pitch of the segment as the pitch on the absolute pitch axis for which it is possible to determine that the pitch information is closest based on the push information of the segment thus segmented, and further identifies the interval of the identified continuous segment. Segmentation is again performed based on whether or not are the same (steps SP7 and SP8).

Thereafter, the CPU 1 obtains a product sum of the frequency of occurrence of the pitch obtained by summing the pitch information after tuning along the pitch axis and a predetermined weighting coefficient determined according to the key, and obtains the maximum information of the product sum. On the basis of this, for example, the key of the music of the input sound signal is determined, such as C major or A minor, and the predetermined pitch in the scale of the determined key is reviewed with respect to the pitch information to confirm and correct the pitch ( Step SP
9, SP10). Next, the CPU 1 executes a review of the segmentation based on whether or not there is the same continuous segment from the finally determined pitch, and whether or not there is a change in power between the consecutive segments. , Perform the final segmentation (step SP
11).

Once the pitch and segment are determined in this way,
The CPU 1 extracts measures from the viewpoint that the music starts from the first beat, the last sound of the phrase does not extend to the next measure, and there is a break in each measure, and determines the time signature from the measure information and the segmentation information. The tempo is determined from the determined time signature information and the length of the bar (step SP1).
2, SP13).

Then, the CPU 1 organizes the information on the determined pitch, pitch, key, beat, and tempo to finally create the musical score data (step SP14).

Segmentation Based on Power Information Next, a segmentation process based on power information in such an automatic transcription system (see step SP5) will be described in detail with reference to the flowcharts of FIGS. FIG. 1 is a flowchart showing such processing at a functional level, and FIG. 2 is a flowchart showing FIG. 1 in more detail.

As power information of the acoustic signal, the acoustic signal is squared at each sampling point in the analysis cycle, and
The sum of the power values is used as power information in the analysis cycle.

First, the CPU 1 calculates a change function of the power information for each analysis point, extracts a rise of the power information based on the function, and marks the start analysis point as a segment start (steps SP20 and SP21).

The reason for performing the segmentation by extracting the rising edge of the power information in this manner is that when the pitch of the sound signal is changed, the power becomes the largest at the start point of a new sound, and then gradually. This is because it has a property of attenuating.

Thereafter, the CPU 1 measures the length from the start point of the segment to the start point of the next segment, that is, the segment length, removes the segment having an insufficient segment length, and connects the section to the preceding and succeeding segments (step SP22, Two
3).

When the length of the segment is short, it is not treated as a segment. In the case of an audio signal, the power information may fluctuate, and noise may be mixed. Even when the user intends the same sound, a plurality of peaks of the power change may occur, and this is to prevent a segmentation error due to the plurality of peaks.

Thus, it is possible to execute the segmentation based on the rising information of the power information and in consideration of the segment length.

Next, such processing will be described in more detail with reference to FIG.

CPU1 first clears the analysis point parameter t to 0,
After confirming that the data of the analysis point to be processed has not been completed, the power information P of the analysis point t is determined.
D (t) = {P (t + k) -P (t)} P (t + k) + P (P) t)｝ A change function of the power information shown in (1) (hereinafter, referred to as a rising extraction function because it is used for extraction of a rising edge in the subsequent processing) d
(T) is calculated (steps SP30 to SP32). Note that k is selected to be an appropriate time difference suitable for capturing a change in power information.

Thereafter, the CPU 1 determines whether or not the rising extraction function d (t) at the analysis point t is equal to or larger than the threshold value θd. If the result is smaller than the threshold value θd and a positive result is obtained, the parameter t is incremented and the process returns to step SP31. (Step SP33-3
Four).

By repeating such processing, the CPU 1 eventually finds an analysis point immediately after the rising extraction function d (t) has changed to the threshold value θd or more, and obtains a negative result in step SP33.
At this time, after marking the start of the segment, the CPU 1 confirms that the data of the analysis point to be processed has not been completed, and for that analysis point t, the power information P (t) of that point and k Power information P of analysis point t + k advanced by only
A rising extraction function d (t) of the power information is calculated again based on (t + k) (steps SP35 to SP37).

Thereafter, the CPU 1 determines whether or not the rising extraction function d (t) at the analysis point t is smaller than the threshold θd,
If the result is not less than d and a negative result is obtained, the analysis point parameter t
Is incremented, and the process returns to the above-mentioned step SP36 (steps SP38 to SP39). On the other hand, if the result is smaller than the threshold value θd and a positive result is obtained in step SP38, the process returns to step SP31 to extract the analysis point immediately after the next rising extraction function d (t) has changed to the threshold value θd or more. Proceed to.

By repeating such processing, all the analysis points at the time of rising of the power information are marked with the start of the segment,
Eventually, the process is completed for all the power information, a positive result is obtained in step SP31 or SP36, and the process proceeds to the segment review process after step SP40.

As the segment review processing, the CPU 1 first clears the parameter t to 0, confirms that the analysis point data to be processed has not ended, and determines whether a segment start mark has been added to the analysis point t. Is determined (steps SP40 to SP42). If the CPU 1 obtains a negative result without the mark, the CPU 1 increments the parameter t and returns to the above-mentioned step SP41 (step SP43). By repeating such a process, the analysis point marked with is eventually obtained.
A positive result is obtained in step SP42.

At this time, the CPU 1 sets 1 as the length parameter L, increments the parameter t, confirms that the analysis point data to be processed has not ended, and attaches a segment start mark to the analysis point t. It is determined whether or not it has been performed (steps SP44 to SP47). If the analysis point being processed is not marked and a negative result is obtained, the CPU 1 increments both the length parameter L and the analysis point parameter t, and returns to step SP46 (steps SP48 and SP49).

By repeating this processing, the analysis point marked as the start of the segment is eventually obtained, and the processing is started at step SP.
A positive result is obtained at 47. The length parameter L at this time
Corresponds to the distance between the analysis point to be processed marked and the analysis point marked immediately before it, that is, to the length of the segment. If an affirmative result is obtained in step SP47, the CPU 1 determines whether or not the parameter L (segment length) is shorter than the threshold value θL. Returning to step SP41, if smaller than the threshold value θL,
The mark of the start of the front segment is removed, that is, this segment is connected to the front segment, and the process returns to step SP41 (steps SP50 and SP51).

When returning to step SP41 from step SP50 or 51, if the data has not been completed, step SP42
In step SP44, a positive result is obtained immediately, and the process proceeds to the processing after step SP44, and the operation shifts to an operation of searching for a mark next to the mark just found, finding the next mark in the same manner as described above, and determining the segment length. Perform a review.

By repeating such an operation, the review of all segment lengths is completed, and a positive result is obtained in step SP41, and the process is terminated.

FIG. 3 shows an example of a time change of the power information P (t) and a time change of the rising extraction function d (t). In this example, the segment up to step SP39 in FIG. S1, S2... SN are classified. But step
By executing the processing after SP40, the short segment S4 is eliminated, and the segment S3 and the segment S4 are made into one segment S34.

Therefore, according to the above-described embodiment, the rising edge of the power information is extracted and the segmentation is performed at the rising edge. Therefore, it is possible to execute the high-precision segmentation that can be divided at the start point of the sound. In addition, since the review is performed based on the segment length, even when the power information has noise or fluctuation, the segmentation can be performed without being affected by the noise or fluctuation.

Other Embodiments In the above-described embodiment, the power information is obtained by using the sum of squares of the audio signal. However, other parameters may be used. For example, a parallel root of the sum of squares may be used.

Further, in the above-described embodiment, the function shown in Expression (1) is applied as the function for extracting the rising edge, but another function may be applied. For example, a differential function with a fixed denominator may be applied.

Furthermore, in the above-described embodiment, the segment having an insufficient segment length is connected to the immediately preceding segment. However, the segment may be connected to the immediately following segment. It may be connected to the immediately preceding segment, and if it is a rest section, connected to the immediately following segment.

Further, in the above-described embodiment, the CPU 1 executes all the processing shown in FIG. 5 according to the program stored in the main storage device 3, but a part or all of the processing is performed by a hardware configuration. May be executed. For example, as shown in FIG. 6 in which the same reference numerals are given to the corresponding parts in FIG. 4, the audio signal from the audio signal input device 8 is amplified through the amplifier circuit 10, and then the pre-filter is added.
The digital signal is supplied to an analog / digital converter 12 via an analog-to-digital converter 12, and is converted into a digital signal. To extract power information and CP
It may be provided to a software processing system by U1. As the signal processor 13 used in such a hardware configuration (10 to 13), a processor capable of processing a signal in a voice band in real time and providing an interface signal with a host CPU 1 (for example, NEC Corporation) Co., Ltd. μPD7720) can be applied.

[Effects of the Invention] As described above, according to the present invention, the rising edge of the power information is extracted and the segmentation is executed at the rising edge.
In addition, since the segment is corrected in consideration of the segment length, highly accurate segmentation that can be segmented at the start point of the sound can be performed, and even when the power information has noise or fluctuation, The segmentation can be executed without being affected by the influence, and the subsequent processes such as the pitch identification process using the segmentation result can be favorably executed.

[Brief description of the drawings]

FIGS. 1 and 2 are flowcharts showing a segmentation process based on power information according to an embodiment of the present invention. FIG. 3 is a characteristic curve diagram showing temporal changes of power information and a rising extraction function together with a segmentation result.
Fig. 5 is a block diagram showing a configuration of an automatic transcription system to which the present invention is applied, Fig. 5 is a flowchart showing an automatic transcription process, and Fig. 6 is a block diagram showing another configuration of the automatic transcription system. 1 ... CPU, 3 ... main storage device, 6 ... auxiliary storage device,
7 ... A / D converter, 8 ... Acoustic signal input device.

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Masaki Fujimoto 5-7-15 Shiba, Minato-ku, Tokyo Inside NEC Technical Information System Development Co., Ltd. (72) Inventor Masanori Mizuno 5-7-1 Shiba, Minato-ku, Tokyo No. 15 Examiner, NEC Technical Information Systems Development Co., Ltd. Shigeo Arai

Claims (2)

(57) [Claims] 1. A method for extracting power information from an input audio signal, and a segmentation process for dividing the audio signal into sections that can be regarded as having the same pitch based on the power information. In the automatic transcription method for converting, the segmentation process extracts a rising change point of the power information, divides the sound signal by the extracted rising change point, and divides the section into segments that are shorter than a predetermined length. An automatic music transcription method characterized by a process of making it continuous with any one of the preceding and following sections. 2. The sound processing apparatus according to claim 1, further comprising: power extraction means for extracting power information from the input audio signal; and segmentation means for dividing the audio signal into sections that can be regarded as having the same pitch based on the extracted power information. In an automatic transcription apparatus for converting a signal into musical score data, the segmentation means includes: a change point extraction unit that extracts a rising change point of the power information; and a division unit that separates the acoustic signal by the extracted rising change point. An automatic music transcription apparatus, comprising: a section correction unit that connects a section shorter than a predetermined length to one of the preceding and following sections among the divided sections.