JP7651901B2 - IMAGE PROCESSING METHOD, IMAGE PROCESSING SYSTEM, AND PROGRAM - Google Patents

Description

This disclosure relates to technology for analyzing a user's performance.

For example, there have been proposed techniques for estimating the area in an image captured by a camera in which a specific object exists. For example, Patent Literature 1 discloses a technique for detecting an object using a deep neural network.

Special table 2020-528176 publication

However, if it were possible to extract a specific area, such as the keyboard area, from a performance image captured while playing a musical instrument such as a keyboard instrument, this would be useful for analyzing the user's fingering, for example. In consideration of the above circumstances, one aspect of the present disclosure aims to improve the convenience of performance images.

To solve the above problems, an image processing method according to one aspect of the present disclosure estimates a specific region that includes an image of a musical instrument from a performance image that includes an image of the musical instrument and an image of the fingers of a user playing the musical instrument, and extracts the specific region from the performance image.

An image processing system according to one aspect of the present disclosure includes an area estimation unit that estimates a specific area including an image of a musical instrument from a performance image that includes an image of the musical instrument and an image of the fingers of a user playing the musical instrument, and an area extraction unit that extracts the specific area from the performance image.

A program according to one aspect of the present disclosure causes a computer system to function as an area estimation unit that estimates a specific area that includes an image of a musical instrument from a performance image that includes an image of the musical instrument and an image of the fingers of a user playing the musical instrument, and an area extraction unit that extracts the specific area from the performance image.

1 is a block diagram illustrating a configuration of a performance analysis system according to a first embodiment. FIG. 1 is a block diagram illustrating an example of a functional configuration of a performance analysis system. FIG. 13 is a schematic diagram of an analysis screen. 13 is a flowchart of a finger position estimation process. 13 is a flowchart of a left/right determination process. FIG. 11 is an explanatory diagram of an image extraction process. 13 is a flowchart of an image extraction process. FIG. 1 is an explanatory diagram of machine learning for establishing an estimation model. FIG. 2 is a schematic diagram of a reference image. 13 is a flowchart of a matrix generation process. 13 is a flowchart of an initial setting process. FIG. 4 is a schematic diagram of a setting screen. 13 is a flowchart of a performance analysis process. FIG. 1 is an explanatory diagram regarding a problem of fingering estimation. FIG. 13 is a block diagram illustrating the configuration of a performance analysis system according to a second embodiment. FIG. 11 is a schematic diagram of control data in the second embodiment. 13 is a flowchart of a performance analysis process in the second embodiment. 13 is a flowchart of a performance analysis process according to the third embodiment. 13 is a flowchart of an initial setting process in the fourth embodiment. FIG. 13 is a block diagram illustrating the configuration of a performance analysis system according to a fifth embodiment. FIG. 13 is a block diagram illustrating an example of a functional configuration of an image processing system according to a sixth embodiment. 23 is a flowchart of a first image processing according to the sixth embodiment. FIG. 23 is a block diagram illustrating an example of a functional configuration of an image processing system according to a seventh embodiment. 23 is a flowchart of second image processing in the seventh embodiment.

1: First embodiment FIG. 1 is a block diagram illustrating the configuration of a musical performance analysis system 100 according to a first embodiment. A keyboard instrument 200 is connected to the musical performance analysis system 100 by wire or wirelessly. The keyboard instrument 200 is an electronic musical instrument having a keyboard 22 on which a plurality of (N) keys 21 are arranged. Each of the plurality of keys 21 of the keyboard 22 corresponds to a different pitch n (n=1 to N). A user (i.e., a performer) operates desired keys 21 of the keyboard instrument 200 sequentially with his/her left and right hands. The keyboard instrument 200 supplies performance data P representing a performance by the user to the musical performance analysis system 100. The performance data P is time-series data that specifies the pitch n of each of a plurality of notes that the user sequentially plays. For example, the performance data P is data in a format that complies with the MIDI (Musical Instrument Digital Interface) standard.

The performance analysis system 100 is a computer system that analyzes a user's performance of the keyboard instrument 200. Specifically, the performance analysis system 100 analyzes the user's fingering. Fingering is the way in which the user uses the fingers of his or her left and right hands when playing the keyboard instrument 200 (i.e., fingering). In other words, information about which fingers the user uses to operate each key 21 of the keyboard instrument 200 is analyzed as the user's fingering.

The performance analysis system 100 comprises a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. The performance analysis system 100 is realized by a portable information device such as a smartphone or a tablet terminal, or a portable or stationary information device such as a personal computer. The performance analysis system 100 may be realized as a single device, or as multiple devices configured separately from each other. The performance analysis system 100 may also be mounted on a keyboard instrument 200.

The control device 11 is composed of one or more processors that control each element of the performance analysis system 100. For example, the control device 11 is composed of one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

The storage device 12 is a single or multiple memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a known recording medium, such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that the storage device 12 may be a portable recording medium that is detachable from the performance analysis system 100, or a recording medium (e.g., cloud storage) that the control device 11 can write to or read from via a communication network such as the Internet.

The operation device 13 is an input device that accepts instructions from a user. The operation device 13 is, for example, a control operated by the user, or a touch panel that detects contact by the user. Note that an operation device 13 (for example, a mouse or keyboard) separate from the performance analysis system 100 may be connected to the performance analysis system 100 by wire or wirelessly.

The display device 14 displays images under the control of the control device 11. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device 14. Note that the display device 14, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly.

The photographing device 15 is an image input device that generates a time series of image data D1 by photographing a subject. The time series of image data D1 is video data representing a video. For example, the photographing device 15 includes an optical system such as a photographing lens, an image sensor that receives incident light from the optical system, and a processing circuit that generates image data D1 according to the amount of light received by the image sensor. Note that the photographing device 15, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly.

The user adjusts the position or angle of the image capture device 15 relative to the keyboard instrument 200 so that the image capture conditions recommended by the provider of the performance analysis system 100 are realized. Specifically, the image capture device 15 is installed above the keyboard instrument 200 and captures the keyboard 22 of the keyboard instrument 200 and the user's left and right hands. Therefore, as illustrated in FIG. 2, a time series of image data D1 representing a performance image G1 including an image of the keyboard 22 of the keyboard instrument 200 (hereinafter referred to as a "keyboard image") g1 and an image of the user's left and right hands (hereinafter referred to as a "fingering image") g2 is generated by the image capture device 15. That is, video data representing a video of the user playing the keyboard instrument 200 is generated in parallel with the performance. The image capture conditions by the image capture device 15 are, for example, the image capture range or image capture direction. The image capture range is the range (angle of view) captured by the image capture device 15. The image capture direction is the direction of the image capture device 15 relative to the keyboard instrument 200.

Figure 3 is a block diagram illustrating the functional configuration of the performance analysis system 100. The control device 11 executes a program stored in the storage device 12 to function as a performance analysis unit 30 and a display control unit 40. The performance analysis unit 30 generates fingering data Q representing the fingering of the user by analyzing the performance data P and image data D1. The fingering data Q specifies which of the user's multiple fingers has operated each of the multiple keys 21 of the keyboard instrument 200. Specifically, the fingering data Q specifies the pitch n corresponding to the key 21 operated by the user and the number of the finger (hereinafter referred to as the "finger number") k used by the user to operate the key 21. The pitch n is, for example, a note number in the MIDI standard. The finger number k is a number assigned to each finger of the user's left and right hands.

The display control unit 40 causes various images to be displayed on the display device 14. For example, the display control unit 40 causes the display device 14 to display an image (hereinafter referred to as the "analysis screen") 61 showing the results of the analysis by the performance analysis unit 30. FIG. 4 is a schematic diagram of the analysis screen 61. The analysis screen 61 is an image in which multiple note images 611 are arranged on a coordinate plane in which a horizontal time axis and a vertical pitch axis are set. The note images 611 are displayed for each note played by the user. The position of the note image 611 in the direction of the pitch axis is set according to the pitch n of the note represented by the note image 611. The position and total length of the note image 611 in the direction of the time axis is set according to the sounding period of the note represented by the note image 611.

A symbol (hereinafter referred to as a "fingering symbol") 612 corresponding to the finger number k specified for the note by the fingering data Q is arranged on the note image 611 of each note. The letter "L" of the fingering symbol 612 means the left hand, and the letter "R" of the fingering symbol 612 means the right hand. The numbers of the fingering symbol 612 mean each finger. Specifically, the number "1" of the fingering symbol 612 means the thumb, the number "2" means the index finger, the number "3" means the middle finger, the number "4" means the ring finger, and the number "5" means the little finger. Therefore, for example, the fingering symbol 612 "R2" means the index finger of the right hand, and the fingering symbol 612 "L4" means the ring finger of the left hand. The note image 611 and the fingering symbol 612 are displayed in different modes (e.g., hues or gradations) for the right hand and the left hand. The display control unit 40 uses the fingering data Q to display the analysis screen 61 in FIG. 4 on the display device 14.

In addition, for notes among the multiple note images 611 in the analysis screen 61 for which the estimation result of the finger number k is unreliable, the note image 611 is displayed in a manner different from that of a normal note image 611 (for example, with a dashed frame), and a specific symbol "??" is displayed, which means that the estimation result of the finger number k is invalid.

As illustrated in FIG. 3, the performance analysis unit 30 includes a finger position data generation unit 31 and a fingering data generation unit 32. The finger position data generation unit 31 generates finger position data F by analyzing the performance image G1. The finger position data F is data representing the positions of each finger on the user's left hand and each finger on the right hand. As described above, in the first embodiment, the positions of each finger on the user's left hand and right hand are differentiated, so that fingering can be estimated separately for the user's left hand and right hand. On the other hand, the fingering data generation unit 32 generates fingering data Q using the performance data P and the finger position data F. The finger position data F and the fingering data Q are generated for each unit period on the time axis. Each unit period is a period (frame) of a predetermined length.

A: Finger position data generating unit 31
The finger position data generating unit 31 includes an image extracting unit 311 , a matrix generating unit 312 , a finger position estimating unit 313 , and a projective transformation unit 314 .

[Finger position estimation unit 313]
The finger position estimation unit 313 estimates the position c[h,f] of each finger of the left hand and the right hand of the user by analyzing the performance image G1 represented by the image data D1. The position c[h,f] of each finger is the position of each fingertip in the x-y coordinate system set in the performance image G1. The position c[h,f] is expressed by a combination (x[h,f], y[h,f]) of the coordinate x[h,f] on the x-axis and the coordinate y[h,f] on the y-axis in the x-y coordinate system of the performance image G1. The positive direction of the x-axis corresponds to the right direction of the keyboard 22 (the direction from the low note to the high note), and the negative direction of the x-axis corresponds to the left direction of the keyboard 22 (the direction from the high note to the low note). The symbol h is a variable indicating either the left hand or the right hand (h=1, 2). Specifically, the value "1" of the variable h means the left hand, and the value "2" of the variable h means the right hand. The variable f is the number of each finger on the left and right hands (f=1 to 5). The value "1" of the variable f means the thumb, the value "2" means the index finger, the value "3" means the middle finger, the value "4" means the ring finger, and the value "5" means the little finger. Thus, for example, the position c[1,2] illustrated in FIG. 2 is the position of the tip of the index finger (f=2) on the left hand (h=1), and the position c[2,4] is the position of the tip of the ring finger (f=4) on the right hand (h=2).

Figure 5 is a flowchart illustrating the specific steps of the process (hereinafter referred to as the "finger position estimation process") in which the finger position estimation unit 313 estimates the position of each finger of the user. The finger position estimation process includes an image analysis process Sa1, a left/right determination process Sa2, and an interpolation process Sa3.

The image analysis process Sa1 is a process for estimating the position c[h,f] of each finger on one of the user's left and right hands (hereinafter referred to as the "first hand") and the position c[h,f] of each finger on the other of the user's left and right hands (hereinafter referred to as the "second hand") by analyzing the performance image G1. Specifically, the finger position estimation unit 313 estimates the positions c[h,1] to c[h,5] of each finger on the first hand and the positions c[h,1] to c[h,5] of each finger on the second hand by image recognition processing that estimates the user's skeleton or joints by analyzing the image. The image analysis process Sa1 uses a known image recognition process such as MediaPipe or OpenPose. Note that if a fingertip is not detected from the performance image G1, the coordinate x[h,f] of the fingertip on the x-axis is set to an invalid value such as "0".

In the image analysis process Sa1, the positions c[h,1] to c[h,5] of the fingers of the first hand and the positions c[h,1] to c[h,5] of the fingers of the second hand of the user are estimated, but it is not possible to determine whether the first hand and the second hand correspond to the left hand or the right hand of the user. In addition, when playing the keyboard instrument 200, the right arm and the left arm of the user may cross, so it is not appropriate to determine the left hand/right hand only from the coordinate x[h,f] of each position c[h,f] estimated by the image analysis process Sa1. If the user's arms and a part including the torso are photographed by the imaging device 15, the user's left hand/right hand can be estimated from the performance image G1 based on the coordinates of the user's shoulders and arms. However, there are problems in that a wide range needs to be photographed by the imaging device 15, and in that the processing load of the image analysis process Sa1 increases.

Taking the above into consideration, the finger position estimation unit 313 of the first embodiment executes the left/right determination process Sa2 in FIG. 5, which determines whether each of the first and second hands corresponds to the user's left or right hand. That is, the finger position estimation unit 313 determines the variable h at the finger position c[h,f] of each of the first and second hands to be either the number "1" indicating the left hand or the number "2" indicating the right hand.

When playing the keyboard instrument 200, the backs of both the left and right hands are positioned vertically upward, so the performance image G1 captured by the image capture device 15 includes images of the backs of both the left and right hands of the user. Therefore, in the left hand in the performance image G1, the thumb position c[h,1] is located to the right of the little finger position c[h,5], and in the right hand in the performance image G1, the thumb position c[h,1] is located to the left of the little finger position c[h,5]. Taking the above into consideration, in the left-right determination process Sa2, the finger position estimation unit 313 determines that the hand whose thumb position c[h,1] is located to the right (positive direction of the x-axis) of the little finger position c[h,5] is the left hand (h=1) of the first and second hands. On the other hand, the finger position estimation unit 313 determines that the hand whose thumb position c[h,1] is located to the left (in the negative direction of the x-axis) of the little finger position c[h,5] of the first or second hand is the right hand.

数式(1)の記号μ[h]は、第１手および第２手の各々における５本の手指の座標ｘ[h,1]～ｘ[h,5]の平均値（例えば単純平均）である。数式(1)から理解される通り、親指から小指にかけて座標ｘ[h,f]が減少する場合（左手）には判定指標γ[h]が負数となり、親指から小指にかけて座標ｘ[h,f]が増加する場合（右手）には判定指標γ[h]が正数となる。そこで、指位置推定部３１３は、第１手および第２手のうち判定指標γ[h]が負数である手を左手と判定し、変数ｈを数値「１」に設定する（Ｓa22）。また、指位置推定部３１３は、第１手および第２手のうち判定指標γ[h]が正数である手を右手と判定し、変数ｈを数値「２」に設定する（Ｓa23）。以上に説明した左右判定処理Ｓa2によれば、親指の位置と小指の位置との関係を利用した簡便な処理により、利用者の各手指の位置ｃ[h,f]を右手と左手とに区別できる。 6 is a flowchart illustrating a specific procedure of the left-right determination process Sa2. The finger position estimation unit 313 calculates a determination index γ[h] for each of the first hand and the second hand (Sa21). The determination index γ[h] is calculated, for example, by the following formula (1).

The symbol μ[h] in the formula (1) is the average value (e.g., simple average) of the coordinates x[h,1] to x[h,5] of the five fingers on each of the first and second hands. As can be understood from the formula (1), when the coordinate x[h,f] decreases from the thumb to the little finger (left hand), the judgment index γ[h] is negative, and when the coordinate x[h,f] increases from the thumb to the little finger (right hand), the judgment index γ[h] is positive. Therefore, the finger position estimation unit 313 determines that the hand with the negative judgment index γ[h] of the first and second hands is the left hand, and sets the variable h to the value "1" (Sa22). Moreover, the finger position estimation unit 313 determines that the hand with the positive judgment index γ[h] of the first and second hands is the right hand, and sets the variable h to the value "2" (Sa23). According to the left/right determination process Sa2 described above, the positions c[h, f] of the fingers of the user can be distinguished as being on the right hand or the left hand by simple processing that utilizes the relationship between the positions of the thumb and the little finger.

The image analysis process Sa1 and the left/right determination process Sa2 estimate the position c[h,f] of each finger of the user for each unit period. However, due to various circumstances such as noise present in the performance image G1, the position c[h,f] may not be estimated properly. Therefore, when the position c[h,f] is missing in a specific unit period (hereinafter referred to as the "missing period"), the finger position estimation unit 313 calculates the position c[h,f] in the missing period by the interpolation process Sa3 using the position c[h,f] in the unit periods before and after the missing period. For example, when the position c[h,f] is missing in the central unit period (missing period) of three consecutive unit periods on the time axis, the average of the position c[h,f] in the unit period immediately before the missing period and the position c[h,f] in the unit period immediately after the missing period is calculated as the position c[h,f] in the missing period.

[Image extraction unit 311]
As described above, the performance image G1 includes the keyboard image g1 and the finger image g2. The image extraction unit 311 in Fig. 3 extracts a specific area (hereinafter referred to as "specific area") B from the performance image G1, as illustrated in Fig. 7. The specific area B is an area of the performance image G1 that includes the keyboard image g1 and the finger image g2. The finger image g2 corresponds to an image of at least a part of the user's body.

Figure 8 is a flowchart illustrating the specific steps of the process (hereinafter referred to as "image extraction process") in which the image extraction unit 311 extracts a specific area B from the performance image G1. The image extraction process includes an area estimation process Sb1 and an area extraction process Sb2.

The area estimation process Sb1 is a process for estimating a specific area B for the performance image G1 represented by the image data D1. Specifically, the image extraction unit 311 generates an image processing mask M representing the specific area B from the image data D1 by the area estimation process Sb1. As illustrated in FIG. 7, the image processing mask M is a mask of the same size as the performance image G1, and is composed of multiple elements corresponding to different pixels of the performance image G1. Specifically, the image processing mask M is a binary mask in which each element in the area corresponding to the specific area B of the performance image G1 is set to the numerical value "1", and each element in the area other than the specific area B is set to the numerical value "0". The control device 11 executes the area estimation process Sb1 to realize an element (area estimation unit) that estimates the specific area B of the performance image G1.

3, the image extraction unit 311 uses an estimation model 51 to generate the image processing mask M. That is, the image extraction unit 311 generates the image processing mask M by inputting image data D1 representing the performance image G1 to the estimation model 51. The estimation model 51 is a statistical model that learns the relationship between the image data D1 and the image processing mask M by machine learning. The estimation model 51 is configured, for example, by a deep neural network (DNN). For example, any type of deep neural network such as a convolutional neural network (CNN) or a recurrent neural network (RNN) is used as the estimation model 51. The estimation model 51 may be configured by combining multiple types of deep neural networks. In addition, additional elements such as a long short-term memory (LSTM) may be installed in the estimation model 51.

Figure 9 is an explanatory diagram of machine learning that establishes an estimation model 51. For example, the estimation model 51 is established through machine learning by a machine learning system 900 that is separate from the performance analysis system 100, and the estimation model 51 is provided to the performance analysis system 100. The machine learning system 900 is a server system that can communicate with the performance analysis system 100 via a communication network such as the Internet. The estimation model 51 is transmitted from the machine learning system 900 to the performance analysis system 100 via the communication network.

A plurality of learning data T are used for the machine learning of the estimation model 51. Each of the plurality of learning data T is composed of a combination of learning image data Dt and learning image processing mask Mt. The image data Dt represents a known image including a keyboard image g1 of a keyboard instrument and an image of the surroundings of the keyboard instrument. The type of keyboard instrument and the shooting conditions (e.g., shooting range and shooting direction) differ for each image data Dt. That is, the image data Dt is prepared in advance by shooting each of a plurality of types of keyboard instruments under different shooting conditions. Note that the image data Dt may be prepared by a known image synthesis technique. The image processing mask Mt of each learning data T is a mask representing a specific region B of the known image represented by the image data Dt of the learning data T. Specifically, the elements in the area corresponding to the specific region B of the image processing mask Mt are set to the numerical value "1", and the elements in the area other than the specific region B are set to the numerical value "0". That is, the image processing mask Mt means the correct answer that the estimation model 51 should output in response to the input of the image data Dt.

The machine learning system 900 calculates an error function that represents the error between the image processing mask M output by an initial or provisional model (hereinafter referred to as the "provisional model") 51a when image data Dt of each training data T is input, and the image processing mask M of the training data T. Then, the machine learning system 900 updates multiple variables of the provisional model 51a so as to reduce the error function. The provisional model 51a at the time when the above process is repeated for each of the multiple training data T is determined as the estimated model 51. Therefore, the estimated model 51 outputs an image processing mask M that is statistically valid for unknown image data D1 under the underlying relationship between the image data Dt and the image processing mask Mt in the multiple training data T. In other words, the estimated model 51 is a trained model that has learned the relationship between the image data Dt and the image processing mask Mt.

As described above, in the first embodiment, an image processing mask M representing a specific region B is generated by inputting image data D1 of a performance image G1 into a machine-learned estimation model 51. Therefore, a specific region B can be identified with high accuracy for a variety of unknown performance images G1.

The area extraction process Sb2 in FIG. 8 is a process for extracting a specific area B from the performance image G1 represented by the image data D1. Specifically, the area extraction process Sb2 is an image process for relatively emphasizing the specific area B by selectively removing areas other than the specific area from the performance image G1. The image extraction unit 311 in the first embodiment generates image data D2 by applying an image processing mask M to the image data D1 (performance image G1). Specifically, the image extraction unit 311 multiplies the pixel value of each pixel in the performance image G1 by an element of the image processing mask M corresponding to the pixel. By the area extraction process Sb2, as illustrated in FIG. 7, image data D2 representing an image (hereinafter referred to as "performance image G2") from which areas other than the specific area B from the performance image G1 have been removed is generated. That is, the performance image G2 represented by the image data D2 is an image from which the keyboard image g1 and the fingering image g2 have been extracted from the performance image G1. The control device 11 executes the area extraction process Sb2 to realize an element (area extraction unit) that extracts a specific area B of the performance image G1.

[Projective transformation unit 314]
The position c[h,f] of each finger estimated by the finger position estimation process is a coordinate in the x-y coordinate system set in the performance image G1. The shooting conditions of the keyboard instrument 200 by the image capture device 15 may differ depending on various circumstances such as the environment in which the keyboard instrument 200 is used. For example, it is assumed that the shooting range is too wide (or too narrow) compared to the ideal shooting conditions exemplified in FIG. 2, or the shooting direction is inclined with respect to the vertical direction. The numerical values of the coordinate x[h,f] and the coordinate y[h,f] at each position c[h,f] depend on the shooting conditions of the performance image G1 by the image capture device 15. Therefore, the projection transformation unit 314 of the first embodiment transforms (image registration) the position c[h,f] of each finger in the performance image G1 into a position C[h,f] in the X-Y coordinate system that is substantially independent of the shooting conditions by the image capture device 15. The finger position data F generated by the finger position data generation unit 31 is data representing the position C[h,f] after transformation by the projective transformation unit 314. That is, the finger position data F specifies the positions C[1,1] to C[1,5] of each finger on the user's left hand and the positions C[2,1] to C[2,5] of each finger on the user's right hand.

The X-Y coordinate system is set to a predetermined image (hereinafter referred to as the "reference image") Gref, as exemplified in FIG. 10. The reference image Gref is an image of the keyboard of a standard keyboard instrument (hereinafter referred to as the "reference instrument") photographed under standard photographing conditions. Note that the reference image Gref is not limited to an image of an actual keyboard. For example, an image synthesized by a known image synthesis technique may be used as the reference image Gref. Image data (hereinafter referred to as the "reference data") Dref representing the reference image Gref and auxiliary data A related to the reference image Gref are stored in the storage device 12.

The auxiliary data A is data that specifies the combination of an area (hereinafter referred to as a "unit area") Rn in which each key 21 of the reference instrument exists in the reference image Gref, and the pitch n corresponding to that key 21. In other words, the auxiliary data A is data that defines the unit area Rn corresponding to each pitch n in the reference image Gref.

To convert a position c[h,f] in the x-y coordinate system to a position C[h,f] in the X-Y coordinate system, a projective transformation using a transformation matrix W is used, as expressed in the following formula (2). The symbol X in formula (2) means the coordinate on the X-axis in the X-Y coordinate system, and the symbol Y means the coordinate on the Y-axis. The symbol s is an adjustment value for matching the scale between the x-y coordinate system and the X-Y coordinate system.

[Matrix generation unit 312]
The matrix generation unit 312 in Fig. 3 generates a transformation matrix W of Equation (2) that the projection transformation unit 314 applies to the projection transformation. Fig. 11 is a flowchart illustrating a specific procedure of the process (hereinafter referred to as "matrix generation process") in which the matrix generation unit 312 generates the transformation matrix W. The matrix generation process of the first embodiment is executed with the performance image G2 (image data D2) after the image extraction process as the processing target. According to the above configuration, compared to a configuration in which the matrix generation process is executed with the entire performance image G1 including areas other than the specific area B as the processing target, it is possible to generate an appropriate transformation matrix W that accurately approximates the keyboard image g1 to the reference image Gref.

The matrix generation process includes an initial setting process Sc1 and a matrix update process Sc2. The initial setting process Sc1 is a process for setting the initial matrix W0, which is the initial value of the transformation matrix W. The details of the initial setting process Sc1 will be described later.

The matrix update process Sc2 is a process for generating a transformation matrix W by iteratively updating the initial matrix W0. That is, the projection transformation unit 314 generates the transformation matrix W by iteratively updating the initial matrix W0 so that the keyboard image g1 of the performance image G2 approaches the reference image Gref by projective transformation using the transformation matrix W. For example, the transformation matrix W is generated so that the coordinate X/s on the X axis of a specific point in the reference image Gref approximates or matches the coordinate x on the x axis of a point corresponding to the point in the keyboard image g1, and the coordinate Y/s on the Y axis of a specific point in the reference image Gref approximates or matches the coordinate y on the y axis of a point corresponding to the point in the keyboard image g1. That is, the transformation matrix W is generated so that the coordinates of the key 21 corresponding to a specific pitch in the keyboard image g1 are transformed into the coordinates of the key 21 corresponding to the pitch in the reference image Gref by projective transformation to which the transformation matrix W is applied. The control device 11 executes the matrix update process Sc2 illustrated above to realize an element (matrix generation unit 312) that generates the transformation matrix W.

The matrix update process Sc2 is assumed to be a process that updates the transformation matrix W so that image features such as SIFT (Scale-Invariant Feature Transform) are closer between the reference image Gref and the keyboard image g1. However, because the keyboard image g1 repeats a pattern in which multiple keys 21 are arranged in the same way, it may not be possible to properly estimate the transformation matrix W using the image features.

Considering the above circumstances, the matrix generation unit 312 of the first embodiment iteratively updates the initial matrix W0 in the matrix update process Sc2 so that the enhanced correlation coefficient (ECC) between the reference image Gref and the keyboard image g1 increases (ideally maximizes). According to the above embodiment, compared to the above embodiment using image features, it is possible to generate an appropriate transformation matrix W that can accurately approximate the keyboard image g1 to the reference image Gref. The generation of the transformation matrix W using the enhanced correlation coefficient is also disclosed in Georgios D. Evangelidis and Emmanouil Z. Psarakis, "Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 10, October 2008. As mentioned above, the extended correlation coefficient is suitable for generating the transformation matrix W used to transform the keyboard image g1, but the transformation matrix W may be generated so that the image features of the aforementioned SIFT, etc., are closer between the reference image Gref and the keyboard image g1.

The projection transformation unit 314 in FIG. 3 executes a projection transformation process. The projection transformation process is a projection transformation of the performance image G1 using the transformation matrix W generated by the matrix generation process. The projection transformation process transforms the performance image G1 into an image (hereinafter referred to as a "transformed image") captured under the same shooting conditions as the reference image Gref. For example, the area of the transformed image corresponding to the key 21 of pitch n substantially coincides with the unit area Rn of the pitch n in the reference image Gref. In addition, the x-y coordinate system of the transformed image substantially coincides with the X-Y coordinate system of the reference image Gref. In the projection transformation process described above, the projection transformation unit 314 transforms the position c[h,f] of each finger into the position C[h,f] of the X-Y coordinate system, as expressed by the above-mentioned formula (2). The control device 11 executes the projection transformation process exemplified above, thereby realizing an element (projection transformation unit 314) that executes the projection transformation of the performance image G1.

The display control unit 40 causes the display device 14 to display the transformed image generated by the projective transformation process. For example, the display control unit 40 causes the display device 14 to display the transformed image and the reference image Gref in a mutually overlapping state. As described above, the area of the transformed image corresponding to the key 21 of each pitch n and the unit area Rn of the reference image Gref corresponding to that pitch n mutually overlap.

As described above, in the first embodiment, a transformation matrix W is generated so that the keyboard image g1 of the performance image G1 approaches the reference image Gref, and a projective transformation process using the transformation matrix W is executed on the performance image G1. Therefore, the performance image G1 of the keyboard instrument 200 played by the user can be transformed into a transformed image that corresponds to the shooting conditions of the reference instrument in the reference image Gref.

Figure 12 is a flow chart illustrating the specific steps of the initial setting process Sc1. When the initial setting process Sc1 starts, the projective transformation unit 314 displays the setting screen 62 illustrated in Figure 13 on the display device 14 (Sc11). The setting screen 62 includes a performance image G1 captured by the image capture device 15 and instructions 622 for the user. The instructions 622 are a message to select an area (hereinafter referred to as a "target area") 621 corresponding to one or more specific pitches (hereinafter referred to as a "target pitch") n from the keyboard image g1 in the performance image G1. The user operates the operation device 13 while viewing the setting screen 62 to select the target area 621 corresponding to the target pitch n from the performance image G1. The projective transformation unit 314 accepts the selection of the target area 621 by the user (Sc12).

The projection transformation unit 314 identifies one or more unit regions Rn specified by the auxiliary data A for the target pitch n in the reference image Gref represented by the reference data Dref (Sc13). The projection transformation unit 314 then calculates an initial matrix W0 as a matrix for projectively transforming the target region 621 of the performance image G1 onto one or more unit regions Rn identified from the reference image Gref (Sc14). As can be understood from the above explanation, the initial setting process Sc1 of the first embodiment is a process for setting the initial matrix W0 so that the target region 621 specified by the user in the keyboard image g1 approaches the unit region Rn corresponding to the target pitch n in the reference image Gref by projection transformation using the initial matrix W0.

In order to generate an appropriate transformation matrix W by the matrix update process Sc2, it is important to set the initial matrix W0. In particular, in the case where the extended correlation coefficient is used in the matrix update process Sc2, the suitability of the initial matrix W0 tends to affect the suitability of the final transformation matrix W. In the first embodiment, the initial matrix W0 is set so that the target area 621 in the performance image G1 according to the instruction from the user approaches the unit area Rn corresponding to the target pitch n in the reference image Gref. Therefore, an appropriate transformation matrix W that can accurately approximate the keyboard image g1 to the reference image Gref can be generated. In the first embodiment, the area in the performance image G1 specified by the user through the operation of the operation device 13 is used as the target area 621 to set the initial matrix W0. Therefore, compared to the case where the area corresponding to the target pitch n in the performance image G1 is estimated by calculation processing, for example, an appropriate initial matrix W0 can be generated while reducing the processing load. Note that in the above description, the initial setting process Sc1 is performed on the performance image G1, but the initial setting process Sc1 may be performed on the performance image G2.

B: Fingering data generating unit 32
As described above, the fingering data generating unit 32 in Fig. 3 generates fingering data Q by using the performance data P generated by the keyboard instrument 200 and the finger position data F generated by the finger position data generating unit 31. The fingering data Q is generated for each unit period. The fingering data generating unit 32 in the first embodiment includes a probability calculating unit 321 and a fingering estimating unit 322. Note that in the above description, one finger of the user is represented by a combination of variables h and f, but in the following description, one finger of the user is represented by a finger number k (k = 1 to 10). Therefore, in the following description, the position C[h, f] specified for each finger by the finger position data F will be expressed as position C[k].

[Probability Calculation Unit 321]
The probability calculation unit 321 calculates the probability p that the pitch n specified by the performance data P is played by the finger with finger number k for each finger number k. The probability p is an index of accuracy (likelihood) that the finger with finger number k operated the key 21 of the pitch n. The probability calculation unit 321 calculates the probability p depending on whether the position C[k] of the finger with finger number k is within the unit area Rn of the pitch n. The probability calculation unit 321 calculates the probability p for each unit period on the time axis. Specifically, when the performance data P specifies the pitch n, the probability calculation unit 321 calculates the probability p(C[k]|ηk=n) by the calculation of the following mathematical formula (3).

The condition "ηk=n" in the probability p(C[k]|ηk=n) means that the finger with finger number k is playing pitch n. In other words, the probability p(C[k]|ηk=n) means the probability that position C[k] is observed for finger with finger number k when that finger is playing pitch n.

The symbol I(C[k]∈Rn) in formula (3) is an indicator function that is set to the value "1" when the position C[k] is within the unit region Rn, and is set to the value "0" when the position C[k] is outside the unit region Rn. The symbol |Rn| means the area of the unit region Rn. The symbol ν(0,σ ² E) means the observation noise, and is expressed by a normal distribution with mean 0 and variance σ ^2. The symbol E is a 2-row, 2-column unit matrix. The symbol * means the convolution of the observation noise ν(0,σ ² E).

As can be understood from the above explanation, the probability p(C[k]|ηk=n) calculated by the probability calculation unit 321 is the likelihood that, under the condition that the pitch n specified by the performance data P is played by the finger with finger number k, the position of that finger is the position C[k] specified for that finger by the finger position data F. Therefore, the probability p(C[k]|ηk=n) is maximum when the position C[k] of the finger with finger number k is within the unit area Rn of the performance state, and decreases as the position C[k] moves away from the unit area Rn.

数式(4)の記号|Ｒ|は、参照画像ＧrefにおけるＮ個の単位領域Ｒ1～ＲNの総面積を意味する。数式(4)から理解される通り、利用者が何れの鍵２１も操作していない状態では、確率ｐ(C[k]|ηk=0)は、全部の指番号ｋについて共通の数値（１/|Ｒ|）に設定される。 On the other hand, when the performance data P does not specify any pitch n, that is, when the user does not operate any of the N keys 21, the probability calculation unit 321 calculates the probability p(C[k]|ηk=0) of each fingering using the following formula (4).

The symbol |R| in formula (4) means the total area of N unit regions R1 to RN in the reference image Gref. As can be seen from formula (4), when the user is not operating any of the keys 21, the probability p(C[k]|ηk=0) is set to a common value (1/|R|) for all finger numbers k.

As described above, during the period in which the performance data P specifies pitch n, multiple probabilities p(C[k]|ηk=n) corresponding to different fingers are calculated for each unit period on the time axis. On the other hand, during each unit period in which the performance data P does not specify pitch n, multiple probabilities p(C[k]|ηk=0) corresponding to different fingers are set to a sufficiently small fixed value (1/|R|).

[Fingering Estimation Unit 322]
The fingering estimation unit 322 estimates the fingering of the user. Specifically, the fingering estimation unit 322 estimates the finger (finger number k) that played the pitch n specified by the performance data P from the probability p(C[k]|ηk=n) of each finger. The fingering estimation unit 322 estimates the finger number k (generates the fingering data Q) every time the probability p(C[k]|ηk=n) of each finger is calculated (i.e., every unit period). Specifically, the fingering estimation unit 322 identifies the finger number k that corresponds to the maximum value among a plurality of probabilities p(C[k]|ηk=n) corresponding to different fingers. Then, the fingering estimation unit 322 generates the fingering data Q that specifies the pitch n specified by the performance data P and the finger number k identified from the probability p(C[k]|ηk=n).

If the maximum value of the multiple probabilities p(C[k]|ηk=n) falls below a predetermined threshold during the period in which the performance data P specifies the pitch n, this means that the reliability of the fingering estimation result is low. Therefore, during the unit period in which the maximum value of the multiple probabilities p(C[k]|ηk=n) falls below the threshold, the fingering estimation unit 322 sets the finger number k to an invalid value, meaning that the estimation result is invalid. For notes in which the finger number k is set to an invalid value, the display control unit 40 displays the note image 611 in a manner different from the normal note image 611, as shown in the example of FIG. 4, and displays the symbol "??", meaning that the estimation result of the finger number k is invalid. The configuration and operation of the fingering data generation unit 32 are as described above.

Figure 14 is a flowchart illustrating the specific steps of the process executed by the performance analysis unit 30 (hereinafter referred to as the "performance analysis process"). For example, the performance analysis process is started in response to an instruction from the user via the operation device 13.

When the performance analysis process is started, the control device 11 (image extraction unit 311) executes the image extraction process of FIG. 8 (S11). That is, the control device 11 generates a performance image G2 by extracting a specific area B that includes the keyboard image g1 and the fingering image g2 from the performance image G1. As described above, the image extraction process includes an area estimation process Sb1 and an area extraction process Sb2.

After executing the image extraction process, the control device 11 (matrix generation unit 312) executes the matrix generation process of FIG. 11 (S12). That is, the control device 11 generates the transformation matrix W by iteratively updating the initial matrix W0 so as to increase the extended correlation coefficient between the reference image Gref and the keyboard image g1. As described above, the matrix generation process includes the initial setting process Sc1 and the matrix update process Sc2.

Once the transformation matrix W is generated, the control device 11 repeats the process exemplified below (S13 to S18) for each unit period. First, the control device 11 (finger position estimation unit 313) executes the finger position estimation process of FIG. 5 (S13). That is, the control device 11 estimates the positions c[h,f] of the fingers of the user's left and right hands by analyzing the performance image G1. As described above, the finger position estimation process includes the image analysis process Sa1, the left/right determination process Sa2, and the interpolation process Sa3.

The control device 11 (projective transformation unit 314) executes a projective transformation process (S14). That is, the control device 11 generates a transformed image by projective transformation of the performance image G1 using the transformation matrix W. In the projective transformation process, the control device 11 transforms the positions c[h,f] of each of the user's fingers into positions C[h,f] in the X-Y coordinate system, and generates finger position data F that represents the positions C[h,f] of each finger.

When the finger position data F is generated by the above process, the control device 11 (probability calculation unit 321) executes a probability calculation process (S15). That is, the control device 11 calculates the probability p(C[k]|ηk=n) that the pitch n specified by the performance data P was played by each finger with finger number k. Then, the control device 11 (fingering estimation unit 322) executes a fingering estimation process (S16). That is, the control device 11 estimates the finger number k of the finger that played the pitch n from the probability p(C[k]|ηk=n) of each finger, and generates fingering data Q that specifies the pitch n and finger number k.

When the fingering data Q is generated by the above process, the control device 11 (display control unit 40) updates the analysis screen 61 according to the fingering data Q (S17). The control device 11 also determines whether a predetermined termination condition is met (S18). For example, if the user issues an instruction to end the performance analysis process by operating the operation device 13, the control device 11 determines that the termination condition is met. If the termination condition is not met (S18: NO), the control device 11 repeats the processes following the finger position estimation process (S13 to S18) for the immediately following unit period. On the other hand, if the termination condition is met (S18: YES), the control device 11 ends the performance analysis process.

As described above, in the first embodiment, fingering data Q is generated using finger position data F generated by analyzing the performance image G1 and performance data P representing the performance by the user. Therefore, fingering can be estimated with a high degree of accuracy compared to a configuration in which fingering is estimated only from the performance data P.

In addition, in the first embodiment, the position c[h,f] of each finger estimated by the finger position estimation process is transformed using a transformation matrix W for projective transformation that brings the keyboard image g1 closer to the reference image Gref. That is, the position C[h,f] of each finger is estimated based on the reference image Gref. Therefore, fingering can be estimated with high accuracy compared to a configuration in which the position c[h,f] of each finger is not transformed to a position based on the reference image Gref.

In the first embodiment, a specific region B including the keyboard image g1 is extracted from the performance image G1. Therefore, as described above, an appropriate transformation matrix W can be generated that can accurately approximate the keyboard image g1 to the reference image Gref. Furthermore, by extracting the specific region B, it is possible to improve the usability of the performance image G1. In particular, in the first embodiment, a specific region B including the keyboard image g1 and the fingering image g2 is extracted from the performance image G1. Therefore, a performance image G2 can be generated that allows efficient visual confirmation of the appearance of the keys 22 of the keyboard instrument 200 and the appearance of the user's fingers.

2: Second embodiment A second embodiment will be described. Note that, in each of the following exemplary embodiments, elements that have the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions of each will be omitted as appropriate.

In the first embodiment, the probability p(C[k]|ηk=n) is calculated depending on whether the position C[k] of finger number k is present within the unit area Rn of pitch n. Assuming that only one finger is present within the unit area Rn, the fingering can be estimated with high accuracy even in the first embodiment. However, in an actual performance of the keyboard instrument 200, it is assumed that there are multiple finger positions C[k] within one unit area Rn.

For example, as illustrated in FIG. 15, when a user operates one key 21 with the middle finger of the left hand and moves the index finger of the left hand vertically upward, the middle finger and index finger of the left hand overlap each other in the performance image G1. That is, the position C[k] of the middle finger and the position C[k] of the index finger of the left hand are present in one unit area Rn. In addition, in a performance method in which a user operates a key 21 with one finger and passes other fingers above or below the finger (finger passing), multiple fingers may overlap each other. When multiple fingers overlap each other in one unit area Rn as described above, the method of the first embodiment may not be able to estimate the fingering with high accuracy. The second embodiment is a form for solving the above problems. Specifically, in the second embodiment, the positional relationship of multiple fingers and the temporal fluctuation (variation) of the position of each finger are taken into account in the fingering estimation.

Figure 16 is a block diagram illustrating the functional configuration of the performance analysis system 100 in the second embodiment. The performance analysis system 100 in the second embodiment has the same elements as the first embodiment, and further includes a control data generator 323.

The control data generator 323 generates N pieces of control data Z[1] to Z[N] corresponding to different pitches n. FIG. 17 is a schematic diagram of the control data Z[n] corresponding to any one pitch n. The control data Z[n] is vector data that represents the characteristics of the relative position (hereinafter referred to as "relative position") C'[k] of each finger with respect to the unit area Rn of pitch n. The relative position C'[k] is information obtained by converting the position C[k] represented by the finger position data F into a relative position with respect to the unit area Rn.

The control data Z[n] corresponding to one pitch n includes the pitch n, as well as the position average Za[n,k], position variance Zb[n,k], velocity average Zc[n,k], and velocity variance Zd[n,k] for each of the multiple fingers. The position average Za[n,k] is the average of the relative position C'[k] within a period of a predetermined length including the current unit period (hereinafter referred to as the "observation period"). The observation period is, for example, a period equivalent to multiple unit periods arranged forward on the time axis with the current unit period at the end. The position variance Zb[n,k] is the variance of the relative position C'[k] within the observation period. The velocity average Zc[n,k] is the average of the velocity (i.e., the rate of change) at which the relative position C'[k] changes within the observation period. The velocity variance Zd[n,k] is the variance of the velocity at which the relative position C'[k] changes within the observation period.

As described above, the control data Z[n] includes information about the relative position C'[k] for each of the multiple fingers (Za[n,k], Zb[n,k], Zc[n,k], Zd[n,k]). Therefore, the control data Z[n] is data that reflects the positional relationships of the user's multiple fingers. In addition, the control data Z[n] includes information about the fluctuation in the relative position C'[k] for each of the multiple fingers (Zb[n,k], Zd[n,k]). Therefore, the control data Z[n] is data that reflects the temporal fluctuation in the position of each finger.

The probability calculation process by the probability calculation unit 321 in the second embodiment uses multiple estimation models 52[k] (52[1] to 52[10]) prepared in advance for different fingers. The estimation model 52[k] for each finger is a trained model that has learned the relationship between the control data Z[n] and the probability p[k] for that finger. The probability p[k] is an index (probability) of the likelihood that the finger with finger number k played the pitch n specified by the performance data P. The probability calculation unit 321 calculates the probability p[k] for each of the multiple fingers by inputting N pieces of control data Z[1] to Z[N] into the estimation model 52[k] for that finger.

The estimation model 52[k] corresponding to any one finger number k is a logistic regression model expressed by the following formula (5).

The variables βk and ωk,n in formula (5) are set by machine learning using the machine learning system 900. That is, each estimation model 52[k] is established by machine learning using the machine learning system 900, and each estimation model 52[k] is provided to the performance analysis system 100. For example, the variables βk and ωk,n of each estimation model 52[k] are transmitted from the machine learning system 900 to the performance analysis system 100.

Fingers located above a finger in a key-pressing state, or fingers moving above or below a finger in a key-pressing state, tend to move more easily than fingers in a key-pressing state. Taking the above tendencies into consideration, the estimation model 52[k] learns the relationship between the control data Z[n] and the probability p[k] so that the probability p[k] is a small value for fingers with a high rate of change in the relative position C'[k]. The probability calculation unit 321 inputs the control data Z[n] to each of the multiple estimation models 52[k], thereby calculating multiple probabilities p[k] for different fingers for each unit period.

The fingering estimation unit 322 estimates the fingering of the user by a fingering estimation process that applies multiple probabilities p[k]. Specifically, the fingering estimation unit 322 estimates the finger (finger number k) that played the pitch n specified by the performance data P from the probability p[k] of each finger. The fingering estimation unit 322 estimates the finger number k (generates the fingering data Q) every time the probability p[k] of each finger is calculated (i.e., every unit period). Specifically, the fingering estimation unit 322 identifies the finger number k that corresponds to the maximum value among the multiple probabilities p[k] corresponding to different fingers. The fingering estimation unit 322 then generates the fingering data Q that specifies the pitch n specified by the performance data P and the finger number k identified from the probability p[k].

Figure 18 is a flow chart illustrating the specific steps of the performance analysis process in the second embodiment. In the performance analysis process of the second embodiment, the generation of control data Z[n] (S19) is added to the same process as in the first embodiment. Specifically, the control device 11 (control data generation unit 323) generates N pieces of control data Z[1] to Z[N] corresponding to different pitches n from the finger position data F (i.e., the positions C[h,f] of each finger) generated by the finger position data generation unit 31.

The control device 11 (probability calculation unit 321) calculates a probability p[k] corresponding to finger number k by a probability calculation process in which N pieces of control data Z[1] to Z[N] are input to each estimation model 52[k] (S15). In addition, the control device 11 (fingering estimation unit 322) estimates the user's fingering by a fingering estimation process that applies the multiple probabilities p[k] (S16). The operations of the elements other than the fingering data generation unit 32 (S11 to S14, S17 to S18) are the same as in the first embodiment.

In the second embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the second embodiment, the control data Z[k] input to the estimation model 52[k] includes the average Za[n,k] and variance Zb[n,k] of the relative position C'[k] of each finger, and the average Zc[n,k] and variance Zd[n,k] of the rate of change of the relative position C'[k]. Therefore, the fingering of the user can be estimated with high accuracy even in a state where multiple fingers overlap each other due to, for example, finger under-taking.

In the above explanation, a logistic regression model is exemplified as the estimation model 52[k], but the type of estimation model 52[k] is not limited to the above example. For example, a statistical model such as a multi-layer perceptron may be used as the estimation model 52[k]. Also, a deep neural network such as a convolutional neural network or a recurrent neural network may be used as the estimation model 52[k]. A combination of multiple types of statistical models may be used as the estimation model 52[k]. The various estimation models 52[k] exemplified above are collectively expressed as trained models that have learned the relationship between the control data Z[n] and the probability p[k].

19 is a flow chart illustrating a specific procedure of the performance analysis process in the third embodiment. After executing the image extraction process and the matrix generation process, the control device 11 refers to the performance data P to determine whether or not the user is playing the keyboard instrument 200 (S21). Specifically, the control device 11 determines whether or not any of the multiple keys 21 of the keyboard instrument 200 is being operated.

If the keyboard instrument 200 is being played (S21: YES), the control device 11 generates finger position data F (S13-S14), generates fingering data Q (S15-S16), and updates the analysis screen 61 (S17), as in the first embodiment. On the other hand, if the keyboard instrument 200 is not being played (S21: NO), the control device 11 transitions to step S18. In other words, the generation of finger position data F (S13-S14), the generation of fingering data Q (S15-S16), and the update of the analysis screen 61 (S17) are not executed.

The third embodiment achieves the same effects as the first embodiment. Furthermore, in the third embodiment, when the keyboard instrument 200 is not being played, the generation of finger position data F and fingering data Q is stopped. Therefore, the processing load required to generate fingering data Q can be reduced compared to a configuration in which the generation of finger position data F continues regardless of whether the keyboard instrument 200 is being played or not. The third embodiment is also applicable to the second embodiment.

4: Fourth embodiment The fourth embodiment is an embodiment in which the initial setting process Sc1 in each of the above-mentioned embodiments is modified. Fig. 20 is a flowchart illustrating a specific procedure of the initial setting process Sc1 executed by the control device 11 (matrix generation unit 312) of the fourth embodiment.

When the initial setting process Sc1 starts, the user operates a key 21 corresponding to a desired pitch (hereinafter referred to as a "specific pitch") n among the multiple keys 21 of the keyboard instrument 200 with a specific finger (hereinafter referred to as a "specific finger"). The specific finger is, for example, a finger (e.g., the index finger of the right hand) notified to the user by display on the display device 14 or an instruction manual for the keyboard instrument 200. As a result of the user's performance, performance data P specifying the specific pitch n is supplied from the keyboard instrument 200 to the performance analysis system 100. The control device 11 recognizes the user's performance of the specific pitch n by acquiring the performance data P from the keyboard instrument 200 (Sc15). The control device 11 identifies a unit area Rn corresponding to the specific pitch n among the N unit areas R1 to RN of the reference image Gref (Sc16).

On the other hand, the finger position data generation unit 31 generates finger position data F by finger position estimation processing. The finger position data F includes the positions C[h,f] of the specific fingers used by the user to play the specific pitch n. The control device 11 acquires the finger position data F to identify the positions C[h,f] of the specific fingers (Sc17).

The control device 11 sets the initial matrix W0 using the unit area Rn corresponding to the specific pitch n and the position C[h,f] of the specific finger represented by the finger position data F (Sc18). That is, the control device 11 sets the initial matrix W0 so that the position C[h,f] of the specific finger represented by the finger position data F approaches the unit area Rn of the specific pitch n in the reference image Gref. Specifically, a matrix for projecting the position C[h,f] of the specific finger onto the center of the unit area Rn is set as the initial matrix W0.

The fourth embodiment also achieves the same effect as the first embodiment. In addition, in the fourth embodiment, when a user plays a desired specific pitch n with a specific finger, the initial matrix W0 is set so that the position c[h,f] of the specific finger in the performance image G1 approaches the portion (unit area Rn) of the reference image Gref corresponding to the specific pitch n. Since the user only needs to play the desired pitch n, the burden of the user's work required to set the initial matrix W0 is reduced compared to the first embodiment in which the user must select the target area 621 by operating the operation device 13. On the other hand, according to the first embodiment in which the user specifies the target area 621, it is not necessary to estimate the position C[h,f] of the user's finger, so compared to the second embodiment, it is possible to set an appropriate initial matrix W0 while reducing the influence of estimation errors. Note that the fourth embodiment is also applicable to the second or third embodiment.

In the fourth embodiment, it is assumed that the user plays one specific pitch n, but the user may play multiple specific pitches n with specific fingers. The control device 11 sets the initial matrix W0 for each of the multiple specific pitches n so that the position C[h,f] of the specific finger when playing the specific pitch n approaches the unit area Rn of the specific pitch n.

5: Fifth embodiment Fig. 21 is a block diagram illustrating a functional configuration of a performance analysis system 100 in a fifth embodiment. The performance analysis system 100 in the fifth embodiment includes a sound collection device 16. The sound collection device 16 collects the sound reproduced from the keyboard instrument 200 by the user's performance to generate an audio signal V. The audio signal V is a time-domain audio signal representing the waveform of the sound reproduced by the keyboard instrument 200. The sound collection device 16, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly. The time series of samples constituting the audio signal V may be interpreted as "performance data P".

The control device 11 of the performance analysis system 100 functions as a performance analysis unit 30 by executing a program stored in the storage device 12. The performance analysis unit 30 generates fingering data Q using the audio signal V supplied from the sound collection device 16 and the image data D1 supplied from the image capture device 15. As in the first embodiment, the fingering data Q specifies the pitch n corresponding to the key 21 operated by the user and the finger number k of the finger used by the user to operate the key 21. In the first embodiment, the pitch n is specified by the performance data P, but the audio signal V in the fifth embodiment is not a signal that directly specifies the pitch n. Therefore, the performance analysis unit 30 simultaneously estimates the pitch n and the finger number k using the audio signal V and the image data D1.

A latent variable wt _,n,k is assumed to estimate the pitch n and the finger number k. The symbol t is a variable indicating time. One unit period on the time axis may be indicated by the variable t. In addition, the finger number k in the fifth embodiment is set to one of eleven values including ten values (k=1 to 10) corresponding to different fingers and a predetermined invalid value (k=0).

A latent variable wt _,n,k is prepared for each combination of pitch n and finger number k. The latent variable _wt,n,k is a variable for one-hot expression that is set to either of the two values "0" and "1." The value "1" of the latent variable wt _,n,k means that pitch n is played by the finger with finger number k, and the value "0" of the latent variable wt _,n,k means that no finger is used in the performance.

Also, assume a posterior probability U _t,n and a probability π _t,n,k . The posterior probability U _t,n is the posterior probability that pitch n is sounded at time t under the conditions under which the audio signal V is observed. Therefore, the probability (1-U _t,n ) corresponds to the probability that the latent variable w _t,n,0 is the numerical value "1" under the conditions under which the audio signal V is observed (the probability that no pitch n is played). The posterior probability U _t,n is estimated by a known estimation model that has learned the relationship between the audio signal V and the posterior probability U t, _n . The estimation model is a trained model for automatic music transcription. For example, a deep neural network such as a convolutional neural network or a recurrent neural network is used as an estimation model for estimating the posterior probability U _t,n . The probability π _t,n,k is the probability that pitch n is played by a finger with finger number k when pitch n is being played.

数式(6)における右辺の第１項は、何れの音高ｎも発音されていない確率を意味し、第２項は、音高ｎが発音されている場合に当該音高ｎが指番号ｋの手指により演奏されている確率を意味する。 The probability p(w|V,π) of the latent variable w t, _n,k when an acoustic signal V and a probability π t, _n,k are observed is expressed by the following equation (6).

The first term on the right hand side of equation (6) represents the probability that no pitch n is sounded, and the second term represents the probability that, when pitch n is sounded, that pitch n is played by the finger with finger number k.

数式(7)における確率ｐ(C[k]|σ²,Rn)は、前掲の数式(3)または数式(4)で表現される確率である。 Furthermore, the probability p(C[k]|w) that a position C[k] is observed from the performance image G1 when a latent variable w _t,n,k is observed is expressed by the following formula (7).

The probability p(C[k]|σ ² , Rn) in formula (7) is the probability expressed by formula (3) or formula (4) shown above.

数式(8)の記号αは、対称ディリクレ分布の形状を規定する変数である。 In addition, as the prior distribution of the probability π _t,n,k , the symmetric Dirichlet distribution (Dir) expressed by the following equation (8) is assumed.

The symbol α in equation (8) is a variable that defines the shape of the symmetric Dirichlet distribution.

Under the above assumptions, by performing maximum a posteriori (MAP) estimation to maximize the posterior probability p(z|V,π,C[k]) of the latent variable wt _,n,k , it is possible to simultaneously estimate the presence or absence of pitch n and finger number k. However, since it is difficult to estimate the probability distribution of the posterior probability p(z|V,π,C[k]), in the fifth embodiment, a mean field approximation (variational Bayesian estimation) is considered.

Specifically, among the distributions factorized as shown in the following formula (9), a distribution that is most similar to the probability distribution of the posterior probability p(z|V,π,C[k]) is identified. For example, a distribution that has a minimum Kullback-Leibler (KL) distance from the posterior probability p(z|V,π,C[k]) is identified.

数式(10)の記号ｃは、複数の指番号ｋにわたる確率分布ρ_t,n,kの合計が「１」となるように当該確率分布ρ_t,n,kを正規化する係数である。また、記号〈 〉は、期待値を意味する。 Specifically, the performance analysis unit 30 repeats the calculations of the following formulas (10) and (11).

The symbol c in Equation (10) is a coefficient that normalizes the probability distribution ρ _t,n,k over multiple finger numbers k so that the sum of the probability distribution ρ _t,n,k is “1.” Also, the symbol <> denotes the expected value.

Specifically, the performance analysis unit 30 repeats the calculation of formula (10) and formula (11) for all combinations of pitch n and finger number k for one time t on the time axis. The performance analysis unit 30 determines the calculation result of formula (10) at the time point when the calculation of formula (10) and formula (11) is repeated a predetermined number of times as the probability distribution ρ t, _{n,k of the latent variable w t} , _n,k . The probability distribution ρ _t,n,k is calculated for each time t on the time axis.

However, in a configuration in which pitch n and finger number k are calculated for each time t from a probability distribution ρ _t,n,k calculated individually for each time t on the time axis, finger number k may change between times t before and after a user plays a note, or the duration of pitch n may be excessively short. Therefore, the performance analysis unit 30 of the fifth embodiment uses a hidden Markov model (HMM) to which the probability distribution ρ _t,n,k is applied to generate a time series of combinations of pitch n and finger number k (i.e., fingering data Q).

Specifically, the HMM for fingering estimation is composed of latent states corresponding to the sounding (key pressing) and mute of pitch n, and multiple latent states corresponding to different finger numbers k. Only three types of state transitions are allowed: (1) self-transition, (2) silence → any finger number k, and (3) any finger number k → silence, and the transition probabilities for other state transitions are set to "0". The above conditions are constraints for not changing the finger number k during the period in which one note is sounded. In addition, the expected value of the probability distribution ρ _t,n,k calculated by the calculation of formula (10) and formula (11) is set as the observation probability for each latent state of the HMM. The performance analysis unit 30 uses the above-described HMM to estimate a state sequence by dynamic programming such as the Viterbi algorithm. The performance analysis unit 30 generates a time series of fingering data Q according to the result of estimating the state sequence.

According to the fifth embodiment, fingering data Q is generated using the audio signal V and image data D1. That is, fingering data Q can be generated even in a situation where performance data P cannot be acquired. In addition, in the fifth embodiment, pitch n and finger number k are simultaneously estimated using audio signal V and image data D1, so fingering can be estimated with high accuracy while reducing the processing load compared to a form in which pitch n and finger number k are estimated separately. Note that the fifth embodiment is also applicable to the second to fourth embodiments.

6: Sixth Embodiment As exemplified in each of the above embodiments, the projective transformation unit 314 generates a transformed image from the performance image G1. That is, the projective transformation unit 314 changes the shooting conditions of the performance image G1. The sixth embodiment is an image processing system 700 that utilizes the above function of changing the shooting conditions of the performance image G1. Note that the performance analysis systems 100 of the first to fifth embodiments can also be expressed as image processing systems 700 if attention is focused on the processing of the performance image G1 by the projective transformation unit 314. Note that in the sixth embodiment, it is not essential to estimate the user's fingering.

Fig. 22 is a block diagram illustrating the functional configuration of an image processing system 700 in the sixth embodiment. Like the performance analysis system 100 in the first embodiment, the image processing system 700 includes a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. Like the first embodiment, the photographing device 15 photographs the keyboard instrument 200 under specific photographing conditions to generate a time series of image data D1 representing a performance image G1.

The storage device 12 stores multiple reference data Dref. Each of the multiple reference data Dref represents a reference image Gref captured of a reference instrument, which is the keyboard of a standard keyboard instrument. The shooting conditions of the reference instrument differ for each reference image Gref (each reference data Dref). Specifically, for example, one or more conditions of the shooting range or shooting direction differ for each reference image Gref. The storage device 12 also stores auxiliary data A for each reference data Dref.

The control device 11 executes a program stored in the storage device 12 to realize a matrix generation unit 312, a projective transformation unit 314, and a display control unit 40. The matrix generation unit 312 selectively uses one of a plurality of reference data Dref to generate a transformation matrix W. The projective transformation unit 314 generates image data D3 of a transformed image G3 from image data D1 of a performance image G1 by projective transformation using the transformation matrix W. The display control unit 40 causes the display device 14 to display the transformed image G3 represented by the image data D3.

Figure 23 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "first image processing") executed by the control device 11 of the sixth embodiment. For example, the first image processing is started in response to an instruction from a user via the operation device 13.

The user operates the operation device 13 to select one of a plurality of shooting conditions corresponding to different reference images Gref. The control device 11 (matrix generation unit 312) judges whether or not a selection of shooting conditions has been accepted from the user (S31). If a selection of shooting conditions has been accepted (S31: YES), the control device 11 (matrix generation unit 312) acquires reference data Dref (hereinafter referred to as "selected reference data Dref") corresponding to the shooting conditions selected by the user from among the plurality of reference data Dref stored in the storage device 12 (S32). The selection of shooting conditions by the user corresponds to an operation of selecting one of a plurality of reference images Gref (reference data Dref) corresponding to different shooting conditions.

The control device 11 (matrix generation unit 312) executes a matrix generation process similar to that of the first embodiment using the selected reference data Dref (S33). Specifically, the control device 11 sets an initial matrix W0 by an initial setting process Sc1 using the selected reference data Dref. The control device 11 also generates a transformation matrix W by a matrix update process Sc2 that iteratively updates the initial matrix W0 so that the keyboard image g1 of the performance image G1 approaches the reference image Gref of the selected reference data Dref. On the other hand, if the selection of the shooting conditions is not accepted (S31: NO), the selection of the reference data Dref (S32) and the matrix generation process (S33) are not executed.

The control device 11 (projective transformation unit 314) generates a transformed image G3 by performing a projective transformation process using the transformation matrix W on the performance image G1 (S34). The projective transformation process is the same as in the first embodiment. As a result of the projective transformation process, image data D3 representing the transformed image G3 is generated. Specifically, a transformed image G3 corresponding to shooting conditions equivalent to those of the reference image Gref of the selected reference data Dref is generated from the performance image G1. In other words, the transformed image G3 is an image obtained by transforming the shooting conditions of the performance image G1 into shooting conditions equivalent to those of the reference image Gref. As can be understood from the above explanation, according to the sixth embodiment, a transformed image G3 corresponding to the shooting conditions selected by the user is generated.

The control device 11 (display control unit 40) causes the display device 14 to display the transformed image G3 generated by the projective transformation process (S35). The control device 11 determines whether or not the end condition is met (S36). For example, when the user instructs the operation device 13 to end the first image process, the control device 11 determines that the end condition is met. If the end condition is not met (S36: NO), the control device 11 moves the process to step S31. That is, the generation of the transformation matrix W (S32-S33) and the generation and display of the transformed image G3 (S34-S35) are executed on the condition that the selection of the shooting conditions is accepted (S31: YES). On the other hand, if the end condition is met (S36: YES), the control device 11 ends the first image process.

As described above, in the sixth embodiment, a transformation matrix W is generated so that the keyboard image g1 in the performance image G1 approaches the reference image Gref, and a projective transformation process using the transformation matrix W is executed on the performance image G1. Therefore, the performance image G1 of the keyboard instrument 200 played by the user can be transformed into a transformation image G3 that corresponds to the shooting conditions of the reference instrument in the reference image Gref.

In the sixth embodiment, one of the multiple reference data Drefs with different shooting conditions is selectively used in the matrix generation process. Therefore, a converted image G3 corresponding to various shooting conditions can be generated from a performance image G1 captured under specific shooting conditions. In particular, in the sixth embodiment, the reference data Dref corresponding to the shooting conditions selected by the user among the multiple reference data Drefs is used in the matrix generation process, so that a converted image G3 corresponding to the shooting conditions desired by the user can be generated. By changing the shooting conditions of the performance image G1 as described above, a converted image G3 that can be used for various purposes can be generated. For example, a music instructor can perform the first image processing of the sixth embodiment on each of multiple performance images G1 captured by his/her own performance, and thus multiple converted images G3 with unified shooting conditions can be generated, for example, as teaching materials for music training.

7: Seventh Embodiment As exemplified in each of the above embodiments, the image extraction unit 311 extracts a specific area B including a keyboard image g1 and a fingering image g2 from the performance image G1. The seventh embodiment is an image processing system 700 that utilizes the above function of extracting the specific area B from the performance image G1. Note that the performance analysis systems 100 of the first to fifth embodiments can also be expressed as image processing systems 700 if attention is focused on the processing of the performance image G1 by the image extraction unit 311. Note that in the seventh embodiment, estimation of the user's fingering is not essential.

Figure 24 is a block diagram illustrating the functional configuration of an image processing system 700 in the seventh embodiment. Like the performance analysis system 100 in the first embodiment, the image processing system 700 includes a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. The photographing device 15 photographs the keyboard instrument 200 under specific photographing conditions to generate a time series of image data D1 representing a performance image G1. Like the previous embodiments, the performance image G1 includes a keyboard image g1 and a fingering image g2.

The control device 11 functions as an image extraction unit 311 and a display control unit 40 by executing a program stored in the storage device 12. The image extraction unit 311 generates image data D2 representing a performance image G2 obtained by extracting a partial area of the performance image G1. Specifically, similar to the first embodiment, the image extraction unit 311 executes an area estimation process Sb1 for generating an image processing mask M, and an area extraction process Sb2 for applying the image processing mask M to the performance image G1. The display control unit 40 causes the display device 14 to display the performance image G2 represented by the image data D2.

In the first embodiment, a single estimation model 51 is exemplified. In the seventh embodiment, the estimation model 51 used in the area estimation process Sb1 includes a first model 511 and a second model 512. Each of the first model 511 and the second model 512 is configured as a deep neural network such as a convolutional neural network or a recurrent neural network.

The first model 511 is a statistical model for generating a first mask representing a first region of the performance image G1. The first region is a region of the performance image G1 that includes the keyboard image g1. The finger image g2 is not included in the first region. The first mask is, for example, a binary mask in which each element in the first region is set to the numerical value "1" and each element in the region other than the first region is set to the numerical value "0". The image extraction unit 311 generates the first mask by inputting image data D1 representing the performance image G1 to the first model 511. In other words, the first model 511 is a trained model that has learned the relationship between the image data D1 and the first mask (first region) by machine learning.

The second model 512 is a statistical model for generating a second mask representing the second region of the performance image G1. The second region is a region of the performance image G1 that includes the fingering image g2. The keyboard image g1 is not included in the second region. The second mask is, for example, a binary mask in which each element in the second region is set to the numerical value "1" and each element in the region other than the second region is set to the numerical value "0". The image extraction unit 311 generates the second mask by inputting the image data D1 representing the performance image G1 to the second model 512. In other words, the second model 512 is a trained model that has learned the relationship between the image data D1 and the second mask (second region) by machine learning.

Figure 25 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "second image processing") executed by the control device 11 of the seventh embodiment. For example, the second image processing is started in response to an instruction from a user via the operation device 13.

When the second image processing is started, the control device 11 (image extraction unit 311) executes the area estimation processing Sb1 (S41 to S43). The area estimation processing Sb1 of the seventh embodiment includes the first estimation processing (S41), the second estimation processing (S42), and the area synthesis processing (S43).

The first estimation process is a process of estimating a first region of the performance image G1. Specifically, the control device 11 generates a first mask representing the first region by inputting image data D1 representing the performance image G1 to the first model 511 (S41). The second estimation process is a process of estimating a second region of the performance image G2. Specifically, the control device 11 generates a second mask representing the second region by inputting image data D1 representing the performance image G1 to the second model 512 (S42).

The area synthesis process is a process for generating an image processing mask M that represents a specific area B that includes the first area and the second area. Specifically, the specific area B represented by the image processing mask M corresponds to the sum of the first area and the second area. That is, the control device 11 generates the image processing mask M by synthesizing the first mask and the second mask (S43). As can be understood from the above explanation, the image processing mask M is a binary mask for extracting the specific area B that includes the keyboard image g1 and the fingering image g2 from the performance image G1, similar to the first embodiment.

The control device 11 (image extraction unit 311) uses the image processing mask M generated in the area estimation process Sb1 to execute the area extraction process Sb2 similar to that of the first embodiment (S44). That is, the control device 11 generates image data D2 representing the performance image G2 by extracting the specific area B from the performance image G1 represented by the image data D1 using the image processing mask M.

The control device 11 (display control unit 40) causes the display device 14 to display the performance image G2 generated by the area extraction process Sb2 (S45). The control device 11 determines whether or not the end condition is met (S46). For example, when the user instructs the operation device 13 to end the second image process, the control device 11 determines that the end condition is met. If the end condition is not met (S46: NO), the control device 11 shifts the process to step S41. That is, the area estimation process Sb1 (S41-S43), the area extraction process Sb2 (S44), and the display of the performance image G2 (S45) are executed. On the other hand, if the end condition is met (S46: YES), the control device 11 ends the second image process.

In the seventh embodiment, as in the first embodiment, a specific region B including the keyboard image g1 is extracted from the performance image G1. Therefore, it is possible to improve the usability of the performance image G1. In particular, in the seventh embodiment, a specific region B including the keyboard image g1 and the finger image g2 is extracted from the performance image G1. Therefore, it is possible to generate a performance image G2 that allows the appearance of the keyboard 22 of the keyboard instrument 200 and the appearance of the user's fingers to be efficiently visually recognized.

Furthermore, according to the seventh embodiment, a first region of the performance image G1 including the keyboard image g1 is estimated by the first model 511, and a second region of the performance image G1 including the finger image g2 is estimated by the second model 512. Therefore, compared to a configuration using a single estimation model 51 that simultaneously extracts both the keyboard image g1 and the finger image g2, a specific region B including the keyboard image g1 and the finger image g2 can be extracted with high accuracy. Furthermore, since each of the first model 511 and the second model 512 is established by individual machine learning, the processing load related to the machine learning of the first model 511 and the second model 512 is reduced.

It is also assumed that the image extraction unit 311 can be configured to switch between the first mode and the second mode. The first mode is an operation mode in which both the keyboard image g1 and the fingering image g2 are extracted from the performance image G1. That is, in the first mode, the image extraction unit 311 executes both the first estimation process and the second estimation process. Therefore, as in the seventh embodiment, an image processing mask M representing the specific region B is generated. That is, in the first mode, the specific region B including both the keyboard image g1 and the fingering image g2 is extracted from the performance image G1.

The second mode is an operating mode in which the keyboard image g1 is extracted from the performance image G1. That is, in the second mode, the image extraction unit 311 executes the first estimation process but does not execute the second estimation process. That is, the first mask generated by the first estimation process is determined as the image processing mask M to be applied to the region extraction process Sb2. Therefore, in the second mode, the keyboard image g1 is extracted from the performance image G1.

As described above, in a form that allows switching between the first mode and the second mode, it is possible to easily switch the target to be extracted from the performance image G1. In the above description, the image extraction unit 311 executes the first estimation process in the second mode, but a form in which the image extraction unit 311 executes the second estimation process while not executing the first estimation process in the second mode is also envisioned. In the above form, the finger image g2 is extracted from the performance image G1. As can be understood from the above example, the second mode is expressed as an operation mode in which one of the first estimation process and the second estimation process is executed.

8: Modifications Specific modifications to the above-mentioned embodiments are given below. Two or more of the following embodiments may be combined as long as they are not mutually contradictory.

(1) In each of the above-described embodiments, the matrix generation process is performed on the performance image G2 after processing by the image extraction process (FIG. 8). However, the matrix generation process may be performed on the performance image G1 captured by the image capture device 15. In other words, the image extraction process (image extraction unit 311) that generates the performance image G2 from the performance image G1 may be omitted.

In each of the above-mentioned embodiments, a finger position estimation process using the performance image G1 has been exemplified, but the finger position estimation process may also be performed using the performance image G2 after processing by the image extraction process. That is, the position C[h,f] of each of the user's fingers may be estimated by analyzing the performance image G2. Also, in each of the above-mentioned embodiments, a projective transformation process was performed on the performance image G1, but a projective transformation process may also be performed on the performance image G2 after processing by the image extraction process. That is, a transformed image may be generated by projective transformation of the performance image G2.

(2) In each of the above-described embodiments, the position c[h,f] of each finger of the user is converted to a position C[h,f] in the X-Y coordinate system by a projective transformation process, but finger position data F representing the position c[h,f] of each finger may be generated. In other words, the projective transformation process (projective transformation unit 314) that converts the position c[h,f] to the position C[h,f] may be omitted.

(3) In the first to fifth embodiments, the transformation matrix W generated immediately after the start of the performance analysis process is continuously used in the subsequent processes. However, the transformation matrix W may be updated at an appropriate time during the execution of the performance analysis process. For example, a form in which the transformation matrix W is updated when the position of the image capture device 15 relative to the keyboard instrument 200 changes is envisioned. Specifically, the transformation matrix W is updated when a change in the position of the image capture device 15 (hereinafter referred to as "position change") is detected by analysis of the performance image G1, or when a change in the position of the image capture device 15 is instructed by the user.

Specifically, the matrix generation unit 312 generates a transformation matrix δ that represents a position change (displacement) of the image capture device 15. For example, the relationship expressed by the following formula (12) is assumed for the coordinates (x, y) in the performance image G (G1, G2) after the position change.

The matrix generation unit 312 generates a transformation matrix δ so that the coordinate x'/ε calculated from the x coordinate of a specific point after the position change using formula (12) approximates or matches the x coordinate of a point in the performance image G before the position change that corresponds to the specific point, and the coordinate y'/ε calculated from the y coordinate of a specific point after the position change using formula (12) approximates or matches the y coordinate of a point in the performance image G before the position change that corresponds to the specific point. The matrix generation unit 312 then generates the product Wδ of the transformation matrix W before the position change and the transformation matrix δ that represents the position change as an initial matrix W0, and updates the initial matrix W0 using matrix update processing Sc2 to generate the transformation matrix W.

In the above configuration, the transformation matrix W after the position change is generated using the transformation matrix W calculated before the position change and the transformation matrix δ representing the position change. Therefore, it is possible to generate a transformation matrix W that can identify the position C[h,f] of each finger with high accuracy while reducing the load of the matrix generation process. Note that while the above explanation assumes the first to fifth embodiments, in the sixth embodiment as well, the transformation matrix W may be updated at an appropriate time during the execution of the first image process.

(4) In each of the above embodiments, a keyboard instrument 200 having a keyboard 22 is exemplified, but the present disclosure can be applied to any type of instrument. For example, the above embodiments can be applied to any instrument that can be manually operated by a user, such as a string instrument, a wind instrument, or a percussion instrument. A typical example of an instrument is a type of instrument that a user plays with the fingers of one or both hands.

(5) The performance analysis system 100 may be realized by a server device that communicates with an information device such as a smartphone or tablet terminal. For example, performance data P generated by a keyboard instrument 200 connected to the information device and image data D1 generated by a photographing device 15 mounted on or connected to the information device are transmitted from the information device to the performance analysis system 100. The performance analysis system 100 generates fingering data Q by executing a performance analysis process on the performance data P and image data D1 received from the information device, and transmits the fingering data Q to the information device. Similarly, the image processing system 700 exemplified in the sixth or seventh embodiment may be realized by a server device that communicates with the information device.

(6) The functions of the performance analysis system 100 according to the first to fifth embodiments, or the image processing system 700 according to the sixth to seventh embodiments, are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12, as described above. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the storage device 12 that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

9: Supplementary Note From the above-described exemplary embodiments, the following configurations, for example, can be understood.

An image processing method according to one aspect (aspect 1) of the present disclosure estimates a specific region that includes an image of a musical instrument from a performance image that includes an image of the musical instrument and an image of the multiple fingers of a user playing the musical instrument, and extracts the specific region from the performance image. In the above aspect, a specific region that includes the image of the musical instrument is extracted from a performance image that includes an image of the musical instrument and an image of the multiple fingers of the user. Therefore, it is possible to improve the convenience of the performance image.

In a specific example of aspect 1 (aspect 2), the specific area is an area that includes an image of the musical instrument and an image of at least a portion of the user's body. In the above aspect, a specific area that includes an image of the musical instrument and an image of the user's body is extracted. Therefore, an image can be generated that allows the appearance of the musical instrument and the appearance of the user's body to be efficiently viewed.

In a specific example (aspect 3) of aspect 2, in estimating the specific region, image data representing the performance image is input into a machine-learned estimation model to generate an image processing mask representing the specific region, and in extracting the specific region, the image processing mask is applied to the performance image to extract the specific region. In the above aspect, an image processing mask representing the specific region is generated by inputting image data of the performance image into a machine-learned estimation model. Therefore, it is possible to identify specific regions with high accuracy for unknown and diverse performance images.

In a specific example (aspect 4) of aspect 3, the estimation model includes a first model and a second model, and the estimation of the specific area includes a first estimation process for estimating a first area including the image of the musical instrument in the performance image by inputting image data representing the performance image into the first model, a second estimation process for estimating a second area including the image of the multiple fingers in the performance image by inputting image data representing the performance image into the second model, and an area synthesis process for generating the image processing mask representing the specific area including the first area and the second area. In the above aspect, the first area including the image of the musical instrument in the performance image is estimated by the first model, and the second area including the image of the user in the performance image is estimated by the second model. Therefore, compared to a configuration using a single model that extracts both the image of the musical instrument and the image of the user at once, the specific area including the image of the musical instrument and the image of the user can be extracted with high accuracy. In addition, since each of the first model and the second model is established by individual machine learning, the processing load related to the machine learning of the first model and the second model is reduced.

In a specific example of aspect 4 (aspect 5), it is possible to switch between a first mode in which both the first estimation process and the second estimation process are executed, and a second mode in which one of the first estimation process and the second estimation process is executed. In the above aspect, in the first mode, a specific area including an image of the musical instrument and an image of the user is extracted from the performance image. On the other hand, in the second mode, a specific area including one of the image of the musical instrument and the image of the user is extracted from the performance image. As described above, it is possible to easily switch the object to be extracted from the performance image.

An image processing system according to one aspect (aspect 6) of the present disclosure includes an area estimation unit that estimates a specific area including an image of a musical instrument from a performance image including an image of a musical instrument and an image of the fingers of a user playing the musical instrument, and an area extraction unit that extracts the specific area from the performance image.

A program according to one aspect (aspect 7) of the present disclosure causes a computer system to function as an area estimation unit that estimates a specific area that includes an image of a musical instrument from a performance image that includes an image of the musical instrument and an image of the fingers of a user playing the musical instrument, and an area extraction unit that extracts the specific area from the performance image.

100...performance analysis system, 11...control device, 12...storage device, 13...operation device, 14...display device, 15...imaging device, 200...keyboard instrument, 21...key, 22...keyboard, 30...performance analysis unit, 31...finger position data generation unit, 311...image extraction unit, 312...matrix generation unit, 313...finger position estimation unit, 314...projection transformation unit, 32...fingering data generation unit, 321...probability calculation unit, 322...fingering estimation unit, 323...control data generation unit, 40...display control unit, 51...estimation model, 51a...tentative model, 52[k]...estimation model, 700...image processing system.

Claims (5)

An image processing method implemented by a computer system for extracting a specific area from a performance image including a keyboard image of a keyboard instrument and an image of a plurality of fingers of a user playing the keyboard instrument , the method comprising:
In the extraction of the specific region,
a first mode in which the specific area including both the keyboard image and the fingering image is extracted from the performance image;
a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image;
It is switchable,
Image processing methods. In the extraction of the specific region,
inputting image data representing the performance image into a machine-learned estimation model to generate an image processing mask representing the specific region;
The image processing method according to claim 1 , further comprising the step of extracting the specific region by applying the image processing mask to the performance image. the estimation model includes a first model and a second model;
In the first mode,
a first estimation process for estimating a first region of the performance image including the keyboard image by inputting image data representing the performance image into the first model;
a second estimation process for estimating a second region including the hand image in the performance image by inputting image data representing the performance image into the second model ;
generating a first image processing mask representing the specific region including the first region and the second region ;
extracting the specific region by applying the first image processing mask to the performance image;
In the second mode,
By executing the first estimation process, a second image processing mask is generated that represents the specific area including the keyboard image and not including the hand image;
The specific region is extracted by applying the second image processing mask to the performance image.
The image processing method according to claim 2 . an image extraction unit that extracts a specific area from a performance image including a keyboard image of a keyboard instrument and an image of a plurality of fingers of a user playing the keyboard instrument ;
The image extraction unit
a first mode in which the specific area including both the keyboard image and the fingering image is extracted from the performance image;
a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image;
It is switchable,
Image processing system. A program that causes a computer system to function as an image extracting unit that extracts a specific area from a performance image including a keyboard image of a keyboard instrument and an image of a plurality of fingers of a user playing the keyboard instrument , the program comprising:
The image extraction unit
a first mode in which the specific area including both the keyboard image and the fingering image is extracted from the performance image;
a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image;
It is switchable,
program.

Images