Efficient methods for joint estimation of multiple fundamental frequencies in music signals
 3.4k Downloads
 4 Citations
Abstract
This study presents efficient techniques for multiple fundamental frequency estimation in music signals. The proposed methodology can infer harmonic patterns from a mixture considering interactions with other sources and evaluate them in a joint estimation scheme. For this purpose, a set of fundamental frequency candidates are first selected at each frame, and several hypothetical combinations of them are generated. Combinations are independently evaluated, and the most likely is selected taking into account the intensity and spectral smoothness of its inferred patterns. The method is extended considering adjacent frames in order to smooth the detection in time, and a pitch tracking stage is finally performed to increase the temporal coherence. The proposed algorithms were evaluated in MIREX contests yielding state of the art results with a very low computational burden.
Keywords
Temporal Coherence Candidate Selection Harmonic Amplitude Adjacent Frame Mirex1 Introduction
The goal of a multiple fundamental frequency (f_{0}) estimation method is to infer the number of simultaneous harmonic sounds present in an acoustic signal and their fundamental frequencies. This problem is relevant in speech processing, structural audio coding, and several music information retrieval (MIR) applications, like automatic music transcription, compression, instrument separation and chord estimation, among others.
In this study, a multiple f_{0} estimation method is presented for the analysis of pitched musical signals. The core methodology introduced in [1] is described and extended considering information about neighbor frames.
Most multiple f_{0} estimation methods are complex systems. The decomposition of a signal into multiple simultaneous sounds is a challenging task due to harmonic overlaps and inharmonicity (when partial frequencies are not exact multiples of the f_{0}). Many different techniques are proposed in the literature to face this task. Recent reviews of multiple f_{0} estimation in music signals can be found in [2, 3, 4].
Some techniques rely on the midlevel representation, trying to emphasize the underlying fundamental frequencies by applying signal processing transformations to the input signal [5, 6, 7]. Supervised [8, 9] and unsupervised [10, 11] learning techniques have also been investigated for this task. The matching pursuit algorithm, which approximates a solution for decomposing a signal into linear functions (atoms), is also adopted in some approaches [12, 13]. Methods based on statistical inference within parametric signal models [3, 14, 15] have also been studied for this task.
Heuristic approaches can also be found in the literature. Iterative cancellation methods estimate the prominent f_{0} subtracting it from the mixture and repeating the process until a termination criterion [16, 17, 18]. Joint estimation methods [19, 20, 21] can evaluate a set of possible f_{0} hypotheses, consisting of f_{0} combinations, selecting the most likely at each frame without corrupting the residual as it occurs with iterative cancellation.
Some existing methods can be switched to another framework. For example, iterative methods can be viewed against matching pursuit background, and many unsupervised learning methods like [11] can be switched to a statistical framework.
Statistical inference provides an elegant framework to deal with this problem, but these methods are usually intended for single instrument f_{0} estimation (typically piano), as exact inference often becomes computationally intractable for complex and very different sources.
Similarly, supervised learning methods can infer models of pitch combinations seen in the training stage, but they are currently constrained to monotimbral sounds with almost constant spectral profiles [4].
In music, consonant chords include harmonic components of different sounds which coincide in some of their partial frequencies (harmonic overlaps). This situation is very frequent and introduces ambiguity in the analysis, being the main challenge in multiple f_{0} estimation. When two harmonics are overlapped, two sinusoids of the same frequency are summed in the waveform, resulting a signal with the same frequency and which magnitude depends on their phase difference.
The contribution of each harmonic to the mixture can not be properly estimated without considering the interactions with the other sources. Joint estimation methods provide an adequate framework to deal with this problem, as they do not assume that sources are mutually independent and individual pitch models can be inferred taking into account their interactions. However, they tend to have high computational costs due to the number of possible combinations to be evaluated.
Novel efficient joint estimation techniques are presented in this study. In contrast to previous joint approaches, the proposed algorithms have a very low computational cost. They were evaluated and compared to other studies in MIREX [22, 23] multiple f_{0} estimation and tracking contests, yielding competitive results with very efficient runtimes.
The core process, introduced in [1], relies on the inference and evaluation of spectral patterns from the mixture. For a proper inference, source interactions must be considered in order to estimate the amplitudes of their overlapped harmonics. This is accomplished by evaluating independent combinations consisting of hypothetical patterns (f_{0} candidates). The evaluation criterion enhances those patterns having high intensity and smoothness. This way, the method takes advantage of the spectral properties of most harmonic sounds, in which first harmonics are usually those with higher energy and their spectral profile tend to be smooth.
Evaluating many possible combinations can computationally intractable. In this study, the efficiency is boosted by reducing the spectral information to be considered for the analysis, adding a f_{0} candidate selection process, and pruning unlikely combinations by applying some constraints, like a minimum intensity for a pattern.
One of the main contributions of this study is the extension of the core algorithm to increase the temporal coherence. Instead considering isolated frames, the combinations sharing the same pitches across neighbor frames are grouped to smooth the detection in time. A novel pitch tracking stage is finally presented to favor smooth transitions of pitch intensities.
The proposed algorithms are publicly available at http://grfia.dlsi.ua.es/cm/projects/drims/software.php.
2 Methodology
Joint estimation methods generate and evaluate competing sets of f_{0} combinations in order to select the most plausible combination directly. This scheme, recently introduced in [24, 25] has the advantage that the amplitudes of overlapping partials can be approximated taking into account the partials of the other candidates for a given combination. Therefore, partial amplitudes can depend on the particular combination to be evaluated, opposite to an iterative estimation scheme like matching pursuit, where a wrong estimate may produce cumulative errors.
The core method performs a frame by frame analysis, selecting the most likely combination of fundamental frequencies at each instant. For this purpose, a set of f_{0} candidates are first identified from the spectral peaks. Then, a set of possible combinations, $\mathcal{C}\left(t\right)$, of candidates are generated, and a joint algorithm is used to find the most likely combination.
In order to evaluate a combination, hypothetical partial sequences HPS (term proposed in [26] to refer to a vector containing hypothetical partial amplitudes) are inferred for its candidates. In order to build these patterns, harmonic interactions with the partials of the other candidates in the combination are considered. The overlapped partials are first identified, and their amplitudes are estimated by linear interpolation using the nonoverlapped harmonic amplitudes.
Once patterns are inferred, they are evaluated taking into account the sum of its hypothetical harmonic amplitudes and a novel smoothness measure.
Combinations are analysed considering their individual candidate scores, and the most likely combination is selected at the target frame.
The method assumes that the spectral envelopes of the analysed sounds tend to vary smoothly as a function of frequency. The spectral smoothness principle has successfully been used in different ways in the literature [7, 26, 27, 28, 29]. A novel smoothness measure based on the convolution of the hypothetical harmonic pattern with a Gaussian window is proposed.
The processing stages, shown in Figure 1, are described below.
2.1 Preprocessing
The analysis is performed in the frequency domain, computing the magnitude spectrogram using a 93 ms Hanning windowed frame with a 9.28 ms hop size. This is the frame size typically chosen for multiple f_{0} estimation of music signals in order to achieve a suitable frequency resolution, and it experimentally showed to be adequate. The selected frame overlap ratio may seem high from a practical point of view, but it was required to compare the method with other studies in MIREX (see 4.3).
To get a more precise estimation of the lower frequencies, zero padding is used multiplying the original window size by a factor z to complete it with zeroes before computing the FFT.
In order to increase the efficiency, many unnecessary spectral bins are discarded for the subsequent analysis using a simple peak picking algorithm to extract the hypothetical partials. At each frame, only those spectral peaks with an amplitude higher than a threshold μ are selected, removing the rest of spectral information and obtaining this way a sparse representation containing a subset of spectral bins. It is important to note that this thresholding does not have a significant effect on the results, as values of μ are quite low, but the efficiency of the method importantly increases.
2.2 Candidate selection
The evaluation of all possible f_{0} combinations in a mixture is computationally intractable, therefore a reduced subset of candidates must be chosen before generating their combinations. For this, candidates are first selected from the spectral peaks within the range [f_{min}, f_{max}] corresponding to the musical pitches of interest. Harmonic sounds with missing fundamentals are not considered, although they seldom appear in practical situations. A minimum spectral peak amplitude ε for the first partial (f_{0}) can also be assumed in this stage.
The spectral magnitudes at the candidate partial positions are considered as a criterion for candidate selection as described next.
2.2.1 Partial search
Slight harmonic deviations from ideal partial frequencies are common in music sounds, therefore inharmonicity must be considered for partial search. For this, a constant margin around each harmonic frequency f_{ h }± f_{ r }is set. If there are no spectral peaks within this margin, the harmonic is considered to be missing. Besides considering a constant margin, frequency dependent margins were also tested assuming that partial deviations in high frequencies are larger than those in low frequencies. However, results decreased, mainly because many false positive harmonics (most of them corresponding to noise) can be found in high frequencies.
Different strategies were also tested for partial search, and finally, like in [30], the harmonic spectral location and spectral interval principles [31] were chosen in order to take inharmonicity into account. The ideal frequency f_{ h }of the first harmonic is initialized to f_{ h }= 2f_{0}. The next ones are searched at f_{h+1}= (f_{ x }+ f_{0}) ± f_{ r }, where f_{ x }= f_{ i }if the previous harmonic h was found at the frequency f_{ i }, or f_{ x }= f_{ h }if the previous partial was missing.
2.2.2 Selection of F candidates
Once the hypothetical partials for all possible candidates are searched, candidates are ordered decreasingly by the sum of their amplitudes and, at most, only the first F candidates of this ordered list are chosen for the following processing stages.
Harmonic summation is a simple criterion for candidate selection, and other alternatives can be found in the literature, including harmonicity criterion [30], partial beating [30], or the product of harmonic amplitudes in the power spectrum [20]. Evaluating alternative criteria for candidate selection is left as future study.
2.3 Generation of candidate combinations
All the possible combinations of the F selected candidates are calculated and evaluated, and the combination with highest score is yielded at the target frame. The combinations consist of different number of fundamental frequencies. In contrast to studies like [26], there is not need for a priori estimation of the number of concurrent sounds before detecting the fundamental frequencies, and the polyphony is implicitly calculated in the f_{0} estimation stage, choosing the combination with highest score independently from the number of candidates.
Therefore, N combinations are evaluated at each frame, so the adequate selection of F and P is critical for the computational efficiency of the algorithm. An experimental discussion on this issue is presented in Sec. 4.2.
2.4 Evaluation of combinations
In order to evaluate a combination ${\mathcal{C}}_{i}\in \mathcal{C}\left(t\right)$, a hypothetical pattern is first estimated for each of its candidates. Then, these patterns are evaluated in terms of their intensity and smoothness, assuming that music sounds have a perceivable intensity and their spectral shapes are smooth, like it occurs for most harmonic instruments. The combination $\widehat{\mathcal{C}}\left(t\right)$ which patterns maximize these measures is yielded at the target frame t.
2.4.1 Inference of hypothetical patterns
Assuming that interactions between noncoincident partials (beating) do not alter significantly the original spectral amplitudes, the nonoverlapped amplitudes are directly assigned to the HPS. However, the contribution of each source to an overlapped partial amplitude must be estimated.
Getting an accurate estimate of the amplitudes of colliding partials is not reliable only with the spectral magnitude information. In this study, the additivity of linear spectrum is assumed as in most approaches in the literature. Assuming additivity and spectral smoothness, the amplitudes of overlapped partials can be estimated similarly to [26, 32] by linear interpolation of the neighboring nonoverlapped partials, as shown in Figure 3 (bottom).
If there are two or more consecutive overlapped partials, then the interpolation is done the same way using the available nonoverlapped values. For instance, if harmonics 2 and 3 of a pattern are overlapped, then the amplitudes of harmonics 1 and 4 are used to estimate them by linear interpolation.
After the interpolation, the estimated contribution of each partial to the mixture is subtracted before processing the next candidates. This calculation (see Figure 3) is done as follows:

If the interpolated (expected) value is greater than the corresponding overlapped harmonic amplitude, then p_{c,h}is set as the original harmonic amplitude, and the spectral peak is completely removed from the residual, setting it to zero for the candidates that share that partial.

If the interpolated value is smaller than the corresponding overlapped harmonic amplitude, then p_{c,h}is set as the interpolated amplitude, and this value is linearly subtracted for the candidates that share the harmonic.
The residual harmonic amplitudes after this process are iteratively analysed for the rest of the candidates in the combination in ascending frequency order.
2.4.2 Candidate evaluation
Assuming that a pattern should have a minimum loudness, those combinations having any candidate with a very low absolute (l(c) <η) or relative $\left(l\left(c\right)<\gamma {L}_{\mathcal{C}},\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{being}}\phantom{\rule{2.77695pt}{0ex}}{L}_{\mathcal{C}}={\text{max}}_{{\forall}_{c}}\left\{l\left(c\right)\right\}\right)$ intensity are discarded.
The underlying hypothesis assumes that a smooth spectral pattern is more probable than an irregular one. This is assessed through a novel smoothness measure s(c) which is based on Gaussian smoothing.
To compute it, the HPS of a candidate is first normalized dividing the amplitudes by its maximum value, obtaining $\stackrel{\u0304}{\mathbf{p}}$. The aim is to compare $\stackrel{\u0304}{\mathbf{p}}$ with a smooth model $\stackrel{\u0303}{\mathbf{p}}$ built from it, in such a way that the similarity between $\stackrel{\u0304}{\mathbf{p}}$ and $\stackrel{\u0303}{\mathbf{p}}$ will give an estimation of the smoothness.
Only three components were chosen for the Gaussian window of unity variance, ${\mathcal{N}}_{0,1}={\left(0.21,0.58,0.21\right)}^{\mathsf{\text{T}}}$, due to the small size of p_{ c }, which is limited by H. Typical values for H are within the range H ∈ [5, 20], as only the first harmonics contain most of the energy of a harmonic source.
where H_{ c }is the index of the last harmonic found for the candidate. This factor was introduced to prevent that high frequency candidates that have less partials than those at low frequencies will have higher smoothness. This way, the smoothness is considered to be more reliable when there are more partials to estimate it.
where κ is a factor that permits to balance the smoothness contribution experimentally.
2.4.3 Combination selection
3 Extension using neighbor frames
In the previously described method, each frame was independently analysed, yielding the combination of fundamental frequencies that maximizes a given measure. One of the main limitations of this approach is that the window size (93 ms) is relatively short to perceive the pitches in a complex mixture, even for an expert musician. Context is very important in music to disambiguate certain situations. In this section the core method is extended, considering information about adjacent frames to produce a smoothed detection across time.
3.1 Temporal smoothing
A simple and effective novel technique is presented in order to smooth the detection across time. Instead of selecting the most likely combination at isolated frames, adjacent frames are also analysed to get the score of each combination.
The method aims to enforce the pitch continuity in time. For this, the fundamental frequencies of each combination $\mathcal{C}$ are mapped into music pitches, obtaining a pitch combination ${\mathcal{C}}^{\prime}$. For instance, the combination ${\mathcal{C}}_{i}=\left\{261\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{Hz}},\phantom{\rule{2.77695pt}{0ex}}416\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{Hz}}\right\}$ is mapped into ${\mathcal{C}}_{i}^{\prime}=\left\{{\mathsf{\text{C}}}_{4},\mathsf{\text{G}}{\u266f}_{\mathsf{\text{4}}}\right\}$.
If there is more than one combination with the same pitches (for instance, ${\mathcal{C}}_{1}=\left\{260\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{Hz}}\right\}$ and ${\mathcal{C}}_{2}=\left\{263\phantom{\rule{2.77695pt}{0ex}}\mathsf{\text{Hz}}\right\}$ are both ${\mathcal{C}}^{\prime}=\left\{{\mathsf{\text{C}}}_{4}\right\}$), it is removed, and the unique combination with the highest score value is only kept.
If ${\widehat{\mathcal{C}}}^{\prime}\left(t\right)$ does not contain any combination because there are no valid candidates in the frame t, then a rest is yielded without evaluating the adjacent frames.
3.2 Pitch tracking
A basic pitch tracking method is introduced in order to favor smooth transitions of $\stackrel{\u0303}{l}\left({c}^{\prime}\left(t\right)\right)$. The proposed technique aims to increase the temporal coherence using a layered weighted directed acyclic graph (wDAG).
Using this scheme, the transition weight between two combinations considers the score of the target combination and the differences between the candidate intensities.
Once the graph is generated, the shortest path that minimizes the sum of weights from the starting node to the final state across the wDAG is found using the Dijkstra [33] algorithm. The vertices that belong to the shortest path are the pitch combinations yielded at each time frame.
Building the wDAG for all possible combinations at all frames could be computationally intractable, but considering only the M most likely combinations at each frame keeps almost the same runtime than without performing tracking for small values of M.
4 Evaluation
Initial experiments were done using a data set of random mixtures to perform a first evaluation and set up the parameters. Then, the proposed approaches were publicly evaluated and compared by a third party to other studies in the MIREX [22, 23] multiple f_{0} estimation and tracking contest.
4.1 Evaluation metrics
Different metrics for multiple f_{0} estimation can be found in the literature. The evaluation can be done both at frame by frame and note levels. The first mode evaluates the correct estimation in a frame by frame basis, whereas note tracking also considers the temporal coherence of the detection, adding more restrictions for a note to be considered correct. For instance, in the MIREX note tracking contest, a note is correct if its f_{0} is closer than half a semitone to the groundtruth pitch and its onset is within a ± 50 ms range of the ground truth note onset.
A false positive (FP) is a detected pitch (or note, if evaluation is performed at note level) which is not present in the signal, and a false negative (FN) is a missing pitch. Correctly detected pitches (OK) are those estimates that are also present in the groundtruth at the detection time.
An alternative metric based on the speaker diarization error score from NIST^{a} was proposed by Poliner and Ellis [34] to evaluate multiple f_{0} estimation methods. The NIST metric consists of a single error score which takes into account substitution errors (mislabeling an active voice, E_{ subs }), miss errors (when a voice is truly active but results in no transcript, E_{ miss }), and false alarm errors (when an active voice is reported without any underlying source, E_{ fa }).
This metric avoids counting errors twice as classical metrics do in some situations. For instance, using accuracy, if there is a C_{3} pitch in the reference groundtruth but the system reports a C_{4}, two errors (a false positive and a false negative) are counted. However, if no pitch was detected, only one error would be reported.
Poliner and Ellis [34] state that, as in the universal practice in the speech recognition community, this is probably the most adequate measure, since it gives a direct feel for the quantity of errors that will occur as a proportion of the total quantity of notes present.
4.2 Parameterization
A data set of random pitch combinations, also used in the evaluation of Klapuri [35] method, was used to tune up the algorithm parameters. The data set consists on 4000 mixtures with polyphony^{b} 1, 2, 4, and 6. The 2842 audio samples from 32 music instruments used to generate the mixtures are from the McGill University master samples collection^{c}, the University of Iowa^{d}, IRCAM studio online^{e}, and recordings of an acoustic guitar. In order to respect the copyright restrictions, only the first 185 ms of each mixture^{f} were used for evaluation. In this dataset, the range of valid pitches is [f_{min} = 38 Hz, f_{max} = 2100 Hz], and the maximum polyphony is P = 6.
In the parameterization stage, the selected parameter values were not those that achieved the highest accuracy in the test set, but those that obtained a good tradeoff between accuracy and low computational cost.
Parameter values experimentally selected
Stage  Parameter  Symbol  Value 

Preprocessing  Partial selection threshold  μ  0.1 
Zero padding factor  z  4  
Candidate selection  Min. f_{0} amplitude  ε  2 
Combination generation  Max. number of candidates  F  10 
Partial search bandwidth  f _{ r }  11 Hz  
HPS size  H  10  
Combination evaluation  Absolute intensity threshold  γ  5 
Relative intensity threshold  η  0.1  
Smoothness weight  κ  2  
Temporal smoothing  Number of adjacent frames  K  2 
4.3 Evaluation and comparison with other methods
The core method was externally evaluated and compared with other approaches in MIREX 2007 [22] multiple f_{0} estimation and tracking contest, whereas the extended method was submitted to MIREX 2008 [23]. The data set used in both MIREX editions were essentially the same, therefore the results can be directly compared. The details of the evaluation and the groundtruth labeling are described in [36]. Accuracy, precision, recall and E_{ tot }were reported for frame by frame estimation, whereas precision, recall and Fmeasure were used for the note tracking task.
The core method (PI107) was evaluated using the parameters specified in Table 1. For this contest, a final postprocessing stage was added. Once the fundamental frequencies were estimated, they were converted into music pitches, and pitch series shorter than d = 56 ms were removed to avoid some local discontinuities.
The extended method was submitted with pitch tracking (PI108) and without it (PI208) for comparison. In the nontracking case, a similar procedure than in the core method was adopted, removing notes shorter than a minimum duration and merging note with short rests between them. Using pitch tracking, the methodology described in Sec. 3.2 was performed instead, increasing the temporal coherence of the estimate with the wDAG using M = 5 combinations at each layer.
MIREX 0708 methods submitted for frame by frame (FBF) and note tracking (NT) evaluation
Id  Referenc  es FBF  NT  Methodology 

AC07  [43]  √  √  Unsupervised learning 
CL07  [18]  √  Iterative cancelation  
CL08  [44]  √  Iterative cancelation  
YRC07  [30]  √  Joint estimation  
DRD08  [45]  √  Iterative cancelation  
EBD07  [20]  √  √  Statistical inference 
EBD08  [46]  √  √  Statistical inference 
EOS08  [47]  √  √  Statistical inference 
EOS07  [48]  √  √  Statistical inference 
MG08  [49]  √  Database matching  
PE07  [34]  √  √  Supervised learning 
PI107  [1]  √  √  Core method 
PI207  [50]  √  Iterative cancellation  
PI108  √  √  Extended method + tracking  
PI208  √  √  Extended method  
PL07  [51]  √  Matching pursuit  
RFF08  [52]  √  √  Supervised learning 
RK07  [17]  √  √  Iterative cancellation + f_{0} tracking 
RK08  [17]  √  √  Iterative cancellation + f_{0} tracking 
SR07  [53]  √  Unsupervised learning  
VBB07  [54]  √  √  Unsupervised learning 
VBB08  [54]  √  √  Unsupervised learning 
YRC108  [37]  √  Joint estimation  
YRC208  [37]  √  √  Joint estimation + f_{0} tracking 
ZR07  [7]  √  Signal processing  
ZR08  [55]  √  Signal processing 
MIREX 0708 frame by frame and note tracking runtimes in seconds of the top ten accuracy methods
Id  FBF (s)  NT (s)  Machine 

ZR07  271  BLACK  
PI107  364  364  ALE Nodes 
PI208  792  790  ALE Nodes 
PI108  955  950  ALE Nodes 
VBB207  2233  ALE Nodes  
RK07  3540  3285  SANDBOX 
RK08  5058  5044  ALE Nodes 
YRC108  57483  ALE Nodes  
YRC208  57483  57483  ALE Nodes 
YRC07  132300  ALE Nodes 
While the proposed approaches achieved the lowest E_{ tot }score, there were very few false alarms compared to miss errors. On the other hand, the methods from Ryynänen and Klapuri [17] and Yeh et al. [37] had a better balanced precision, recall, as well as a good balance in the three error types, and as a result, high accuracies.
Quoting Bay et al. [36], "Inspecting the methods used and their performances, we can not make generalized claims as to what type of approach works best. In fact, statistical significance testing showed that the top three methods (YRC, PI, and RK) were not significantly different."
5 Conclusions and discussion
In this study, an efficient methodology is proposed for multiple f_{0} estimation of real music signals assuming spectral smoothness and strong harmonic content without any other a priori knowledge of the sources.
The method can infer and evaluate hypothetical spectral patterns from the analysis of different hypotheses taking into account the interactions with other sources.
The algorithm is extended considering adjacent frames to smooth the temporal detection. In order to increase the temporal coherence of the detection, a novel pitch tracking stage based on a wDAG has been included. The proposed algorithms were evaluated and compared to other works by a third party in a public contest (MIREX), obtaining a high accuracy, the highest precision and the lowest E_{ tot }among all the multiple f_{0} methods submitted. Although many possible combinations of candidates are evaluated at each frame, the presented approach has a very low computational cost, showing that it is possible to make an efficient joint estimation method by applying some constraints, like the sparse representation of only certain spectral peaks, the candidate filtering stage, and the combination pruning process.
The pitch tracking stage could be replaced by a more reliable method in a future study. For instance, the transition weights could be learned from a labeled test set, or a more complex tracking method like the highorder HMM scheme from Chang et al. [38] could be used instead. Besides intensity, the centroid of an HPS should also have a temporal coherence when belonging to the same source, therefore this feature could also be considered for tracking. Using stochastic models, a probability could be assigned to each pitch in order to remove those that are less probable given their context. Musical probabilities can be taken into account, like in [17], to remove very unlikely notes. The adaptation to polyphonic music of the stochastic approach from PerezSancho [39] is also planned as future study, in order to complement the multiple f_{0} estimation method to obtain a musically coherent detection. Besides frame by frame analysis and the analysis of adjacent frames, the possibility of the extended method for combining similar information across frames allows to consider different alternative architectures.
This novel methodology permits interesting schemes. For example, the beginnings of musical events can be estimated using an onset detection algorithm like [40]. Then, combinations of those frames that are between two consecutive onsets can be merged to yield the pitches within the interonset interval. This technique is close to segmentation, and it can obtain reliable results when the onsets are correctly estimated, as it happens with sharp attack sounds like piano, but a wrong estimate in the onset detection stage will affect the results.
Beats, that can be defined as a sense of equally spaced temporal units [41], can also be detected to merge combinations with a quantization grid. Once the beats are estimated (for example with a beat tracking algorithm like BeatRoot [42]), a grid split with a given beat divisor 1/q can be used, assuming that the minimum note duration is q. For instance, if q = 4, each interbeat interval can be split in q sections. Then, the combinations of the frames that belong to the quantization unit can be merged to obtain the results at each minimum grid unit. Like in the onset detection scheme, the success rate of this approach depends on the success of the beat estimation. The extended method can be applied using any of these schemes. The adequate choice of the architecture depends on the signal to be analysed. For instance, for timbres with sharp attacks, it is recommended to use onset information, which is very reliable for these kind of sounds. These alternative architectures have been perceptually evaluated using some example real songs, but a more rigorous evaluation of these schemes is left for future study, since an aligned dataset of real musical pieces with symbolic data is required for this task.
Endnotes
^{a}National Institute of Standards and Technology.
^{b}There are 1000 mixtures for each polyphony.
^{c} http://www.music.mcgill.ca/resources/mums/html/index.htm
^{d} http://theremin.music.uiowa.edu/MIS.html
^{e} http://forumnet.ircam.fr/402.html?&L=1
^{f}The authors would like to thank A. Klapuri for providing this data set for evaluation.
Notes
Acknowledgements
This study was supported by the project DRIMS (code TIN200914247C02), the Consolider Ingenio 2010 research programme (project MIPRCV, CSD200700018), and the PASCAL2 Network of Excellence, IST2007216886.
Supplementary material
References
 1.Pertusa A, Iñesta JM: Multiple fundamental frequency estimation using Gaussian smoothness. In Proc of the IEEE Int Conf on Acoustics, Speech, and Signal Processing (ICASSP). Volume I. Las Vegas, NV; 2008:105108.Google Scholar
 2.Klapuri A, Davy M: Signal Processing Methods for Music Transcription. Springer Science+Business Media LCC, New York; 2006.CrossRefGoogle Scholar
 3.Christensen MG, Jakobsson A: MultiPitch estimation. In Synthesis Lectures on Speech and Audio Processing. Morgan & Claypool publishers, Seattle, WA, USA; 2009.Google Scholar
 4.Pertusa A: Computationally efficient methods for polyphonic music transcription. PhD thesis. Universidad de Alicante; 2010.Google Scholar
 5.Tolonen A, Karjalainen M: A computationally efficient multipitch analysis model. IEEE Trans Speech Audio Process 2000, 8(6):708716. 10.1109/89.876309CrossRefGoogle Scholar
 6.Peeters G: Music pitch representation by periodicity measures based on combined temporal and spectral representations. In Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Volume V. Toulouse, France; 2006:5356.Google Scholar
 7.Zhou R, Reiss JD, Mattavelli M, Zoia G: A computationally efficient method for polyphonic pitch estimation. EURASIP J Adv Signal Process 2009., 2009(28):Google Scholar
 8.Marolt M: Networks of adaptive oscillators for partial tracking and transcription of music recordings. J New Music Res 2004, 33: 4959. 10.1076/jnmr.33.1.49.35391CrossRefGoogle Scholar
 9.Pertusa A, Iñesta JM: Polyphonic monotimbral music transcription using dynamic networks. Pattern Recogn Lett 2005, 26(12):18091818. 10.1016/j.patrec.2005.03.001CrossRefGoogle Scholar
 10.Cont A: Realtime multiple pitch observation using sparse nonnegative constraints. In Proc of the 7th International Symposium on Music Information Retrieval (ISMIR). Victoria, Canada; 2006:206211.Google Scholar
 11.Bertin N, Badeau R, Vincent E: Enforcing harmonicity and smoothness in bayesian nonnegative matrix factorization applied to polyphonic music transcription. IEEE Trans Audio Speech Language Process 2010, 18(3):538549.CrossRefGoogle Scholar
 12.CarabiasOrti JJ, VeraCandeas P, CañadasQuesada FJ, RuizReyes N: Music sceneadaptive harmonic dictionary for unsupervised noteevent detection. IEEE Trans Audio Speech Language Process 2010, 18(3):473486.CrossRefGoogle Scholar
 13.Leveau P, Vincent E, Richard G, Daudet L: Instrumentspecific harmonic atoms for midlevel music representation. IEEE Trans Audio Speech Language Process 2008, 16: 116128.CrossRefGoogle Scholar
 14.Cemgil AT: Bayesian Music Transcription. PhD thesis.Radboud University of Nijmegen, Netherlands; 2004. [http://wwwsigproc.eng.cam.ac.uk/~atc27/papers/cemgilthesis.pdf]Google Scholar
 15.Davy M: An introduction to signal processing. In Signal Processing methods for music transcription. Springer Science+Business Media LCC, New York; 2006.Google Scholar
 16.Klapuri A: Multipitch analysis of polyphonic music and speech signals using an auditory model. IEEE Trans Audio, Speech and Language Process 2008, 16(2):255266.CrossRefGoogle Scholar
 17.Ryynänen M, Klapuri A: Polyphonic Music Transcription Using Note Event Modeling. In Proc IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). New Paltz, New York, USA; 2005:319322.Google Scholar
 18.Cao C, Li M, Liu J, Yan Y: Multiple F0 estimation in polyphonic music. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 19.Yeh C, Roebel A, Rodet X: Multiple fundamental frequency estimation and polyphony inference of polyphonic music signals. IEEE Trans Audio, Speech Language Process 2010, 18(6):11161126.CrossRefGoogle Scholar
 20.Emiya V, Badeau R, David B: Automatic transcription of piano music based on HMM tracking of jointlyestimated pitches. In Proc European Signal Processing Conference (EUSIPCO). Rhodes, Greece; 2008.Google Scholar
 21.CañadasQuesada FJ, VeraCandeas P, RuizReyes N, CarabiasOrti JJ: Polyphonic transcription based on temporal evolution of spectral similarity of Gaussian Mixture Models. In 17th European Signal Processing Conference (EUSIPCO). Glasgow, Scotland; 2009:1014.Google Scholar
 22.MIREX, Music Information Retrieval Evaluation eXchange. Multiple fundamental frequency estimation and tracking contest2007. [http://www.musicir.org/mirex/wiki/2007:MultipleFundamentalFrequencyEstimation&TrackingResults]
 23.MIREX, Music Information Retrieval Evaluation eXchange. Multiple fundamental frequency estimation and tracking contest2008. [http://www.musicir.org/mirex/wiki/2008:MultipleFundamentalFrequencyEstimation&TrackingResults]
 24.Yeh C: Multiple F0 estimation for MIREX 2007. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 25.Pertusa A, Iñesta JM: Multiple fundamental frequency estimation based on spectral pattern loudness and smoothness. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 26.Yeh C, Röbel A, Rodet X: Multiple fundamental frequency estimation of polyphonic music signals. In IEEE, Int Conf on Acoustics, Speech and Signal Processing (ICASSP). Volume III. Philadelphia, PA; 2005:225228.Google Scholar
 27.CañadasQuesada FJ, VeraCandeas P, RuizReyes N, MataCampos R, CarabiasOrti JJ: Noteevent detection in polyphonic musical signals based on Harmonic matching pursuit and spectral smoothness. J New Music Res 2008, 37(3):167183. 10.1080/09298210802474475CrossRefGoogle Scholar
 28.Klapuri A: Multiple fundamental frequency estimation based on harmonicity and spectral smoothness. IEEE Trans Speech Audio Process 2003, 11(6):804816. 10.1109/TSA.2003.815516CrossRefGoogle Scholar
 29.Badeau R, Emiya V, David B: Expectationmaximization algorithm for multipitch estimation and separation of overlapping harmonic spectra. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). Volume I. Taipei, Taiwan; 2009:30733076.Google Scholar
 30.Yeh C: Multiple fundamental frequency estimation of polyphonic recordings. PhD thesis. Université Paris VI  Pierre et Marie Curie; 2008.Google Scholar
 31.Klapuri A: Signal processing methods for the automatic transcription of music. PhD thesis. Tampere Univ. of Technology; 2004.Google Scholar
 32.Maher RC: Evaluation of a method for separating digitized duet signals. J Audio Eng Soc 1990, 38: 956979.Google Scholar
 33.Dijkstra EW: A note on two problems in connexion with graphs. Numerische Mathematik 1959, 1: 269271. 10.1007/BF01386390MathSciNetCrossRefMATHGoogle Scholar
 34.Poliner GE, Ellis DPW: A Discriminative Model for Polyphonic Piano Transcription. EURASIP J Adv Signal Process 2007: 2007.Google Scholar
 35.Klapuri A: Multiple Fundamental Frequency Estimation by Summing Harmonic Amplitudes. In Proc of the Int Conference on Music Information Retrieval (ISMIR). Victoria, Canada; 2006:216221.Google Scholar
 36.Bay M, Ehmann AF, Downie JS: Evaluation of multipleF0 estimation and tracking systems. In Proc of the 10th International Conference on Music Information Retrieval (ISMIR). Kobe, Japan; 2009:315320.Google Scholar
 37.Yeh C, Roebel A, Chang WC: Multiple F0 estimation for MIREX 08. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 38.Chang WC, Su AWY, Yeh C, Roebel A, Rodet X: MultipleF0 tracking based on a highorder HMM model. In Proc of the 11th Int Conference on Digital Audio Effects (DAFx). Espoo, Finland; 2008:379386.Google Scholar
 39.PérezSancho C: Stochastic Language Models for Music Information Retrieval. PhD thesis. Universidad de Alicante, Spain; 2009.Google Scholar
 40.Pertusa A, Klapuri A, Iñesta JM: Recognition of note onsets in digital music using semitone bands. Lecture Notes in Computer Science 2005, 3773: 869879. 10.1007/11578079_90CrossRefGoogle Scholar
 41.Handel S: Listening: An introduction to the perception of auditory events. Bradford Books MIT Press, Cambridge; 1989.Google Scholar
 42.Dixon S: Onset detection revisited. In Proc of the Int Conf on Digital Audio Effects (DAFx). Montreal, Canada; 2006:133137.Google Scholar
 43.Cont A: Realtime transcription of music signals: MIREX 2007 submission description. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 44.Cao C, Li M: Multiple F0 estimation in polyphonic music (MIREX 2008). In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 45.Durrieu JL, Richard G, David B: Singer melody extraction in polyphonic signals using source separation methods. In Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Volume I. Las Vegas, NV; 2008:169172.Google Scholar
 46.Emiya V, Badeau R, David B: Automatic transcription of piano music based on HMM tracking of jointlyestimated pitches. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 47.Egashira K, Ono N, Sagayama S: Sequential estimation of multiple fundamental frequencies through HarmonicTemporalStructured clustering. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 48.Kameoka H, Nishimoto T, Sagayama S: A Multipitch analyser based on harmonic temporal structured clustering. IEEE Trans Audio Speech Language Process 2007, 5(3):982994.CrossRefGoogle Scholar
 49.Groble M: Multiple fundamental frequency estimation. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 50.Lidy T, Rauber A, Pertusa A, Iñesta JM: Improving genre classification by combination of audio and symbolic descriptors using a transcription system. In Proc of the 8th International Conference on Music Information Retrieval (ISMIR). Vienna, Austria; 2007:6166.Google Scholar
 51.Leveau P: A multipitch detection algorithm using a sparse decomposition with instrumentspecific harmonic atoms. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 52.Reis G, Fernandez F, Ferreira A: Genetic algorithm approach to polyphonic music transcription for MIREX 2008. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
 53.Raczynski SA, Ono N, Sagayama S: Multipitch analisys with harmonic nonnegative matrix approximation. In Proc of the 8th Int Conference on Music Information Retrieval (ISMIR). Vienna, Austria; 2007:381386.Google Scholar
 54.Vincent E, Bertin N, Badeau R: Two nonnegative matrix factorization methods for polyphonic pitch transcription. In Proc of the 3rd Music Information Retrieval Evaluation eXchange (MIREX). Vienna, Austria; 2007.Google Scholar
 55.Zhou R, Reiss JD: A realtime polyphonic music transcription system. In Proc of the 4th Music Information Retrieval Evaluation eXchange (MIREX). Philadelphia, PA; 2008.Google Scholar
Copyright information
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.