Public benchmark · OLiMPiC · optical music recognition (OMR)
OLiMPiC piano OMR benchmark: PlayableNotes on all 1,493 scanned systems
OLiMPiC is a public piano OMR test set from Charles University (ICDAR 2024): 1,493 piano systems cut from IMSLP scans of 19th-century song accompaniments, each paired with a hand-made MusicXML answer and scored with the dataset authors’ own TEDn code. Across all 1,493 systems in one unedited run (0 failed), PlayableNotes scored 12.07% TEDn full, 8.08% TEDn (LMX) and 9.96% symbol error rate (lower is better), compared with 18.40% TEDn full and 17.72% SER for the authors’ Zeus model and 13.67% SER for U-MusT.
Checked MethodRun by PlayableNotes. All 1,493 predictions are public on GitHub.
Table 1 · full test set
All 1,493 scanned OLiMPiC systems (lower is better)
| System | TEDn full | TEDn (LMX) | Symbol error rate | Test systems | Source |
|---|---|---|---|---|---|
| PlayableNotes | 12.07% | 8.08% | 9.96% | All 1,493 | Run 2026-11-19 (this repo) |
| U-MusT | – | – | 13.67% | All 1,493 | Jung et al. 2025, Table VII |
| Zeus retrain | – | – | 14.45% | All 1,493 | Jung et al. 2025, Table VII |
| Zeus | 18.40% | 14.85% | 17.72% | All 1,493 | Mayer et al. ICDAR 2024, Table 3 |
Lower is better on every column. PlayableNotes read all 1,493 systems across 100 songs once, unedited, on 19 November 2026: 0 failed, 2.96% note errors. Zeus figures are from Mayer et al. (ICDAR 2024, Table 3, scanned, augmented, all 1,493). U-MusT and the Zeus retrain are from Jung et al. (2025, arXiv:2505.12863, Table VII, OLiMPiC scanned), which reports symbol error rate only (dashes where no TEDn figure is published).
Table 2 · head-to-head on 60 systems
Same 60 systems: PlayableNotes, Sheetly and Audiveris 5.11.0
Sheetly published raw MusicXML outputs for 60 OLiMPiC scanned systems — the first system (p1-s1) of 60 different songs — together with Audiveris 5.11.0 outputs for 50 of them (loveranmar/omr-benchmark, olimpic60/, commit 83f0077). Scoring PlayableNotes’ predictions from the full run against those same 60 systems gives a direct comparison.
| System | TEDn full | TEDn (LMX) | Symbol error rate | No output |
|---|---|---|---|---|
| PlayableNotes | 12.66% | 8.45% | 9.55% | 0 / 60 |
| Sheetly | 38.78% | 35.92% | 35.68% | 0 / 60 |
| Audiveris 5.11.0 | 45.14% | 42.52% | 47.96% | 10 / 60 |
Lower is better. All three engines’ MusicXML outputs were canonicalized identically before TEDn using sort_chords.py (chord notes ascending) and voice_order.py (each measure written voice by voice in MuseScore’s canonical order); without that step, Sheetly scores 59.7% TEDn full. Audiveris 5.11.0 produced no output on 10 of the 60 systems, which score as empty output. System list and per-system scores: olimpic/compare-60/.
At a glance
- Full test set (1,493)
- PlayableNotes scored 12.07% TEDn full, 8.08% TEDn (LMX) and 9.96% SER across all 1,493 scanned systems (2.96% note errors, 0 failed).
- Published papers
- Zeus (ICDAR 2024): 18.40% TEDn full, 14.85% TEDn (LMX), 17.72% SER. U-MusT (2025): 13.67% SER; Zeus retrain: 14.45% SER (SER only).
- Same 60 systems
- PlayableNotes 12.66% TEDn full (9.55% SER), Sheetly 38.78% (35.68% SER), Audiveris 5.11.0 45.14% (47.96% SER, 10 with no output).
- Dataset
- OLiMPiC scanned test set (Charles University, ICDAR 2024): 1,493 piano systems from 100 IMSLP 19th-century song accompaniments, with gold MusicXML.
- Single unedited run
- Every system was read once, unedited, in one run; failures score as empty output and none failed.
- Check it
- All 1,493 predictions, per-system scores, scripts and the 60-system comparison are public on GitHub.
What OLiMPiC, TEDn and SER mean
OLiMPiC (Mayer, Straka, Hajič jr. and Pecina, Charles University, ICDAR 2024) tests piano optical music recognition on 1,493 grand-staff systems cut from real IMSLP scans of 100 nineteenth-century song accompaniments, each with a hand-made MusicXML answer and the authors’ own evaluation code.
- TEDn full. Normalized tree edit distance between the predicted MusicXML and the gold MusicXML across all notation elements, roughly the share of the score tree that needs editing (lower is better).
- TEDn (LMX). The same normalized tree edit distance restricted to the notation elements represented in OLiMPiC’s linear LMX format, leaving out MusicXML details LMX does not encode (lower is better).
- Symbol error rate (SER). Normalized token edit distance between the linearized LMX token sequence of the prediction and the gold LMX sequence (lower is better).
Method
- Test set. All 1,493 scanned piano systems in the OLiMPiC test split (
samples.test.txt, 100 songs from IMSLP). - One unedited run. Every system was read once by PlayableNotes, unedited, in a single run on 19 November 2026. Any system that failed to produce output would score as empty output; none failed (0 of 1,493).
- Scoring. Each prediction’s MusicXML piano part was linearized and scored against the gold MusicXML and LMX using the OLiMPiC authors’ own
LinearizerandTEDn_lmx_xmlcode (olimpic/score.py), pooling total edit cost over total gold cost across all 1,493 systems as the authors’ scorer does. - Same-60 comparison. The 60 systems in Table 2 are Sheetly’s published selection (the first system of 60 different songs in
loveranmar/omr-benchmark). All three engines’ MusicXML files were passed throughsort_chords.pyandvoice_order.pybefore TEDn; missing outputs (10 for Audiveris 5.11.0) were scored as empty.
Check it yourself
All 1,493 predictions (prediction.musicxml, tedn.json and result.json per system) live in olimpic/runs/2026-11-19/, and the 60-system comparison lives in olimpic/compare-60/, pinned to commit d867293 of nmharmon8/sheet-music-scanner-benchmark. An earlier 300-system sample from 18 November 2026 is kept in olimpic/runs/2026-11-18/ as history.
To verify the full-set numbers with the OLiMPiC authors’ scorer, run olimpic/setup.sh (downloads the scanned dataset and checks out ufal/olimpic-icdar24 at a pinned commit), then run olimpic/.venv/bin/python olimpic/score.py 2026-11-19 to re-score every prediction from scratch, or add --cached to sum the committed tedn.json files. Instructions for reproducing the 60-system table are in olimpic/compare-60/README.md.
Frequently asked questions
What is the OLiMPiC benchmark?
OLiMPiC is a public piano optical music recognition dataset from Charles University (Mayer, Straka, Hajič jr. and Pecina, ICDAR 2024). Its scanned test split has 1,493 piano systems cut from IMSLP scans of 100 nineteenth-century song accompaniments, each paired with a hand-made MusicXML answer and scored with the authors’ own TEDn code.
What do TEDn full, TEDn (LMX) and SER measure?
TEDn (normalized tree edit distance) measures how much of a predicted MusicXML tree must be edited to match the gold MusicXML: TEDn full scores all MusicXML elements, and TEDn (LMX) scores the notation subset represented in the dataset’s linear LMX format. Symbol error rate (SER) is the normalized token edit distance between the linearized prediction and the gold LMX sequence. Lower is better on all three.
Why do U-MusT and the Zeus retrain have dashes in the TEDn columns?
Jung et al. (2025, arXiv:2505.12863, Table VII) report only symbol error rate on OLiMPiC scanned — 13.67% for U-MusT and 14.45% for their retrained Zeus baseline — and do not publish TEDn full or TEDn (LMX) numbers.
How were the 60 systems in the Sheetly and Audiveris comparison chosen?
They are the 60 systems Sheetly published raw MusicXML outputs for in loveranmar/omr-benchmark (olimpic60/, commit 83f007724e16a9fa821f857606156c28a92bc5e9): the first system (p1-s1) of 60 different OLiMPiC test songs. That commit also holds Audiveris 5.11.0 outputs for 50 of the 60 systems; the 10 systems where Audiveris produced no output are scored as empty, matching the OLiMPiC scorer.
Why were the MusicXML files canonicalized in the 60-system comparison?
The OLiMPiC linearizer expects chord notes in ascending pitch order and measures serialized voice by voice (MuseScore’s canonical order). All three engines in the 60-system comparison — PlayableNotes, Sheetly and Audiveris 5.11.0 — were passed through the same two scripts (sort_chords.py and voice_order.py) without changing pitches, durations, chords, beams, ties or notations. Without that step, Sheetly scores 59.7% TEDn full.
How can I verify or reproduce these scores?
Clone nmharmon8/sheet-music-scanner-benchmark at commit d867293, run olimpic/setup.sh to fetch OLiMPiC and the authors’ scorer at a pinned commit, then run olimpic/.venv/bin/python olimpic/score.py 2026-11-19 to re-score all 1,493 predictions (or pass --cached to sum the committed per-system tedn.json files).
Read your own page and check it.
New accounts get 50 free reading credits, about 7 to 10 piano pages. Compare the result with your page, then listen, slow it down and practise hands apart.
Read a page free → Or open a free library piece
50 free reading credits. No card needed, no subscription.
