Paper year
2026
Paper dossier
Multimodal Automatic Music Transcription Using Piano Audio and Hand-Skeleton Information
Detail viewSimilarity handoff
Review source metadata, abstract, authors, topics, and local similarity context before moving into explanation and ranking views.
Citations
0
Authors
0
Topic labels
0
Source readout
Source and corpus status
Venue
Unknown venue
Source slug
unknown
Corpus placement
Controlled edge slice
Similarity rows
Not available yet
Ranking readout
Where this paper lands in the current run
Ranking details could not be loaded (API 503).
Abstract
Automatic Music Transcription (AMT) for piano is difficult for audio-only systems due to dense polyphony, resonance, and reverberation, which lead to false positives and unstable onset decisions. We present a multimodal AMT framework that fuses Omnizart audio probability maps with visual cues from hand-skeleton tracking. A graph-based model called HandSkeletonNet estimates per-key onset probabilities from hand trajectories, and the two modalities are merged via a weighting-and-masking scheme or a compact CNN-based merger. Experiments show consistent improvements over the audio-only baseline on our self-compiled dataset, while evaluations with external datasets primarily improve frame-level sensitivity. The frame-level F1 score improved from 75.12% to 75.76% for the PianoYT dataset and from 54.68% to 57.57% for the PianoVAM dataset compared with the audio-only baseline. Our experiments also reveal limited onset-level gains under domain shift. Remaining errors are largely explained by timing/misalignment and note fragmentation in MIDI decoding, suggesting that robustness to missing hand detections and explicit temporal alignment are key directions.
Authors
No authors available.
Neighborhood labels
Topics
0 labels
Topic labels are imported metadata and can be noisy; use them as coarse navigation hints, not authoritative classifications.
Neighbor surface
Similar papers
Similar papers use a separately configured neighbor embedding; it may differ from the embedding version used by the current ranked run.
No embedding-backed neighbors available for this paper/version yet.
Next handoff
Best next moves from here
01
Check recommendation families
Use Recommended to see whether this paper behaves like an emerging or undercited signal in the current ranked feed, or how it appears on the bridge preview / diagnostics view.
02
Inspect nearby topics
Use Trends to understand whether its attached labels are heating up or cooling down inside the curated corpus.
03
Cross-check evaluation baselines
Use Evaluation to compare the dossier readout against citation and recency baselines for the same resolved family run.