Paper year
2026
Paper dossier
Investigating Auditory-Visual Perception Using Multi-Modal Neural Networks with the SoundActions Dataset
Detail viewSimilarity handoff
Review source metadata, abstract, authors, topics, and local similarity context before moving into explanation and ranking views.
Citations
0
Authors
3
Topic labels
3
Source readout
Source and corpus status
Venue
Transactions of the International Society for Music Information Retrieval
Source slug
tismir
Corpus placement
Core corpus
Similarity rows
Not available yet
Ranking readout
Where this paper lands in the current run
Run shadow-generalization-product-candidate-ranking-v1Top 50 surfaced
This block uses the same resolved ranking run as Recommended. Ranks here are materialized paper_scores ranks; live Emerging may be reordered by the bounded ML scorer. Family rank is global within each family, but rank is only shown when this paper lands inside the surfaced top 50.
Families present
3
Top 50
0
Run label
shadow-generalization-product-candidate-ranking-v1
Snapshot
source-snapshot-shadow-generalization-v1-20260521
Scope: family global | run rank-83787b91ef
Emerging
Present in run, outside top 50
Emerging: embedding slice fit vs included-corpus centroid (title+abstract), plus citation velocity and topic growth; not universal relevance. Bridge signal not used here.
Signals: semantic=0.8687, citation_velocity=0.0000, topic_growth=0.6976, diversity_penalty=0.0000
Why this surfaced | 3 used | 1 penalty | 1 not computed
Embedding slice fit (corpus centroid)used
Embedding slice fit (corpus centroid): high; used in final ranking (contribution to score: 0.1737)
Recent attentionused
Recent attention: low; used in final ranking (contribution to score: 0.0000)
Topic momentumused
Topic momentum: high; used in final ranking (contribution to score: 0.2093)
Cross-cluster signalnot computed
Cross-cluster signal: not computed for this run
Similarity penaltypenalty
Similarity penalty: reduces score when non-zero (contribution to score: 0.0000)
Bridge
Present in run, outside top 50
Multi-topic paper in active topics; no cluster_version on this run so bridge_score was not computed.
Signals: citation_velocity=0.0000, topic_growth=0.6976, diversity_penalty=0.0000
Why this surfaced | 2 used | 1 penalty | 2 not computed
Semantic matchnot computed
Semantic match: not computed for this run
Recent attentionused
Recent attention: low; used in final ranking (contribution to score: 0.0000)
Topic momentumused
Topic momentum: high; used in final ranking (contribution to score: 0.4534)
Cross-cluster signalnot computed
Cross-cluster signal: not computed for this run
Topic breadth penaltypenalty
Topic breadth penalty: reduces score when non-zero (contribution to score: 0.0000)
Under-cited
Present in run, outside top 50
Low-cite candidate pool (see docs/candidate-pool-low-cite.md v0): core corpus, recency floor, citation ceiling, title+abstract gate; popularity penalty among pool members only. Semantic and bridge not yet modeled.
Signals: citation_velocity=0.0000, topic_growth=0.6976, diversity_penalty=0.0000
Why this surfaced | 2 used | 1 penalty | 2 not computed
Semantic matchnot computed
Semantic match: not computed for this run
Recent attentionused
Recent attention: low; used in final ranking (contribution to score: 0.0000)
Topic momentumused
Topic momentum: high; used in final ranking (contribution to score: 0.4883)
Cross-cluster signalnot computed
Cross-cluster signal: not computed for this run
Pool popularity penaltypenalty
Pool popularity penalty: reduces score when non-zero (contribution to score: 0.0000)
Abstract
Musicologists, psychologists, and computer scientists study relationships between auditory and visual stimuli from very different perspectives and using various terminologies and methodologies. This article aims to bridge the gap between phenomenological sound theory, auditory-visual theory, and audio-video processing and machine learning. We introduce the SoundActions dataset, a collection of 365 audio-video recordings of (primarily) short sound actions. Each recording has been human‑labeled and annotated according to Pierre Schaeffer's theory of reduced listening, which describes the property of the sound itself (e.g., 'an impulsive sound') instead of the source (e.g., 'a bird sound'). With these reduced‑type labels in the audio-video dataset, we conducted two experiments: (1) fine‑tuning the latest audio-video transformer model on the reduced‑type labels in the SoundActions dataset, proving that the model can recognize reduced‑type labels, and observing that the modality‑imbalance phenomenon is similar to the added value theory by Michel Chion and (2) proposing the Ensemble of Perception Mode Adapters method inspired by Pierre Schaeffer's three listening modes, improving the audio-video model also on reduced‑type tasks.
Authors
Neighborhood labels
Topics
3 labels
Topic labels are imported metadata and can be noisy; use them as coarse navigation hints, not authoritative classifications.
Music and Audio ProcessingMultisensory perception and integrationTactile and Sensory Interactions
Neighbor surface
Similar papers
Similar papers use a separately configured neighbor embedding; it may differ from the embedding version used by the current ranked run.
No embedding-backed neighbors available for this paper/version yet.
Next handoff
Best next moves from here
01
Check recommendation families
Use Recommended to see whether this paper behaves like an emerging or undercited signal in the current ranked feed, or how it appears on the bridge preview / diagnostics view.
02
Inspect nearby topics
Use Trends to understand whether its attached labels are heating up or cooling down inside the curated corpus.
03
Cross-check evaluation baselines
Use Evaluation to compare the dossier readout against citation and recency baselines for the same resolved family run.