Music-Conditioned Decisions
Beat-level musical context driving every formation selection in real time.
Group ID: 25-26j-358
SyncForm
Computational Approach for Automated Generation of Synchronized Group Dance Choreography Formations
SyncForm is an end-to-end intelligent choreography support system that transforms uploaded music into spatially coherent, collision-aware dance formation timelines. Powered by a four-stage machine learning pipeline — music structure analysis, formation generation, transition planning, and quality evaluation — it bridges the gap between isolated academic methods and a practical choreography generation workflow deployable in real-world settings.
4
ML Modules
E2E
Pipeline
Pipeline Overview
M1
Music Analysis
M2
Formation Gen
M3
Transition
M4
Evaluation
Formation Preview
Domain Overview
Explore the research landscape, methodology, and technical foundation of SyncForm
Literature Survey & Background
Music-driven choreography generation sits at the intersection of audio signal processing, generative machine learning, motion planning, HCI, and performance analytics. The literature base relevant to SyncForm groups into five primary themes, surveyed from peer-reviewed sources:
Audio Understanding
Beat tracking, onset detection, spectral features, and MFCC-based embeddings form the standard foundation for music-aware movement generation. Early work by Tzanetakis & Cook [3] established genre classification from audio signals using timbral texture, rhythmic, and pitch-based features, laying groundwork for music-context-aware systems. McFee et al.'s librosa [1] later consolidated these techniques into an accessible Python framework for audio loading, beat-synchronous feature extraction, and spectral analysis. For tempo estimation, SyncForm adopts Böck et al.'s recurrent neural network approach to joint beat and downbeat tracking [2], which significantly outperforms autocorrelation-based methods on varied musical genres. Together, these methods produce the beat-indexed conditioning structures consumed downstream by the formation generation module.
Conditional Generative Models
The generative backbone of SyncForm draws from two principal lines of research. Kingma & Welling's Variational Autoencoder (VAE) [4] introduced a principled probabilistic framework for learning compact latent representations from which diverse outputs can be sampled; SyncForm's CVAE path conditions this latent space on 37-dimensional audio feature vectors. More recently, Ho et al.'s Denoising Diffusion Probabilistic Models (DDPM) [5] demonstrated superior sample quality over GANs for complex data distributions, motivating their adoption as SyncForm's primary generation path. Crucially, SyncForm constrains its diffusion model to predict an 18-dimensional formation parameter vector — 14 formation-type logits, spread, rotation, stagger, and vertical offset — rather than raw dancer coordinates. This parameter-space approach provides substantially improved stability when real annotated training data is limited.
Group Dance Generation
The sub-field of music-conditioned dance generation has advanced considerably in recent years. Li et al.'s AIST++ dataset and model [11] established cross-modal conditioning benchmarks for solo dance using 3D pose estimation, and its 37-dimensional music feature representation directly informs SyncForm's conditioning vector design. Tseng et al.'s EDGE [7] demonstrated editable dance generation through a diffusion-based architecture with music conditioning, validating that diffusion models can generate temporally coherent motion sequences. Le et al.'s GDANCE [6] represents the closest prior work to SyncForm, introducing a framework for music-driven group choreography; however, GDANCE focuses on motion trajectories rather than discrete spatial formation patterns and does not address collision avoidance or deployable system interfaces — gaps that SyncForm directly addresses.
Multi-Agent Path Planning
Transitioning a group of dancers between formations introduces a combinatorial assignment problem and a collision-avoidance problem. Kuhn's Hungarian method [8] solves the former optimally in polynomial time by computing a minimum-cost bipartite matching between current dancer positions and target formation slots. For the latter, Sharon et al.'s Conflict-Based Search (CBS) algorithm [9] provides a complete and optimal multi-agent pathfinding solution through a two-level constraint tree that isolates and resolves inter-agent conflicts. SyncForm additionally employs Schulman et al.'s Proximal Policy Optimization (PPO) [10] to train a reinforcement learning agent that selects section-aware transition curve types — ease_in_out, anticipation, or bezier — based on the musical character of each segment, giving transitions an expressively choreographed quality beyond simple linear interpolation.
Formation Evaluation & Applied Systems
Objective evaluation of choreographic quality remains an open research challenge. Aristidou et al. [12] demonstrated music-driven motion synthesis with global structural coherence evaluation, confirming that rule-based spatial metrics can meaningfully quantify visual quality in performance contexts. SyncForm builds on this by defining a Formation Quality Index (FQI) encompassing symmetry, inter-dancer spacing, center bias, compactness, collision safety, stage coverage, and visual appeal — supplemented by a CNN classifier operating on formation heatmaps. Beyond evaluation, SyncForm departs from prior academic work by exposing the full pipeline through a FastAPI ML backend, a Node.js real-time collaboration server, and a React-based frontend, reflecting applied ML system design principles and making the research accessible to non-technical choreographers.
References
- [1] B. McFee, C. Raffel, D. Liang, D. P. W. Ellis, M. McVicar, E. Battenberg, and O. Nieto, "librosa: Audio and Music Signal Analysis in Python," in Proc. 14th Python in Science Conf. (SciPy), Austin, TX, USA, Jul. 2015, pp. 18–25. DOI ↗
- [2] S. Böck, F. Krebs, and G. Widmer, "Joint Beat and Downbeat Tracking with Recurrent Neural Networks," in Proc. 17th Int. Soc. Music Inf. Retrieval Conf. (ISMIR), New York, NY, USA, Aug. 2016, pp. 255–261. PDF ↗
- [3] G. Tzanetakis and P. Cook, "Musical Genre Classification of Audio Signals," IEEE Trans. Speech Audio Process., vol. 10, no. 5, pp. 293–302, Jul. 2002. DOI ↗
- [4] D. P. Kingma and M. Welling, "Auto-Encoding Variational Bayes," in Proc. 2nd Int. Conf. Learning Representations (ICLR), Banff, AB, Canada, Apr. 2014. arXiv ↗
- [5] J. Ho, A. Jain, and P. Abbeel, "Denoising Diffusion Probabilistic Models," in Proc. 34th Conf. Neural Inf. Process. Syst. (NeurIPS), Vancouver, BC, Canada, Dec. 2020, pp. 6840–6851. arXiv ↗
- [6] N. Q. Le, B. X. Pham, T. M. Tran, D. L. Tran, C.-M. Bui, and T.-T. Do, "Music-Driven Group Choreography," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, Jun. 2023, pp. 10765–10774. arXiv ↗
- [7] J. Tseng, O. Castellon, and K. Liu, "EDGE: Editable Dance Generation From Music," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, Jun. 2023. arXiv ↗
- [8] H. W. Kuhn, "The Hungarian Method for the Assignment Problem," Naval Res. Logist. Q., vol. 2, no. 1–2, pp. 83–97, 1955. DOI ↗
- [9] G. Sharon, R. Stern, A. Felner, and N. R. Sturtevant, "Conflict-Based Search for Optimal Multi-Agent Pathfinding," Artif. Intell., vol. 219, pp. 40–66, Feb. 2015. DOI ↗
- [10] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal Policy Optimization Algorithms," arXiv preprint arXiv:1707.06347, Jul. 2017. arXiv ↗
- [11] R. Li, S. Yang, D. A. Ross, and A. Kanazawa, "AI Choreographer: Music Conditioned Automatic Dance Generation with AIST++," in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), Montreal, QC, Canada, Oct. 2021, pp. 13401–13410. arXiv ↗
- [12] A. Aristidou, C. Yiannakidis, K. Ntalianis, and Y. Chrysanthou, "Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structure," IEEE Trans. Vis. Comput. Graph., vol. 28, no. 3, pp. 1367–1378, Mar. 2022. DOI ↗
Research Gap
Most existing solutions do not address all of the following within a single integrated system:
Geometrically Valid Generations
Spatial coherence maintained for varying dancer counts without manual constraints.
Collision-Aware Transitions
Explicit collision detection during movement transitions, not just static formations.
Objective Evaluation
Multi-dimensional quality metrics beyond subjective judgment or single scalars.
Deployable APIs & UI
Full-stack system for real-world choreography support, beyond offline notebooks.
Collaborative Workflow
Real-time socket-based collaboration for distributed choreography teams.
Research Problem
How can an end-to-end system generate musically aligned, spatially coherent, collision-aware dance formations for a configurable number of dancers, while also exposing objective quality metrics and deployable software interfaces suitable for real-world choreography support?
Sub-Problems Explicitly Tackled
- Identifying beat-level musical structure from uploaded audio files
- Translating musical context into meaningful formation choices
- Generating spatial layouts that remain bounded, balanced, and visually valid
- Ensuring transitions are physically plausible and low-risk for all dancers
- Evaluating outputs with repeatable, multi-dimensional metrics
- Exposing the pipeline through usable backends and interactive frontend tooling
Research Objectives
The main objective is to design and implement a complete music-driven dance formation generation system. Specific objectives:
Music Analysis
Analyze music files to derive beat maps, section labels, energy signals, and 28-dimensional DSP embeddings usable by downstream modules.
Formation Generation
Generate dance formations conditioned on musical context and dancer count using CVAE and diffusion generative models.
Transition Planning
Plan transitions using Hungarian assignment and PPO reinforcement learning for collision-aware, beat-indexed path planning.
Quality Evaluation
Evaluate formations and timelines through geometric FQI metrics and a CNN-based style classifier operating on formation heatmaps.
Deployable Interfaces
Expose the pipeline through FastAPI, Node.js services, MongoDB persistence, and a React 19 + Three.js frontend studio.
Research Infrastructure
Support experimentation through Colab notebooks, training checkpoints, validation utilities, and reproducible JSON outputs.
System Methodology
The implemented methodology is modular and pipeline-oriented:
M1
Music Structure Analysis
Audio loaded via librosa at 22.05 kHz. BeatTracker detects beats; FeatureExtractor computes per-beat and global DSP features. StructureAnalyzer segments sections. FormationPointSelector picks change-points. Output: beat-indexed conditioning structure with 28-D embeddings, timing, energy, section labels, and dynamics.
M2
Formation Generation — CVAE vs Diffusion
Diffusion model predicts an 18-dimensional parameter vector: 14 formation-type logits + spread, rotation, stagger, and vertical offset — decoded via geometric templates. CVAE provides a comparison and fallback path. Conditioning vector: 37-D (28 DSP + 7 section one-hot + 1 energy + 1 dancer-count). This parameter-space approach improves stability over raw-coordinate generation on limited data.
M3
Transition Planning — PPO Reinforcement Learning
Assignment: Hungarian algorithm for minimum-cost dancer-to-slot mapping. Path planning: Conflict-based search for collision-aware multi-dancer paths. Motion shaping: PPO-trained RL agent selects section-aware transition curves (ease_in_out, anticipation, bezier). Timeline is beat-indexed, not frame-indexed — a key design decision affecting export and playback.
M4
Evaluation — CNN Style Classifier
HybridEvaluationFramework combines rule-based FQI metrics (symmetry, spacing, center bias, compactness, balance, collision safety, stage coverage, visual appeal) with a CNN style classifier operating on formation heatmaps. Sequence-level metrics include variety, transition smoothness, and aggregate timeline scores.
Technologies Used
Machine Learning
PyTorch
PyTorch Lightning
Stable-Baselines3
scikit-learn
NumPy
SciPy
Audio Processing
librosa
soundfile
FFmpeg
Backend
FastAPI
Flask
Node.js
Express.js
Socket.IO
Frontend
React 19
Vite
Zustand
Three.js
Tailwind CSS
Data & Storage
MongoDB
Mongoose
JSON
Research Tools
Jupyter
Google Colab
matplotlib
OpenCV
Docker
Project Milestones
Key assessment points and deliverables for this research project
-
Proposal
Project Proposal
Initial presentation of the research concept, problem statement, objectives, and proposed methodology.
-
Progress P1
Progress Presentation 1
Progress Presentation reviews the 50% completion status of the project. This reveals any gaps or inconsistencies in the design/requirements.
-
Progress P2
Progress Presentation 2
Progress Presentation II reviews the 90% completion status demonstration of the project.
-
Final
Final Presentation and Viva
Comprehensive evaluation of the complete SyncForm system including full pipeline demonstration, evaluation metrics, and complete thesis documentation.
-
Website
Website Evaluation and Logbook Submission
-
Research Paper
Research Paper
-
Final Thesis Report
Final Thesis Report
Project Presentations
Access slide decks from each milestone presentation
Proposal
Proposal Presentation
PP-1
Progress Presentation-1
PP-2
Progress Presentation-2
Final
Final Presentation
About Us
Meet the researchers behind SyncForm — Group 25-26J-358
Supervisor
Co Supervisor
Leader
Member
Member
Member
External Supervisor
External Supervisor
Contact Us
Have questions about SyncForm? Reach out to the research team.
Phone
+94 70 535 6643 (Team Lead)
Institution
Sri Lanka Institute of Information Technology (SLIIT)
Department of Information Technology
Group Information
Group ID: 25-26j-358
Academic Year: 2025/2026