SyncViolinist: Music-Oriented Violin Motion Generation Based on Bowing and Fingering

Hiroki Nishizawa* · Keitaro Tanaka* · Asuka Hirata* · Shugo Yamaguchi
Qi Feng · Masatoshi Hamanaka · Shigeo Morishima
(* - Equal contribution)

SyncViolinist is a multi-stage end-to-end framework that generates synchronized violin performance motion solely from audio input. For more details please refer to the Paper.

For more details check out the YouTube video below.

Table of Contents

• Description
• Requirements
• Installation
• Getting Started
• Examples
• Citation
• Acknowledgments
• Contact

Description

This repository includes the code base for the SyncViolinst and captured dataset.

Installation

To install the dependencies please follow the next steps:

• Clone this repository:

  git clone https://github.com/Kakanat/SyncViolinist.git
  cd SyncViolinist
  

Getting started

In order to run SyncViolinst, download the dataset and create a data/ directory and follow the steps below:

SyncViolinst Dataset and models

• Contact phys.keitaro1227@ruri.waseda.jp to request access to the dataset and the pre-trained model.
• Store the files under dataset/ to under SyncViolinst/data/.
• The final structure of data should look as below:

    SyncViolinst
    └── data
        ├── aud
        │    ├── test
        │    ├── train
        │    └── validation
        ├── joint_aligned
        │    ├── test
        │    ├── train
        │    └── validation
        ├── keyps_norm_hiroki_ver_6ch
        │    ├── test
        │    ├── train
        │    └── validation
        ├── mfcc
        │    ├── test
        │    ├── train
        │    └── validation
        ├── pp_one_hot
        │    ├── test
        │    ├── train
        │    └── validation
        ├── skl_hiroki_ver
        │    ├── test
        │    ├── train
        │    └── validation
        └── wav_normalized
             ├── test
             ├── train
             └── validation

Pre-trained Checkpoints

• You can download the pre-trained model from the dataset.
• Place the pre-trained models in SyncViolinst/models as follows:

    models
    ├── A2BD
    │   ├── best.pth
    │   └── config.yaml
    ├── hiroki
    │   ├──
    │   ...
    │   └──
    ├── TGM2B
    │   ├── best.pth
    │   └── config.yaml
    └── TVCG
        ├── best.pth
        └── config.yaml

Environment and Installation

We use the docker container. Please use the Dockerfile to set the implement environment.

docker build -t {image name} .
docker run -it --gpus [GPU number] -v $(pwd):/workspace -p 8000:8000 --name {container name} {image name}

Examples

After installing the dependencies and downloading the data and the models, you should be able to run the following examples:

bash evaluation.sh

The results are saved under "./results/{model name}/"

Citation

@inproceedings{}

Acknowledgments

This research is supported by JSPS KAKENHI No. 21H05054, 24H00742, and 24H00748.

Contact

This repository is maintained by Hiroki Nishizawa, Keitaro Tanaka, and Qi Feng.

For questions, please contact phys.keitaro1227@ruri.waseda.jp.