MusGen

Context

A study project for a Machine Learning course at university. We had to come up with a project topic and I had a rough time thinking about something fun to do. After a consultation with the lecturer, I had an idea to try processing various pieces of music by a composer and creating a track resembling that style, and I wanted to use neural nets because that seemed cool. (At that point we didn't have an established concept of neural style transfer)

I soon realized this was quite a challenging task both compute-time-wise and in regard to actually creating something meaningful. I intuitively simplified the task to a memorization task for the neural net. I utilized the LSTM architecture which came up in initial topic research.

What I Built

I actually did a bit of research in academic papers about the approaches used and potential avenues to try. This already gave me a hint that most of the research dealt with a single track, often monophonic, so I was way above my league and needed to reconsider the scope.

The semester work was parsing input MIDI files, converting them to a custom serialization vector and feeding them into an LSTM neural network to train the weights. It allowed me to generate a music track from an input initial state and serialize it to a MIDI file.

The project was built to utilize MIDI as input, which was serialized as a series of fixed-width feature vectors. I used an existing Java library for the LSTM neural network architecture (XLBP for Java) with a custom training and evaluation loop. This was an early exploration of machine learning for creative/artistic output that influenced a few other projects.

The generated tracks actually had a recognizable main motif from the original track, but they were far from perfect copies. The parts between the core motifs were mostly improvisation or straight up silence.

Training on multiple songs was attempted with 10-song and 30-song sets, but the 30-song model degraded into oscillation/no output.

Interesting Problems

• Music file type — at that point I couldn't imagine dealing with MP3, WAV or other "real sound" formats, so I pivoted to MIDI tracks that provided nice piano-roll-like serialization, that could be mapped onto a feature vector and also could encode polyphonic melodies.
• Serialization of the music track — prior academic works were dealing with very simplified monophonic encoding or chord progressions. I decided I wanted to tackle that in a more general approach and went with (A) slicing of the track per constant time intervals. The features were encoded as sound volumes of individual (128) notes at a given slice (multiple tracks in a MIDI file were merged into a single track). I decided to use 1/32 note for the slicing interval based on the trade-off between actual accuracy and memory demand. And (B) serialization of track events into vectors of size 4 [time_since_last_event; pitch; event_length; volume]. In theory this could be memory efficient with strong generalization, but the training experiment was unsuccessful as the error curve was rising from the start.
• Volume information for learning — keeping the volume information encoded into a vector proved to be problematic for efficient neural net training. The generated tracks had frequent volume skips between very loud and silent notes even for the same stable tone. I decided to encode only on/off states via 0.99 and 0.01 values to simplify the learning and provide a smoother-sounding generated track.
• Hyperparameters of neural net — I studied the behavior of various hidden layer sizes ranging from 25 to 10000 neurons. The performance peaked around 500 neurons and additional neurons provided marginal improvement at best. More often the larger nets started oscillating in the learning process after some iterations and provided worse generation quality.

Related projects

This inspired other creative projects:

• Muscell - generating music based on cellular automaton