ARIA: Artistic Rendering of Images into Audio

Upload an image and ARIA will analyze its emotional content to generate matching music!

How it works:

  1. ARIA first analyzes the emotional content of your image along two dimensions:
    • Valence: How positive or negative the emotion is (-1 to 1)
    • Arousal: How calm or excited the emotion is (-1 to 1)
  2. These emotions are then used to generate music that matches the mood

Quick Start

  1. Click an example artwork below or upload your own image above.
  2. (Optional) Open Advanced Settings to fine-tune the generation.
  3. Hit Generate Music to inference the model!

Generation Settings
Conditioning Type

How the emotional information is incorporated into the music generation
256 4096
0.1 2
0.1 2
0.1 1
1 5
Example Artworks (click to load)
Upload Image Conditioning Type Generation Length Note Temperature Rest Temperature Top-p Sampling Minimum Instruments

About ARIA

ARIA is a deep learning system that generates music from artwork by:

  1. Using a image-emotion model to extract emotional content from images
  2. Generating matching music using an emotion-conditioned music generation model

The emotion-conditioned MIDI generation model is based on the work by Serkan Sulun et al. in their paper "Symbolic music generation conditioned on continuous-valued emotions". Original implementation: github.com/serkansulun/midi-emotion

Conditioning Types

continuous_concat (Recommended)
Creates a single vector from valence and arousal values, repeats it across the sequence, and concatenates it with every music token embedding. This approach gives the emotion information global influence throughout the entire generation process, allowing the transformer to access emotional context at every timestep. Research shows this method achieves the best performance in both note prediction accuracy and emotional coherence.

continuous_token
Converts each emotion value (valence and arousal) into separate condition vectors with the same length as music token embeddings, then concatenates them in the sequence dimension. The emotion vectors are inserted at the beginning of the input sequence during generation. This treats emotions similarly to music tokens but can lose influence as the sequence grows longer.

discrete_token
Quantizes continuous emotion values into 5 discrete bins (very low, low, moderate, high, very high) and converts them into control tokens. These tokens are placed before the music tokens in the sequence. While this represents the current state-of-the-art approach in conditional text generation, it suffers from information loss due to binning and can lose emotional context during longer generations when tokens are truncated.