Nvidia unveils Fugatto: A new AI generator that can make sounds never heard before

Nvidia has introduced Fugatto, a groundbreaking AI-powered music and sound editor capable of generating entirely new and unique audio experiences. The tool, which Nvidia describes as a “creative breakthrough,” uses text and audio prompts to craft sounds, music, and speech it has never been explicitly trained on. This innovative capability includes bizarre yet intriguing examples like a trumpet that meows or a saxophone mimicking howling and barking.

Fugatto allows users to create audio compositions from wild and imaginative prompts. In one example shared by Nvidia, the tool generates a track titled, “Create a saxophone howling, barking then electronic music with dogs barking.” The AI can also produce intricate soundscapes, such as “Deep, rumbling bass pulses paired with intermittent, high-pitched digital chirps, like the sound of a massive sentient machine waking up.”

Fugatto’s capabilities extend beyond just music creation. It can:

    •    Transform voices by altering tone, accent, or emotional expression (e.g., from calm to angry).
    •    Edit music by isolating vocals, adding instruments, or changing melodies (e.g., swapping a piano for an opera singer).
    •    Generate custom sound effects based on detailed textual descriptions.

To develop Fugatto, Nvidia’s researchers assembled a dataset of millions of audio samples. According to Nvidia, the tool was built using advanced instruction-based models that expanded its performance range while enabling it to learn new tasks without additional training data. The dataset also included a variety of sound libraries, including contributions from the BBC, adding to its versatility.

The company emphasizes Fugatto’s ability to deliver unprecedented creative control in audio production, positioning it as a potential game-changer for artists, filmmakers, and sound designers.

While several companies, including Stability AI, OpenAI, and Google DeepMind, have ventured into AI audio tools, Nvidia claims Fugatto stands out by creating entirely unheard-of sounds. Existing AI tools often rely on pre-trained datasets to produce derivative outputs, but Fugatto introduces a new dimension of originality, enabling users to dream up and generate audio that breaks conventional norms.

The rise of AI in music creation hasn’t been without controversy. Several startups are already grappling with copyright lawsuits over their AI-generated music tools. Nvidia itself has faced scrutiny, with reports revealing that AI models were trained on subtitles from thousands of YouTube videos.

Fugatto’s development involved the use of millions of audio samples, but Nvidia has not disclosed how licensing or copyright issues will be handled. However, the tool’s ability to generate unique sounds might help it steer clear of legal grey areas often associated with AI-generated music.

Despite its promising capabilities, Nvidia hasn’t announced when or if Fugatto will be made widely available.