[FEATURE REQUEST] Quality simulation for AudioLDM and maybe others about audio-webui

[pravdomil]

great thanks!

[gitmylo]

What does it do?

[pravdomil]

try it, it improves the audio quality a lot

[gitmylo]

Looks like it resamples to 44khz, which is higher than models usually output, and it tries to fix some of the imprecision, also looks like it applies stereo, which should sound better.

i can see why you want that, i'll add it to the utils soon.

[pravdomil]

this improves a quality a lot

sample_rate = 44100


def main(prompt):
    pipe = diffusers.AudioLDMPipeline.from_pretrained("cvssp/audioldm-s-full-v2", torch_dtype=torch.float16).to("cuda")

    audio = pipe(prompt, num_inference_steps=10, audio_length_in_s=a.duration).audios[0]
    audio_resampled = librosa.resample(audio, orig_sr=16000, target_sr=sample_rate)
    audio_enhanced = enhance_audio(audio_resampled)

    return audio_enhanced


def enhance_audio(a: numpy.ndarray):
    octave_up = librosa.effects.pitch_shift(a, sr=sample_rate, n_steps=12, res_type="soxr_vhq")
    return a + octave_up

[gitmylo]

I'll try it out, i'll most likely add it in utils though, not directly in audioldm here.

[pravdomil]

put it also into audioldm, it is a must have, otherwise the quality is not good

[gitmylo]

I'm not sure how that code would enhance the audio, instead, it will just give it a higher pitch it seems. I'll test a bit further

[gitmylo]

Maybe that's just something with trying to use it separately.

Nevermind, i think i'm just missing something.

[gitmylo]

Oh yeah, it can definitely sound better, not always though.

Prompt: Chopping carrots on a wooden plank
seed: 2919297160245668629
other options default.

original.mp4

enhanced.mp4

[pravdomil]

that sounds great, much more cleaner