REFLOW-TTS: A RECTIFIED FLOW MODEL FOR HIGH-FIDELITY TEXT-TO-SPEECH


Abstract

The Diffusion models including Denoising Diffusion Probabilistic Models (DDPM) and score-based generative models have demonstrated excellent performance in speech synthesis tasks. However, its effectiveness comes at the cost of numerous sampling steps, resulting in prolonged sampling time required to synthesize high-quality speech. This drawback hinders its practical applicability in real-world scenarios. In this paper, we introduce ReFlow-TTS, a novel rectified flow based method for speech synthesis with high-fidelity. Specifically, our ReFlow-TTS is simply an Ordinary Differential Equation (ODE) model that transports Gaussian distribution to the ground-truth Mel-spectrogram distribution by straight line paths as much as possible. Furthermore, our proposed approach enables high-quality speech synthesis with a single sampling step and eliminates the need for training a teacher model. Our experiments on LJSpeech Dataset show that our ReFlow-TTS method achieves the best performance compared with other diffusion based models. And the ReFlow-TTS with one step sampling achieves competitive performance compared with existing one-step TTS models.

Text-to-Speech results

The below audio samples show the comparison between Recordings, FastSpeech 2, DiffGAN-TTS, ProDiff, DiffSpeech, GradTTS, CoMoSpeech (Teacher, 50 step), CoMoSpeech (Teacher, 1 step), CoMoSpeech (1 step) and our ReFlow-TTS (1step, 50step, RK45 solver) . Samples are selected from the test dataset of LJSpeech.



Sample 1: but the first bible actually dated which also was printed at maintz by peter schoeffer in the year fourteen sixty two

GT
FastSpeech 2
(NFE:1, RTF: 0.00293)		 DiffGAN-TTS
(NFE:4, RTF: 0.00856)		 ProDiff
(NFE:4, RTF:0.01276)
DiffSpeech
(NFE:71, RTF:0.3728)		 Diff-TTS
(NFE:1000, RTF:1.2639)		 Grad-TTS
(NFE:50, RTF:0.3185)
CoMoSpeech (Teacher, 50 step)
(NFE:50, RTF:0.2256)		 CoMoSpeech (Teacher, 1 step)
(NFE:1, RTF:0.00609)		 CoMoSpeech (1 step)
(NFE:1, RTF:0.00606)
ReFlow-TTS (1step)
(NFE:1, RTF:0.00577)		 ReFlow-TTS (50step)
(NFE:50, RTF:0.0964)		 ReFlow-TTS (RK45 solver)
(NFE:152, RTF:0.3703)


Sample 2: and was used there with very little variation all through the sixteenth and seventeenth centuries , and indeed into the eighteenth .

GT
FastSpeech 2
(NFE:1, RTF: 0.00293)		 DiffGAN-TTS
(NFE:4, RTF: 0.00856)		 ProDiff
(NFE:4, RTF:0.01276)
DiffSpeech
(NFE:71, RTF:0.3728)		 Diff-TTS
(NFE:1000, RTF:1.2639)		 Grad-TTS
(NFE:50, RTF:0.3185)
CoMoSpeech (Teacher, 50 step)
(NFE:50, RTF:0.2256)		 CoMoSpeech (Teacher, 1 step)
(NFE:1, RTF:0.00609)		 CoMoSpeech (1 step)
(NFE:1, RTF:0.00606)
ReFlow-TTS (1step)
(NFE:1, RTF:0.00577)		 ReFlow-TTS (50step)
(NFE:50, RTF:0.0964)		 ReFlow-TTS (RK45 solver)
(NFE:152, RTF:0.3703)


Sample 3: america has produced a good many showy books , the typography , paper , and illustrations of which are , however , all wrong ,

GT
FastSpeech 2
(NFE:1, RTF: 0.00293)		 DiffGAN-TTS
(NFE:4, RTF: 0.00856)		 ProDiff
(NFE:4, RTF:0.01276)
DiffSpeech
(NFE:71, RTF:0.3728)		 Diff-TTS
(NFE:1000, RTF:1.2639)		 Grad-TTS
(NFE:50, RTF:0.3185)
CoMoSpeech (Teacher, 50 step)
(NFE:50, RTF:0.2256)		 CoMoSpeech (Teacher, 1 step)
(NFE:1, RTF:0.00609)		 CoMoSpeech (1 step)
(NFE:1, RTF:0.00606)
ReFlow-TTS (1step)
(NFE:1, RTF:0.00577)		 ReFlow-TTS (50step)
(NFE:50, RTF:0.0964)		 ReFlow-TTS (RK45 solver)
(NFE:152, RTF:0.3703)

Text-to-speech results for ReFlow-TTS and 2-ReFlow-TTS

The below audio samples show the comparison between ReFlow-TTS (1step), ReFlow-TTS (50step), ReFlow-TTS (RK45 solver), 2-ReFlow-TTS (1step), 2-ReFlow-TTS (50step), 2-ReFlow-TTS (RK45 solver).



    Sample 1: but the first bible actually dated which also was printed at maintz by peter schoeffer in the year fourteen sixty two

    ReFlow-TTS (1step)
    (NFE:1, RTF:0.00577    		 ReFlow-TTS (50step)
    (NFE:1, RTF:0.0964    		 ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3703)
    2-ReFlow-TTS (1step)
    (NFE:1, RTF:0.00562    		 2-ReFlow-TTS (50step)
    (NFE:1, RTF:0.0886    		 2-ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3669)

    Sample 2: and was used there with very little variation all through the sixteenth and seventeenth centuries , and indeed into the eighteenth .

    ReFlow-TTS (1step)
    (NFE:1, RTF:0.00577    		 ReFlow-TTS (50step)
    (NFE:1, RTF:0.0964    		 ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3703)
    2-ReFlow-TTS (1step)
    (NFE:1, RTF:0.00562    		 2-ReFlow-TTS (50step)
    (NFE:1, RTF:0.0886    		 2-ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3669)

    Sample 3: america has produced a good many showy books , the typography , paper , and illustrations of which are , however , all wrong ,

    ReFlow-TTS (1step)
    (NFE:1, RTF:0.00577    		 ReFlow-TTS (50step)
    (NFE:1, RTF:0.0964    		 ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3703)
    2-ReFlow-TTS (1step)
    (NFE:1, RTF:0.00562    		 2-ReFlow-TTS (50step)
    (NFE:1, RTF:0.0886    		 2-ReFlow-TTS (RK45 solver)
    (NFE:1, RTF:0.3669)