Robustness issues
Thank you for your work! I've read through your SpeechJudge paper and found it very informative.
After using this model I have noticed that some of its quality scores don't make much sense. As an example, here are some results I got from modifying the example wav_a.wav in the Github repository using the provided sample code:
Baseline (no changes): 4.53
+ Reverse original audio: 25.38
+ Split audio into 0.25s chunks, shuffle them, then splice them together: 36.25
+ Replace original text with an unrelated transcript: 22.75
In all of these cases, SpeechJudge strongly preferred the modified audio-text pairs over the baseline, even though they are clearly worse than the original. In practice, I found that the SpeechJudge tends to prefer gibberish audio with no relation to the text in general. Are you able to reproduce these results, or is there something wrong with my setup?