The default fade-out length feels sluggish to me. I often pause a song when I'm wearing closed headphones and I need to hear an outside noise. If I'm listening to loud music I can't hear the noise until the fade has ended, but since the fade-out duration is 370 ms at 44.1 kHz, it creates a significant delay.
On the other hand, if I disable "smooth fade when stopping", the song stops abruptly and this creates artefacts for bass-rich music. Since CDs normally don't have sound under 20 Hz, I believe a fade-out length around 50 ms would do a good job of avoiding artefacts while feeling much more responsive.
My recommendation would be to use this short fade-out when "smooth fade when stopping" is unchecked. This avoids adding another option to the UI, and I don't think people who already have the option unchecked would mind a very short fade-out instead of an abrupt stop.