For my understanding, what is it that you want to do exactly that is slowed down with this new behaviour?
If you click a letter it will still immediately take you there?
For as far as I can see it is only slowing you down when you click a wrong letter, and for the rare cases that happens 3 sec of 'punishment' doesn't seem a big deal to me?
It's the equivalent of flipping through records in a crate, but after each "flip", you're told "now wait 3 seconds". I clearly use this in a different way to you, but then to me, the A-Z list is a quick way to get to all albums from artists that begin with N, for example, rather than being a way to type out "NIRV" because you're looking for Nirvana album in a way that's not as efficient as just using a keyboard and the existing search box.
But with the changes made by Steven in the latest version, we're both fine, as if you right-click the A-Z list you can check the option to do multi-character searches, while I can leave it unchecked. (Thanks, Steven!)