Let me start out by saying that I'm not entirely sure about this suggestion as it may get out of hand fairly quickly but I at least wanted to hear some opinions on it.
As for me, being swedish and having a lot of swedish artists in my library who's name starts with or contains the letters Å, Ä or Ö, I would find it useful to be able to access them using the A-Z jump bar. This would of course also be useful for anyone who listens to these artists even if they don't speak the language or if the song in question has english lyrics.
But as I said, Sweden is not the only country in the world that has additional letters in its alphabet so where would you draw the line, thoughts?