Welcome to the forum, Ikarus.
I believe it should already be doing that. If it isn't, it might be a refreshment issue.
So try refreshing that tab (press F5) to see whether or not it'll load album covers for the missing artist thumbnails.
If that doesn't do it, first locate these artists in the Music Explorer tab, and then go back to refresh your original tab again.
I already spend hours on end on social media. Might as well spare a few of those to a greater purpose here.