Tbh, I also quite liked your idea of separate commands in the right-click context menu ('remove from library' vs 'delete from computer') and mapping these actions to hotkeys could also be good. But I do understand the inclination not to increase the number of options any further here.
In a way I understand that, but I think there is also an argument to be made that it would simplify things.
Currently the 'delete' entry in the menu is ambiguous.
And when you click it, it will always open a pop-up panel that you (kind of) need to read to clarify the ambiguity, and then you will need to click a second time.
With my proposal you would only need to click once.
To me, that seems both: simpler, faster, and clearer.