its supposed to do that already, so I guess there must be a route the files are being organised but not invoking the playlist cleanup.
Do you see a ! mark, blank line and a message when hovering over the ! saying the path does not exist, where its still saying the old path?
For these cases, do you recall any pattern eg. are they all files you had in the playlist while the file was in the Inbox, and the playlist didnt get updated when dragging to your auto-organised library?