Didn't hitting the red X prompt for a save action?
This is what I mean by not saving on exit. I would close the window, and it held on to the changes internally, but didn't apply them or prompt to save the file.
Yes, I noticed that's a few years old but it solved the problem for me today. That's why I made this post today.
I thought it might have affected other people because I would swear I've never seen that settings page before, let alone messed with it. I definitely wouldn't have connected the setting with the effect it had.