I did have that setting ticked, but it shouldn't have mattered as my files are organized using: <Album Artist>\<Album>\<Disc-Track#> <Title> and all of those tags were present.
Now I unchecked the setting (just to be sure), moved an affected album back to my inbox and then back to my library. Didn't work though. For now I've manually moved the files, but I'd rather not have to do that every time.
Side note: files do move from my Inbox folder when I add them to my library.