Auto DJ doesn't seem to be following this setting. I went through and marked a bunch of classical stuff with the setting, and now it seems to grab the first track of a set, but forget to add the rest of them.
I thought it might be related to my Auto DJ filter, which is Play Count is < 1. Some of the sets got played out of order and so they have mismatched play counts. But the current group was all 0 play counts (I added the missing tracks when the first one started).