I have requested such direct use of functions in auto-playlists & filters, but it's not implemented yet: https://getmusicbee.com/forum/index.php?topic=25062.0
For now you have to use a virtual tag in auto-playlists if you want to use functions.
For your case, create a virtual tag with the following code:
$IsNull(<Compound>,,$If(<Compound>=<Happyness>" - "<Rating>,,<Happyness>" - "<Rating>))
And create an auto-playlist with the rule that the virtual tag has a value. Then it will only list out-of-sync tracks (I assume both <Compound> and <Happyness> fields are custom tags, otherwise it won't work).