Welcome to the forum, vankyan,
It's been a few months, but I believe there were some changes made to how songs were added to a playlist.
Not sure if it's just in the wording, but I recall the <right click on track> feature wasn't called "Include In Playlist" at some point.
Baseless assumption, but maybe during those changes, there was a side-effect like the one you are experiencing now.
Anyway, what MusicBee version are you using?
I'm on a fairly old v3.5.8338P and when I try to include/tick a song to a static playlist where it already exists, I am only able to remove it from said playlist.
There is no pop-up saying "This song exists in the playlist, are you sure you want to add it again"