OK, I did some more testing and I think I have pinned the cause of the problem. MusicBee file naming was indeed fixed at one point, as I was able to find from other forum threads. However it is possible to still get an error while ripping or during file conversion, as it does happen with both the same way.
1. If the title of a track is very long, say 400 letters, MusicBee will rename the end with some kind of renaming convention and the encoding is successful.
2. The error occurred when I tried to convert a file that has a title with 197 characters including spaces. Could it be that this happens because the title by itself does not exceed the maximum characters allowed, so MusicBee does not use the renaming convention for files that are too long, but then when it tries to add it to a folder, the total path length then exceeds the total number of characters allowed?
I believe this should be reported as a Bug, if a moderator reads this, can you please move the topic to the Bugs section, if you find it appropriate. Thanks