If the bitrate of each FLAC file is consistently less than 320 kbps for the entire audio stream, the overall file size will be smaller than a file with a CBR of 320 kbps.
The file size is just the bitrate multiplied by the duration in seconds*.
Whether it's lossless or lossy doesn't matter - the only reason a typical FLAC file is much larger isn't because it's lossless, it's because it requires a much higher bitrate in order to be lossless.
*To be clear, I should have said may make a larger file - because FLAC uses a variable bit rate you can't really do a direct comparison to a CBR MP3 (or CBR aac, opus or anything else). It would depend on the file.
However, it sounds like you have some outliers that are either:
a) very receptive to FLAC compression, or
b) reporting their bitrate incorrectly.
You could easily test this though by converting one to mp3 and comparing the sizes.
I'm also curious because all of my FLAC files show bitrates closer to 1000 kbps, which is more typical.
Can you post some example file sizes and durations of the ones that are reporting their bitrate as less than 320?
Bee excellent to each other...