Delete whatever folder is listed for your MusicBee library (next to the library name in File->Library->Open Library).
Obviously, make sure that this is also not the directory where your music files are or it will delete them too.
Then you can make a new library.
If you want to delete your cached information like artist bios, photos, miscellaneous plugins, etc then I think it depends on which version (desktop vs windows store) you have. In my desktop version experience, those files are either in:
Users\{your user name}\AppData\Local\MusicBee
or
Users\{your user name}\AppData\Roaming\MusicBee
or both