Don't scan monitored folders for changes on startup

[Zak]

Maybe not the best etiquette, but I'm going to start by quoting myself from an earlier thread.
(Because it was a couple of years ago, and in response to a different issue that has apparently been resolved)

I've also been wondering for a long time about what exactly MusicBee does on startup.
In particular, I see (and hear) heavy disk use for the first five minutes or so.

I "only" have about 150,000 tracks on a local NAS hard drive.

I manually tag my files outside MusicBee before moving them to my main music folder which is set to be continuously monitored.


However, I don't need MusicBee to check anything when it starts up.

With a monitored folder specified, is MusicBee always scanning that folder on startup regardless of the monitored folders radio button selection?
If so, what does the on startup check for updated or missing files checkbox do differently?

The reply given there was:

monitored folders are checked for new files, so each sub-folder is scanned to detect a new file.

And again, from this now 7-year old post:

when "continuously monitor" is checked, as intended scan on startup is still performed eg. the user may have added files to the scan folder while MB is not running

It's been an annoyance for years now, so today I am requesting an option to allow library folders to be continuously monitored while MusicBee is running, but without scanning them for new files on startup.
The reason is simply that with a large library on a ye olde-fashioned mechanical drive, MusicBee is scanning folders for around ten minutes each time I open the program, resulting in sluggish behaviour in all programs until it finishes. It probably isn't helping the longevity of the drives either.

I know why it does the full scan but to put it another way: "I solemnly pledge not to move tracks into a monitored library folder when MusicBee is not running, or if I ever do, to manually add them to the library afterwards myself.".
(And in the case of that manual update, at least then I'm only scanning the handful of folders/tracks that I need to, rather than my entire Music folder.)

I'm not sure if this would be a new option or a tweak to the existing implementation - it already seems that having continuously monitored folders will contradict the state of the on startup check for updated or missing files option, in that even when unchecked that scan is still taking place.

[Steven]

this is done for v3.6

[Zak]

That's amazing. Thank you! 

[Zak]

== 6 Nov 2023 Update: See full details in post below ==
Thanks to some debugging, I found the reason for the excessive disk activity when scanning my Library.
tldr; If you have your music on a QNAP NAS with snapshots enabled, make sure you exclude the snapshot folder in your library settings.
== End Update ==

I've been using the new separate Continuously monitor option on its own in 3.6 and it's been working great.
Now MusicBee starts up really fast and doesn't slam my hard drive checking every single folder.

For the first time today I reenabled the startup checks to make sure what is in the Library matches the files on my hard drive.
I've been very diligent about only making updates when MusicBee is running, but have no doubt some exceptions will happen occasionally.



I thought these settings would replicate how MusicBee started before, but after running for about 90 minutes, the hard drive is still churning and MusicBee hasn't reported back to say it found any updated or deleted files. As soon as I quit MusicBee the disk reading stopped.

These separated settings are only in MusicBee 3.6 which isn't supported, so I won't be making a separate post in the Bug Report forum.
This post serves as a reminder to follow up when 3.6 is released and a heads up to anyone else using these settings this way that there may be an issue with how it has been implemented.

[Steven]

i dont think 'on startup check for updated or missing files' has been changed
it would be easy enough to log what its doing if you are happy to run a debug version

[Zak]

Okay, if you want to check I'd be happy to log it.

[Steven]

https://mega.nz/file/o4tR2KrR#XB1MVCbLdC9yyH_O486R5F7a4RvipdZLcOFlqqmbhFc
this will potentially create a large error log but just interested in the last entries you the process has taken too long

[Zak]

Thank you for that.
I anticipated it would take a while, so I started MusicBee when I went to bed and left it running until completion, almost... er... six hours later.  

The error log was enough for me to find the cause of the problem, and happily it isn't MusicBee.
I'll edit my previous post to remove any potential future confusion.

THE SHORT ANSWER
The lengthy scanning time was caused by my Excluded folders somehow getting removed from the Library settings.
(I don't recall changing these recently, but can't make MusicBee replicate it now, so will assume I did it when testing something)
Because my music files are stored on a QNAP NAS, this meant MusicBee was also scanning an extra 150,000+ files from the Snapshot folder it creates for backups.

Excluding the Snapshot directory again means MusicBee starts up and completes a full library scan (i.e. with the settings in my previous screenshot) in about five minutes.

THE LONG ANSWER
One has to ask: Why does scanning the additional files in the QNAP Snapshot folder take so much longer?

This is purely speculation, but I think it has to do with how QNAP creates snapshots that allow altered or deleted files to be restored from an earlier version. Their documentation is a bit vague, but does mention that instead of just creating whole copies of every file - which obviously doubles the disk space requirement - the snapshot only stores what data has changed from the current version. It does this at the block level. While it is possible to browse the snapshot files via Windows File Explorer - and copy them or open them - I suspect that when doing so, the NAS is essentially reading both the original file data and the snapshot data to present Windows with what appears to be a normal file. This obviously incurs an overhead on each file. Similarly, when MusicBee scans the Snapshot folder files 100,000 times, the NAS has to internally recreate each one to present it as a "normal" Windows file entry for MusicBee to read.

Even just looking at one random album in the debug log shows the difference. I picked the soundtrack to the film The Abyss.
The log file doesn't measure milliseconds, but it shows the original library files were all processed in less than one second:

6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 5:05:54 AM - load file M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac
6/11/2023 5:05:54 AM - link PDFs M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac
6/11/2023 5:05:54 AM - link artwork M:\Music\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac

The same files in the Snapshot folder took at least three seconds:

6/11/2023 8:12:41 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 8:12:42 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 8:12:42 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\01 - Main Title.flac
6/11/2023 8:12:42 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 8:12:42 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 8:12:42 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\02 - Search the Montana.flac
6/11/2023 8:12:42 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 8:12:42 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 8:12:42 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\03 - The Crane.flac
6/11/2023 8:12:42 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 8:12:42 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 8:12:42 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\04 - The Manta Ship.flac
6/11/2023 8:12:42 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 8:12:42 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 8:12:42 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\05 - The Pseudopod.flac
6/11/2023 8:12:42 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 8:12:43 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 8:12:43 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\06 - The Fight.flac
6/11/2023 8:12:43 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 8:12:43 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 8:12:43 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\07 - Sub Battle.flac
6/11/2023 8:12:43 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 8:12:43 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 8:12:43 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\08 - Lindsey Drowns.flac
6/11/2023 8:12:43 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 8:12:43 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 8:12:43 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\09 - Resurrection.flac
6/11/2023 8:12:43 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 8:12:44 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 8:12:44 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\10 - Bud's Big Dive.flac
6/11/2023 8:12:44 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 8:12:44 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 8:12:44 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\11 - Bud on the Ledge.flac
6/11/2023 8:12:44 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 8:12:44 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 8:12:44 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\12 - Back on the Air.flac
6/11/2023 8:12:44 AM - load file M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac
6/11/2023 8:12:44 AM - link PDFs M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac
6/11/2023 8:12:44 AM - link artwork M:\Music\@Recently-Snapshot\GMT+10_2023-10-31_0600\[Soundtrack]\Movie\A-D\Abyss, The [1989]\13 - Finale.flac

Now multiply that by 150,000...  


OTHER (off-topic) OBSERVATIONS

Log times
It might just be the way you're logging it, but when running the Debug version, the first two entries are five minutes apart:

6/11/2023 3:25:53 PM - enumerate M:\Music\
6/11/2023 3:31:22 PM - get cached files

It looks like there's something happening for five minutes before MusicBee starts scanning the tracks?

Unnecessary folder processing
Most of the initial folder enumeration entries refer to folders that I no longer use. e.g.

6/11/2023 3:31:22 PM - enumerate M:\Test Music\, top only=True

This path exists only in the MusicBee database. I would have used it for testing something at least six months ago, but it no longer appears in any setting file and none of the tracks in that folder are in my Library. I tried the Compress Library command, but it's still included.

Finally, I am surprised that even after running the full startup scan many times now, MusicBee hasn't reported finding a single missing file. And yes, the option to prompt for confirmation is enabled.
Either I'm better at keeping my library organised than I thought, or MusicBee is doing a great job of keeping on top of all my random changes.  

Finally finally, I wonder if there's any value in having a "debug mode" built-in so people who know what to look for can get this kind of additional information without you have to compile a new version each time? Even if it needed to be enabled via an ini file edit - to avoid tire kicking -  I would find it useful sometimes.

Thanks again for your great support.

[Steven]

6/11/2023 3:25:53 PM - enumerate M:\Music\
6/11/2023 3:31:22 PM - get cached files

It looks like there's something happening for five minutes before MusicBee starts scanning the tracks?

is simply scanning the directories in M:\Music\ for a full list of filenames

Unnecessary folder processing
Most of the initial folder enumeration entries refer to folders that I no longer use. e.g.

6/11/2023 3:31:22 PM - enumerate M:\Test Music\, top only=True

This path exists only in the MusicBee database. I would have used it for testing something at least six months ago, but it no longer appears in any setting file and none of the tracks in that folder are in my Library. I tried the Compress Library command, but it's still included.

Compress Library wont remove a non-library file link if its referred to in a playlist or is in the inbox. I might change the file scanner behavior but some users operate on folders/ playlists only without the file being added to a library

[barles]

Hello, i have the same or similar problem, MB scans the 3 HDs where the music files are located, it only scans when MB is initiated for the first time after the computer has started or after a reboot. Once the first scan has been done i am able to close MB, start it again and no more scans are produced. In my case the scan takes ~15 minutes.

I'm using v 3.6.8705P (the debug version) on a portable folder, my error log doesn't seem to output anything relevant, this is from today:
22-Nov-23 07:51:18 - done (MB after the computer started, HD scan)
22-Nov-23 08:13:35 - done (MB after closing it and started again, no HD scan)
22-Nov-23 08:16:45 - done (same as the previous entry)

Image of Preferences/Library (i have all deactivated, no monitored locations and no scan on startup)
Files are manually added to the library using "Scan Folders for New Files" tool

[The Incredible Boom Boom]

Hello, i have the same or similar problem, MB scans the 3 HDs where the music files are located, it only scans when MB is initiated for the first time after the computer has started or after a reboot. Once the first scan has been done i am able to close MB, start it again and no more scans are produced. In my case the scan takes ~15 minutes.

I'm using v 3.6.8705P (the debug version) on a portable folder, my error log doesn't seem to output anything relevant, this is from today:
22-Nov-23 07:51:18 - done (MB after the computer started, HD scan)
22-Nov-23 08:13:35 - done (MB after closing it and started again, no HD scan)
22-Nov-23 08:16:45 - done (same as the previous entry)

Image of Preferences/Library (i have all deactivated, no monitored locations and no scan on startup)
Files are manually added to the library using "Scan Folders for New Files" tool

To confirm, you're saying MusicBee is initiating a startup scan even when you do not have the setting enabled?