Backups Not Incremental, partially

[TateB]

Hope somebody can help me through this one, I'm a bit confused!

I am using v. 3.6.8999.

My library is located on my C drive.  About a year ago, I moved the music files over to an external drive (F), relinked the file paths, worked fine.  I started building a second folder of music files, again on my C drive.  Everything was working fine, at this point, including backups (I backed up the external drive to another external drive, and the new folder on my C drive to a different external drive.

Recently, I endeavoured to do the same process, as this second folder on my C drive began posing space problems.  In this case, I mapped the appropriate backup drive to H, relinked file paths, and established it as the second dedicated external drive.  Again, everything appeared to work just fine.  All data, tags, etc. were relinked seamlessly.

I then connected a new external drive, to backup H.  It appeared to correctly back things up (around 500,000 files, ~1.5Tb).  Nearly all folders appeared to transfer over, the file sizes and quantities were close (not exact, oddly).  This is where I hit the problem.  When I tried to back it up a subsequent time very shortly thereafter, it tried to copy roughly 50% of the total files over entirely, rather than only those that have changed.  I closed the program, tried again, same thing.  I then let it back up those 50% of the files.  It completed successfully.  I ran the backup again to see if that fixed the problem, and it did not.  Once again, it started trying to copy over 50% of the files, yet again.

Any ideas?  I'm a bit stumped why this is happening, but perhaps I don't understand how it evaluates for an incremental backup, and there is a setting somewhere that I need to edit.

[boroda]

check MB prefs for auto-saving of online lyrics and artworks. also, manually check the sizes and modification timestamps of some of re-backed up tracks.

[TateB]

check MB prefs for auto-saving of online lyrics and artworks. also, manually check the sizes and modification timestamps of some of re-backed up tracks.

Thank you for your help.

Lyrics are embedded in the music file.
Artist pictures and soundtrack pictures are saved in subfolders of MusicBee (same location as my Library file).
Album artwork is in the same folder as the music file(s) to which it is linked, or otherwise embedded.

I compared 7 different files at random.  In all cases, the file size and date modified values matched.

[sveakul]

I then connected a new external drive, to backup H.  It appeared to correctly back things up (around 500,000 files, ~1.5Tb).  Nearly all folders appeared to transfer over, the file sizes and quantities were close (not exact, oddly).  This is where I hit the problem.  When I tried to back it up a subsequent time very shortly thereafter, it tried to copy roughly 50% of the total files over entirely, rather than only those that have changed.  I closed the program, tried again, same thing.  I then let it back up those 50% of the files.  It completed successfully.  I ran the backup again to see if that fixed the problem, and it did not.  Once again, it started trying to copy over 50% of the files, yet again.

Any ideas?  I'm a bit stumped why this is happening, but perhaps I don't understand how it evaluates for an incremental backup, and there is a setting somewhere that I need to edit.

What do YOU mean by "incremental backup"?  Are you using some kind of backup/mirror application like SyncBackFree?  You say "it" appeared to correctly back things up, what is "it?"  If you were just transferring a huge amount of music files over to a new data drive, why not just do that in Windows, and then if they are part of your MB library use Relink Music File Paths after the operation to reflect the new drive path?

[TateB]

I then connected a new external drive, to backup H.  It appeared to correctly back things up (around 500,000 files, ~1.5Tb).  Nearly all folders appeared to transfer over, the file sizes and quantities were close (not exact, oddly).  This is where I hit the problem.  When I tried to back it up a subsequent time very shortly thereafter, it tried to copy roughly 50% of the total files over entirely, rather than only those that have changed.  I closed the program, tried again, same thing.  I then let it back up those 50% of the files.  It completed successfully.  I ran the backup again to see if that fixed the problem, and it did not.  Once again, it started trying to copy over 50% of the files, yet again.

Any ideas?  I'm a bit stumped why this is happening, but perhaps I don't understand how it evaluates for an incremental backup, and there is a setting somewhere that I need to edit.

What do YOU mean by "incremental backup"?  Are you using some kind of backup/mirror application like SyncBackFree?  You say "it" appeared to correctly back things up, what is "it?"  If you were just transferring a huge amount of music files over to a new data drive, why not just do that in Windows, and then if they are part of your MB library use Relink Music File Paths after the operation to reflect the new drive path?

When MusicBee's native backup function works, it performs an incremental backup by identifying changed files, and only backing up those.  That isn't happening, at least partially, for some reason I don't understand.  I'm not using any other back-up/mirror/synchronization app.  I'm using the same process that I've used in the past with success.

I essentially did what you have suggested with relinking music files, when I moved them over to the external drive.  What isn't working properly is the backing up of this external drive (H) to the new backup drive.  It appears to back up properly, but it doesn't appear to be properly identifying all tracks that are unchanged, which shouldn't be transferred over again, if the incremental backup is working like it should (and like it does normally for me).  Hence my confusion.

My knowledge of relinking music paths may not be 100%.  If I relinked music paths between my H drive and its back-up, would that screw things up or correct for the back-up paths?

[sveakul]

I think you're talking about the File->Library->Backup Library tool? Unless you have automated that with a scheduler plugin, that's done manually on-demand only.  I suppose you could add music file folders to the backup choices, but how it would choose to "delete files from the backup folder no longer present in the source folder"--I assume by filename--may be the real issue here.

MusicBee is still full of things I'm unaware of, and one of those is your statement that "When MusicBee's native backup function works, it performs an incremental backup by identifying changed files, and only backing up those"--if someone can identify for me exactly where that function is setup, I'd be happy to take that off my list of unknowns.  TateB I'm not trying to be difficult, I just don't see MusicBee having some kind of "SyncBack" built in to it and not having been discussed here before.

[phred]

I think you're talking about the File->Library->Backup Library tool?

This tool copies everything. This is no incremental backup option. It overwrites what's already in the backup destination and adds files that did not exist. I don't it removes files that only exists at the destination and not at the source. Which would be more like SyncBack.

So in short, there's no such thing as an incremental backup when using MB's Backup Library option.

[TateB]

I think you're talking about the File->Library->Backup Library tool?

This tool copies everything. This is no incremental backup option. It overwrites what's already in the backup destination and adds files that did not exist. I don't it removes files that only exists at the destination and not at the source. Which would be more like SyncBack.

So in short, there's no such thing as an incremental backup when using MB's Backup Library option.

It does do exactly that, in my experience of using the program for 9 years.

If I change 10 files and backup my music folder using MusicBee's native function, it will backup the settings folder, any other folders that I have included, and delete/copy only those 10 music files.  My issue is that this is not happening anymore, only in the particular context described above.  Even if no music files are changed, it is backing up 220,000 files in a folder with approx. 500,000 music files, every time that I activate the backup function.

In other words, this is not a typical program behavior.  I'm asking for help as I cannot figure out what is causing the behavior to be atypical in contrast to its normal functioning.  This isn't a situation where I'm trying to perform a novel task and trying to make the program do something that it does not have the capability to do.

[boroda]

i think only Steven can clarify how MB backup function works for music files. the exact behavior is not documented, and most MB users don't use it.

i'm using Syncovery to back up music files.

[phred]

Even if no music files are changed, it is backing up 220,000 files in a folder with approx. 500,000 music files, every time that I activate the backup function.

MB's definition of "library" is the database and related files necessary for MB to run. It does -NOT- include music files. If you added music files as an additional source to be backed up, you shouldn't. I would suggest SyncBack (free or paid) which can do incremental backups and backups of only the files which have changed, added, or deleted. Double-check if the free version does all those things. I use the paid version and it does. The paid version also has the option to schedule the backup. Perhaps the free version does also. If it does, set the backup to run overnight when you're not using MB.

[sveakul]

MB's definition of "library" is the database and related files necessary for MB to run. It does -NOT- include music files. If you added music files as an additional source to be backed up, you shouldn't. I would suggest SyncBack (free or paid) which can do incremental backups and backups of only the files which have changed, added, or deleted. Double-check if the free version does all those things. I use the paid version and it does. The paid version also has the option to schedule the backup. Perhaps the free version does also. If it does, set the backup to run overnight when you're not using MB.

Yes the free version of SyncBack does all those things, including scheduling.  The portable free version is available as 32-bit only, from the website here , the free installer version is 64-bit and available here .

Yes I figured the OP was misunderstanding what and how the Library backup tool actually backs up, but hopefully Steven can give us all the last word.  It is obviously the source of the original issue here one way or the other.

[TateB]

Thanks all for your responses, and suggestion of SyncBack.  I will give this app a shot, if it remedies my issues, then I'll continue with that.

That said, I do understand what an incremental backup is, and I have been using MusicBee's native backup function for years for exactly that purpose, successfully, but for the issue in my OP.  Perhaps it wasn't designed for that purpose (maybe it was? nobody seems sure), but it did perform that function notwithstanding design intent, so perhaps it was just a happy coincidence.

[Steven]

to answer the question of what MB does, the criteria to re-copy an existing file is when the windows file modification time of a source file is greater than the file modification time of the target file then it gets re-copied. Missing files in the source/destination folder as deleted/added to the destination folder as appropriate.

[tjinc]

I can't say that I have ever used this function as I have my own backup regimen, but it is intriguing.

So Steven's response confirms TateB's claim that an incremental backup is/was/should be performed here. You can also set it up to back up your music files easily enough.


Setting up a backup folder on my C:\ drive and testing myself, this incremental backup is what I am seeing. I do not see any issue with copying files that have not changed.

So what might be the issue? TateB - question for you:
  What is the file system of your backup drive (NTFS or FAT32)?
 
Setting up a backup folder on a local USB drive, I see the exact same issue TateB describes. When using WinMerge (folder compare function set to 'Modified Date') I can see that the modification dates of the backup files are up to two seconds later than the source files. I believe that this is caused by the transfer of files between NTFS and FAT32 formats - the FAT32 system rounds the timestamp up to the nearest even number of seconds.
When formatting this drive as NTFS everything works as expected.

Still not sure about this as the backup file modification date is always later (by up to two seconds) and to quote Steven:

the criteria to re-copy an existing file is when the windows file modification time of a source file is greater than the file modification time of the target file then it gets re-copied

[TateB]

I can't say that I have ever used this function as I have my own backup regimen, but it is intriguing.

So Steven's response confirms TateB's claim that an incremental backup is/was/should be performed here. You can also set it up to back up your music files easily enough.


Setting up a backup folder on my C:\ drive and testing myself, this incremental backup is what I am seeing. I do not see any issue with copying files that have not changed.

So what might be the issue? TateB - question for you:
  What is the file system of your backup drive (NTFS or FAT32)?
 
Setting up a backup folder on a local USB drive, I see the exact same issue TateB describes. When using WinMerge (folder compare function set to 'Modified Date') I can see that the modification dates of the backup files are up to two seconds later than the source files. I believe that this is caused by the transfer of files between NTFS and FAT32 formats - the FAT32 system rounds the timestamp up to the nearest even number of seconds.
When formatting this drive as NTFS everything works as expected.

Still not sure about this as the backup file modification date is always later (by up to two seconds) and to quote Steven:

the criteria to re-copy an existing file is when the windows file modification time of a source file is greater than the file modification time of the target file then it gets re-copied

Thank you to Steven for the clarification.

tjinc - my external drive with the music files in use is NTFS.  The external drive that I am using for backup purposes is exFAT.  What you are describing may well be the issue - I never would have figured that one out!  I've since moved over to SyncBack as recommended above.  One thing about SyncBack, under its "Compare Options," that fits with what you are saying, and may be a potential solution for this issue:  SyncBack has an option to "Ignore time & date changes of XX seconds or less" that is set to a 2 second default.