Every album re-copied on each update about beets-alternatives

[geigerzaehler]

Thanks for the update. I think I’ve found the problem and it has to do with modification times on FAT32 only having second resolution. I’ve pushed a patch and you can try it out to see if that fixes your issue:

pip install --user git+https://github.com/geigerzaehler/beets-alternatives.git@fat32-mtime

[geigerzaehler]

Hi @loafofpiecrust, thanks for reporting this. The config looks good to me and the second update should not try to copy the files again. You’re probably right that one of the plugins (or maybe some custom configuration) is causing the problem.

Could you try to isolate the issue by disabling plugins and removing custom configuration? Then I can try to identify the problem with a simpler configuration.

[loafofpiecrust]

Thanks for the quick follow-up! I removed most of my plugins, reduced the config a little. I tested on my local BTRFS filesystem and things worked as expected. Then I tested again with my iPod which has a FAT32 filesystem and the issue recurred, but only with certain files. At first I added some FLACs and it worked fine. Then I added a few more albums and the issue happened with those files (MP3 and FLAC). Must be related to FAT32 filesystem.

[loafofpiecrust]

Okay sorry I forgot something. It seems like I made a mistake. The issue is that when I set my onplayer=true tag I used -W to avoid writing the tag to disk. This is what caused those specific files to be copied over during every single update. I just tested removing that tag entirely, then re-writing it without -W and I got the right behavior with the same files. I didn't want this tag in my files, but so it goes! This might be worth adding to the docs for this plugin though. Interestingly, using -W was still only an issue on FAT32.

EDIT: I've since tested with my whole library again after letting it write onplayer to the files, and still running into the same issue.

[geigerzaehler]

@loafofpiecrust, did you have a chance to try out my branch and see if it fixes the problem?

[loafofpiecrust]

I figured out how to build your branch to include it in my beets build, and tested it out. It does not appear to fix my issue, and it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod. The destination files have a brand new mtime way later than the source mtime, yet it keeps copying them over. It must have something to do with the destination path itself?

[loafofpiecrust]

@geigerzaehler There doesn't seem to be a pattern to which files are getting copied every time. Some are whole albums, some individual tracks. Perhaps more non-alphanumeric characters? I'm not sure. I do have several extra replace entries in my config to replace problematic characters... Including a few here in case that's helpful.

+/run/media/snead/SHELBY_S IP/Music/4 Non Blondes/[1992] Bigger, Better, Faster, More!/03. What's Up¿.flac
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2007] Collaborations (Comp)/01. Ayo Technology.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/01. That's What Love Can Do.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/03. Beautiful (Valentin club mix).mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/05. Fighter (Hellraiser remix).mp3
+/run/media/snead/SHELBY_S IP/Music/Anamanaguchi/[2006] Power Supply (EP)/06. Flora⁄Fauna.mp3
+/run/media/snead/SHELBY_S IP/Music/Anderson .Paak/[2016] Malibu/04. The Season ⁄ Carry Me.mp3
+/run/media/snead/SHELBY_S IP/Music/Nujabes/[2003][Various Artists] Hydeout Productions; First Collection (Comp)/03. Strive.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/01. The Narrator.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/02. Tidal Wave.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/03. High Tide.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/04. Green Machine.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/05. Winter Must Be Cold.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/06. She's Just Like Me ⁄ Taking Time.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/07. Glowworm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/08. Dots 1-2-3.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/09. Lucky Charm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/10. Innerspace.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/11. Show the World.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/12. Love You Alice ⁄ D.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/13. Pine Away.mp3
+/run/media/snead/SHELBY_S IP/Music/Arcade Fire/[2007] Neon Bible/05. Black Wave⁄Bad Vibrations.flac
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/01. Do I Wanna Know¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/02. R U Mine¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/09. Why'd You Only Call Me When You're High¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Grizzly Bear/[2007] Friend (EP)/09. Knife.mp3

[loafofpiecrust]

I had to make a few more changes to use existing untracked destination files imported by my rsync script on this branch (EDIT: I also did remove art embedding since I use cover files). It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary. Much more robust to manual renames/moves. As an aside, it does seem quite slow to finish when there are zero updates needed (4 minutes on my library!).

[geigerzaehler]

it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod.

Yes, you’re right! I didn’t realize that the problem was that files were added.

It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary.

I took a look at your branch and I couldn’t quite figure out what the code you changed does. But to me it looks like it breaks things because we’re effectively doing self.set_path(item, self.destination(item)) which mixes library paths with paths in the alt collection.

I think it works for you at the moment because your basically skipping adding a file when the path is not found which seems to suggest that the paths don’t actually exist or os.path.lexists(syspath(path)) behaves strangely on fat32.

One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.

[wisp3rwind]

I haven't been following this issue up to now; and I'm not sure that I understand correctly which steps are required to trigger the extraneous ADD actions. Maybe you could clarify in terms of the actual beet commands? From what I gather, it should be something like:

• configure an alternative ipod on a FAT32 filesystem (required, other filesystems don't exhibit the issue) which includes tracks according to the onplayer:true query
• beet modify -W <some query> onplayer=true (is the -W required? you used it initially, but in the end, the issue occurred without, right?)
• beet alt update ipod
• beet import <some new music>
• beet modify -W <query for new music> onplayer=true
• beet alt update ipod (should add only the new music, but in fact also adds the old tracks again)

Some questions to ponder:

• Does the same issue occur on a plain FAT32 thumb drive (or a FAT32 loop device) (i.e. is there a chance that the iPod does something weird with mtimes or file name normalization? Although mtimes shouldn't be the issue, since they would lead to WRITEs.)
• Could we add a FAT32 test case to CI by creating a FAT32 partition on a loop device and running the tests there?
• Could you repeat the steps and report the files' mtimes after each of them?
• As @geigerzaehler said:

  One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.