rclone copy creating empty directories it didn't create before about rclone

[nielash]

Hi @sabitm, bisync maintainer here. bisync received a very large update in v1.66, and handling of sync conflicts is indeed one of the things that changed. So it may be that the new behavior you're seeing is intentional, but if there's a bug then I'd definitely like to know about it. Please feel free to open an issue or post in the forum with more details!

As for the original issue -- FYI there is an ongoing discussion about it in the forum:
https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/9

[sabitm]

Hi @nielash, thanks for maintaining bisync!

I'm not entirely sure how to debug though. It happens randomly enough. I'll try to find a way to reproduce it and will opening an issue if I found something interesting.

[ncw]

There are several threads about this on the forum

• https://forum.rclone.org/t/empty-dirs-not-wanted/45059/
• https://forum.rclone.org/t/how-to-ignore-empty-directories-when-uploading-from-windows/45057/

For rclone v1.66.1 I am going to make sure it obeys the --create-empty-src-dirs flag.

What the defaults for this for rclone sync, rclone copy and rclone move I have yet to decide. Currently they are all false. There is an argument for making this true for rclone sync but that might be a step too far for a point release.

[cau118]

I'm on macOS 14.3.1 running rclone v1.66.0 and I am experiencing the same issue. Folders are now being copied/synchronized, and at the destination, they are empty. They are empty because the folders do not contain files match the extension I have specified in the include flag on the rclone command line.

rclone v1.66.0

• os/version: darwin 14.3.1 (64 bit)
• os/kernel: 23.3.0 (arm64)
• os/type: darwin
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.22.1
• go/linking: dynamic
• go/tags: none

OS used: macOS 14.3.1.

Remote is SSH/SFTP.

Command line used:
rclone copy . pi:testfolder -P --include="*.txt" --dry-run -vv

Results of the command with dry-run:

2024/03/26 09:05:39 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "." "pi:testfolder" "-P" "--include=*.txt" "--dry-run" "-vv"]
2024/03/26 09:05:39 DEBUG : Creating backend with remote "."
2024/03/26 09:05:39 DEBUG : Using config file from "/Users/cmlkh/.config/rclone/rclone.conf"
2024/03/26 09:05:39 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/cmlkh/Downloads/testfolder"
2024/03/26 09:05:39 DEBUG : Creating backend with remote "pi:testfolder"
2024/03/26 09:05:39 DEBUG : sftp://[email protected]:22/testfolder: New connection 172.16.0.227:51097->172.16.0.12:22 to "SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u3"
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Shell type "unix" from config
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Relative path resolved to "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Using root directory "/home/cmlkh/testfolder"
2024/03/26 09:05:40 DEBUG : 13.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 19.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 DEBUG : 9.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 NOTICE: 9.txt: Skipped copy as --dry-run is set (size 1.022Ki)
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder01", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder02", newDst=<nil>
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped make directory as --dry-run is set
2024/03/26 09:05:40 DEBUG : Added delayed dir = "examplefolder03", newDst=<nil>
2024/03/26 09:05:40 DEBUG : scratch.txt: Need to transfer - File not found at Destination
2024/03/26 09:05:40 NOTICE: scratch.txt: Skipped copy as --dry-run is set (size 8.616Ki)
2024/03/26 09:05:40 NOTICE: 19.txt: Skipped copy as --dry-run is set (size 1.653Ki)
2024/03/26 09:05:40 NOTICE: 13.txt: Skipped copy as --dry-run is set (size 1.737Ki)
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Waiting for checks to finish
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Waiting for transfers to finish
2024/03/26 09:05:40 NOTICE: examplefolder03: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder01: Skipped set directory modification time as --dry-run is set
2024/03/26 09:05:40 NOTICE: examplefolder02: Skipped set directory modification time as --dry-run is set
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s
2024/03/26 09:05:40 NOTICE:
Transferred:   	   13.029 KiB / 13.029 KiB, 100%, 0 B/s, ETA -
Transferred:            4 / 4, 100%
Elapsed time:         0.2s

2024/03/26 09:05:40 DEBUG : 15 go routines active
2024/03/26 09:05:40 DEBUG : sftp://[email protected]:22/testfolder: Closing 1 unused connections
2024/03/26 09:01:30 DEBUG : 15 go routines active
2024/03/26 09:01:30 DEBUG : sftp://[email protected]:22/testfolder: Closing 1 unused connections

[sabitm]

I'm not 100% sure if this is related, but bisync also has a weird behavior on 1.66.0. It keep producing a file conflict (which I have to resync to resolve it) between local and remote storage. I just assume that is because of experimental nature of bisync, hence I'm not opening an issue and will happily use the previous version for time being.

[YukiUnHappy]

Thank you @ncw for confirming the bug. Additionally, I have a small issue that I won't open a separate issue for.
The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

root@localhost:~/TEST# tree
.
├── ABC
└── D EF

2 directories, 0 files
root@localhost:~/TEST# rclone copy D
D

If we use the old version of rclone to create a completion script and apply it, we can also complete paths normally in 1.66.

root@localhost:~/TEST# rclone-v1.65.2-linux-amd64/rclone completion bash
root@localhost:~/TEST# . /etc/bash_completion
root@localhost:~/TEST# rclone version
rclone v1.66.0
- os/version: debian 10.0 (64 bit)
- os/kernel: 5.14.9 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none
root@localhost:~/TEST# rclone copy D\ EF/

[ncw]

@YukiUnHappy

Additionally, I have a small issue that I won't open a separate issue for. The auto completion improvement in 1.66 has broken the ability to complete paths with spaces in bash.

I can confirm that. Can you open a new issue about this please? So we don't mix the two up - thank you. (BTW I think this is a bug in cobra and here is the fix spf13/cobra#2126 - Note also that if you type a " first eg "D<TAB> you'll get the correct completions.)

[ncw]

Here is a first attempt at a fix - this obeys --create-empty-src-dirs in the same way that 1.65.2 does

v1.67.0-beta.7854.8d02ad3ba.fix-7689-empty-dirs on branch fix-7689-empty-dirs (uploaded in 15-30 mins)

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing. Running with -golden sorted out the linux tests but not Windows or macOS. Can you help? Also any code review gratefully received :-)

[nielash]

@nielash I've been unable to get the bisync tests to pass on Windows or macOS. I have no idea why they are failing.

It looks to me like the issue is that Size() for the dir is returning an unexpected value. It is expecting 4Ki but getting 96 on mac_amd64:

2024/04/05 15:31:30 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:31:30 | @@ -139 +139 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)
2024/04/05 15:31:30 | @@ -141 +141 @@
2024/04/05 15:31:30 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:31:30 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 96)

and 0 on windows:

2024/04/05 15:30:53 | MISCOMPARE  -Golden vs +Results for  test.log
2024/04/05 15:30:53 | @@ -139 +139 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)
2024/04/05 15:30:53 | @@ -141 +141 @@
2024/04/05 15:30:53 | -NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 4Ki)
2024/04/05 15:30:53 | +NOTICE: subdir: Skipped update directory metadata as --dry-run is set (size 0)

This comment seems potentially relevant:

rclone/backend/local/local.go

Lines 1494 to 1497 in 05e5712

	// The value in info.Size() is not always correct
	// - Windows links read as 0 size
	// - Some virtual filesystems (such ash LucidLink) links read as 0 size
	// - Android - some versions the links are larger than readlink suggests

I can take a closer look later!

[nielash]

@ncw I'm doing the closer look at fix-7689-empty-dirs now and I think there's an issue with the --create-empty-src-dirs=false case. Here's a test that demonstrates it: nielash@16d856a

The same test would have passed previously.

[nielash]

@ncw added a few other inline comments:
dd463b2
13c72a9

and a test that (I think) catches another regression: nielash@cd0891d