syncany / syncany

Syncany is a cloud storage and filesharing application with a focus on security and abstraction of storage.

Syncany

The core team of Syncany is on hiatus for an indefinite amount of time. Feel free to do with the code what the license allows and encourages, but please don't expect any maintenance. The team thanks everyone who has contributed to Syncany in one way or another.

Syncany is an open-source cloud storage and filesharing application. It allows users to backup and share certain folders of their workstations using any kind of storage, e.g. FTP, SFTP, WebDAV, Amazon S3 and Samba.

While the basic idea is similar to Dropbox, Syncany is open-source and additionally provides data encryption and more flexibility in terms of storage type and provider:

• Data encryption: Syncany encrypts the files locally, so that any online storage can be used even for sensitive data.
• Any storage: Syncany uses a plug-in based storage system. It can be used with any type of remote storage.

Installation

You can download the current binary packages and installers from the releases page, or from the Syncany download site. For details on how to install Syncany, check out the download/install instructions.

Please be aware that this is still ALPHA code! Do not use it for important files.

Getting Started

To learn how to use Syncany, check out the brand new and awesome Syncany User Guide. It'll tell you everything you want to know. For a step-by-step guide, navigate to the Getting Started page!

Building

Excited? Want to help? Or just build it yourself? For information about building, development, documentation, screencasts, diagrams and contributions, please check out the Syncany wiki page. It'll hopefully give you all the information you need!

Buy us a coffee

If you like what you see and you want to support us, you can buy us a coffee or a beer. There are maaanny ways to do so.

Break some hashes for us by donating some Bitcoins (also via tip4commit); or be a charmer and flattr us. If that's not for you, why not give us some change with PayPal? Any contributions are much appreciated!

Licensing, website and contact

Syncany is licensed under the GPLv3 open source license. It is actively developed by Philipp C. Heckel, the Syncany Team and many others. We are always looking for people to join or help out. Feel free to contact us:

• Syncany Website, the main website
• Syncany User Guide, most important resource for end users
• Syncany Wiki, most important resource for developers
• IRC channel #syncany on Freenode (active!, my nick is binwiederhier)
• @syncany on Twitter, somewhat quiet there, though ...
• Mailing list, still on Launchpad

syncany's Issues

Crypto: Implement user-concept & read/write access to repository

Right now, the repository access is not limited to specific users and does not allow adding read-only or read/write access. In the future, this would be desirable. Further concepts can be discussed here.

Timezone compatibility issues of tests with DST

Regression of #53

CLI: wrong password -> stacktrace; nice user feedback needed

When connecting to an existing repo, entering the wrong password leads to an ugly stack trace.

1. A nicer user feedback is needed
2. The code for the syncany:// link is placed in the ConnectCommand. It should be in the ConnectOperation, so that the GUI can use it as well.

---

With syncany:// link

$ sy connect syncany://storage/1/O2273fW6qk4YGbn/T1hCa3kuWbv64ybyzcUDrcE5OC/tLDsK0UDFMVlfqOphD+AezkAPdfYRiHFbwdnHnzyVkQ==-U3kCBQHMJxZZaSAk++M054oCAbUxm7KZ0JYlqA69x3KngE7KMhe6TFvyr40wgwACbKIly1vRouGrCUwCJDBDaacHMIhMqO0rIRPLL9TZO9WU9r3lNawvuy638wUQh6RdU0PKqw0ZP3jcJkL5sZUhdAafqGbQ66I8RD/GanQ7UhLkVg4VXlxiMcgiupaFT+72YM2k+aLXse+ONTo+ipDZ3FgL2BV2ZBE2I/1x/77NNZfsP44BR2cEgoQVueoGgHHM0Tw8ZtZ1WApk+2y70LTTsRpp18JVd7ZMSVM8NY+yOKvHVJwjZoiSJE5WMovrO3m....

Password: (ENTERED WRONG PASSWORD!)

Creating master key from password (this might take a while) ...
Exception in thread "main" java.io.IOException: java.lang.Exception: Integrity exception: Calculated HMAC and read HMAC do not match.
    at org.syncany.crypto.MultiCipherInputStream.readHeader(MultiCipherInputStream.java:83)
    at org.syncany.crypto.MultiCipherInputStream.read(MultiCipherInputStream.java:56)
    at org.syncany.crypto.CipherUtil.decrypt(CipherUtil.java:280)
    at org.syncany.cli.ConnectCommand.initPluginWithLink(ConnectCommand.java:194)
    at org.syncany.cli.ConnectCommand.createConnectionTO(ConnectCommand.java:114)
    at org.syncany.cli.ConnectCommand.parseConnectOptions(ConnectCommand.java:101)
    at org.syncany.cli.ConnectCommand.execute(ConnectCommand.java:70)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)
Caused by: java.lang.Exception: Integrity exception: Calculated HMAC and read HMAC do not match.
    at org.syncany.crypto.MultiCipherInputStream.readAndVerifyHmac(MultiCipherInputStream.java:144)
    at org.syncany.crypto.MultiCipherInputStream.readHeader(MultiCipherInputStream.java:80)
    ... 9 more

With interactive mode

$ sy connect
Choose a storage plugin. Available plugins are: ftp, local, webdav
Plugin: ftp

Connection details for FTP connection:
- Hostname: syncany.org
- Username: someuser
- Password (not displayed): 
- Path: repo1
- Port (optional, default is 21): 

Password: (WRONG PASSWORD ENTERED)

Creating master key from password (this might take a while) ...
Exception in thread "main" java.lang.Exception: Invalid password given, or repo file corrupt.
    at org.syncany.operations.ConnectOperation.decryptRepoFile(ConnectOperation.java:204)
    at org.syncany.operations.ConnectOperation.execute(ConnectOperation.java:111)
    at org.syncany.Client.connect(Client.java:185)
    at org.syncany.cli.ConnectCommand.execute(ConnectCommand.java:73)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)

Packaging: Create RPM package task for Fedora, etc.

We're looking for someone to create a *.rpm package for the Syncany command line client (syncany-cli) and packages for the plugins (including the GUI).

Old stuff:
The syncany-cli debian package is built by Effing package manager (FPM). For Syncany, I use the FPM plugin for Gradle. Also check out the detailed description about how Syncany uses FPM.

FPM can (among other things) also build *.rpm-packages, so it'd be really easy to do so. In fact, I started with a general deb/rpm packaging task.

The *.deb package works perfectly, I think, but the *.rpm is completely untested.

Create Linux man pages for CLI commands (text already available)

The CLI already has a few man-page oriented --help texts per command. It would be nice to have actual man pages -- ideally using the same texts or text blocks as the help pages.

Desired:

man sy
man syncany
man sy-init
man syncany-init
...

To do:

• Adjust the gradle debian packaging to include man pages
• Convert syncany-cli/src/main/resources/help/... texts to man page format and/or make --help understand basic man page format?!

First steps:
http://www.cyberciti.biz/faq/linux-unix-creating-a-manpage/

Cannot find jar file in app root /taratatta/taratatta

When I run the syncany command, I get this response: Cannot find jar file in app root /home/tamas/fejlesztés/Random GitHub/syncany

I've done the installation as described in the readme.

Create Ubuntu PPA for Syncany

At some point, a PPA needs to be set up; never done this before, so I'd very much appreciate instructions;

https://launchpad.net/~binwiederhier/+archive/syncany

Initializing WebDav repo fails because an existing repo is detected

ERROR: Cannot initialize repository (already exists).

Reported by beta in IRC, I have currently reproduced this and will be looking into it.

Q / FR: data distribution across storage types?

Hello!

I first heard about Syncany a few years ago and got the impression that the different storage types could be added to a "pool" of online/remote storage space across which Syncany would spread the data redundantly. Like Wuala used to do or https://github.com/tahoe-lafs/tahoe-lafs does. Is this in scope for Syncany as well? Or might it be?

Thanks and greetings!

sy init -- Error while uploading file to WebDAV (via https, livedrive.com)

Hi, I read in a Dec 20 commit that HTTPS WebDAV is not yet expected to be fully working. With latest git pull, HTTPS WebDAV does work for me with sd2.1und1.de, however raises the following error with livedrive.com when uploading a file during repo initialisation:

com.github.sardine.impl.SardineException: Unexpected response (500 Internal server error)

I thought I could help with the debug log for

sy --debug init --plugin webdav -P url=https://webdav.livedrive.com/syncanywebdav -Pusername=<username> -Ppassword='<password>' --no-encryption --no-gzip

which is here: http://pastebin.com/UBf6SuAY. Cheers!

integrity constraint violation: foreign key no action; SYS_FK_10133 table: FILECONTENT_CHUNK

Another SQL issue with the foreign keys.

sy cleanup -k 1
Exception in thread "main" java.lang.RuntimeException: java.sql.SQLIntegrityConstraintViolationException: integrity constraint violation: foreign key no action; SYS_FK_10133 table: FILECONTENT_CHUNK
    at org.syncany.database.dao.ChunkSqlDao.removeUnreferencedChunks(ChunkSqlDao.java:89)
    at org.syncany.database.SqlDatabase.removeUnreferencedChunks(SqlDatabase.java:254)
    at org.syncany.database.SqlDatabase.removeUnreferencedDatabaseEntities(SqlDatabase.java:91)
    at org.syncany.operations.CleanupOperation.removeOldVersions(CleanupOperation.java:202)
    at org.syncany.operations.CleanupOperation.execute(CleanupOperation.java:121)
    at org.syncany.Client.cleanup(Client.java:193)
    at org.syncany.cli.CleanupCommand.execute(CleanupCommand.java:38)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)
Caused by: java.sql.SQLIntegrityConstraintViolationException: integrity constraint violation: foreign key no action; SYS_FK_10133 table: FILECONTENT_CHUNK
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.execute(Unknown Source)
    at org.syncany.database.dao.ChunkSqlDao.removeUnreferencedChunks(ChunkSqlDao.java:85)
    ... 9 more
Caused by: org.hsqldb.HsqlException: integrity constraint violation: foreign key no action; SYS_FK_10133 table: FILECONTENT_CHUNK
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.StatementDML.performReferentialActions(Unknown Source)
    at org.hsqldb.StatementDML.delete(Unknown Source)
    at org.hsqldb.StatementDML.executeDeleteStatement(Unknown Source)
    at org.hsqldb.StatementDML.getResult(Unknown Source)
    at org.hsqldb.StatementDMQL.execute(Unknown Source)
    at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
    at org.hsqldb.Session.execute(Unknown Source)
    ... 12 more

Folder permission changes on Linux lead to "conflicted copy"

If an existing folder is encountered by the NewFileSystemAction and it does not match the expected POSIX permissions (DOS attributes also?), a "conflicted copy" folder is created. Instead, the existing folders permissions should be adjusted.

Example (as occurred in #58):

15-3-14 14:32:15.303 | DownOperation   | NotifyThre | FINE :    +  NewFileSystemAction [file1=null, file2=FileVersion [version=1, path=Untitled Folder2/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:30:19 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:44 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]]
15-3-14 14:32:15.303 | FileVersionComp | NotifyThre | INFO :      - [CHANGED_ATTRIBUTES]: Local file DIFFERS from file version, expected POSIX ATTRS = rwxrwxr-x, but actual POSIX ATTRS = rwxr-xr-x, for file Untitled Folder2/python_ctf_workshop
15-3-14 14:32:15.304 | FileSystemActio | NotifyThre | INFO :      - Local version conflicts, moving local file Untitled Folder2/python_ctf_workshop to Untitled Folder2/python_ctf_workshop (pheckel's conflicted copy, 15 Mar 14, 2-32 PM) ...
15-3-14 14:32:15.304 | FileSystemActio | NotifyThre | INFO :      - Creating folder at FileVersion [version=1, path=Untitled Folder2/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:30:19 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:44 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null] ...
15-3-14 14:32:15.304 | FileSystemActio | NotifyThre | INFO :      - Creating folder at /home/pheckel/SyncanyMitGregor/Untitled Folder2/python_ctf_workshop ...
15-3-14 14:32:15.304 | FileSystemActio | NotifyThre | INFO :      - Setting POSIX permissions: rwxrwxr-x ...

There is not yet a test to reproduce this behavior, neither is there a fix for this issue. Feel free to take/fix this issue if you have the time.

Python/C++/Objective-C native: File manager integration: Windows Explorer, Finder, Nautilus, Nemo

The little icons for the Windows Explorer, Nautilus and Finder need to be managed by separate native applications communicating with the Syncany daemon.

The liferay-nativity is a good starting point. It offers file manager integration on all platforms, but the code by is nowhere near a nicely integrable.

I'm looking for someone to implement this. Please let me know if you're interested.

'sy watch' misses remote and local events while sync is running

The WatchOperation misses events if the sync round is currently running. There is no buffer or local queue to record events and immediately start sync again after the sync round.

There is no test to reproduce this yet, neither do I exactly know what's going on. I just thought someone might be interested in analyzing this. Anyone?

Packaging: Create Mac OSX package task

We're looking for someone to build a Mac OSX package for the Syncany command line interface. Ideally it would also be usable for shipping a GUI later on, and could also be created on Linux (so that we can auto-build a new package).

The syncany-cli debian package is built by Effing package manager (FPM). For Syncany, I use the FPM plugin for Gradle. Also check out the detailed description about how Syncany uses FPM.

FPM can (among other things) also build Mac OSX packages, so it'd be really easy to do so. The *.deb package works perfectly, I think, but I haven't even started with the Mac OSX package. In fact, I have no way to test it, because I don't own a Mac.

I am looking for someone to configure and test the Mac OSX package for the Syncany command line client (syncany-cli).

Verification of checksums

Currently we have plenty of checksums (chunks, multichunks, filecontents). These are hardly ever verified anywhere. This should not be the case.

Upon downloading remote entities, their checksums should be verified. This affects (only?) the DownOperation.

No one is working on this currently, if you want to pick this up, comment here.

Partial initialization of repo

If initialization fails at uploading the master file, the local files remain. This is not expected behavior, because the repo initialization failed.

Steps to reproduce: Initialize an ftp connection, which should succeed, but fails for some other reason at uploading the master file. (e.g. insufficient permission to write).

Result: The local repo files exist.

I can reproduce this locally and will pick this up.

Android Native GUI

Hi, with my team we are evaluating existing projects (syncany, syncthing, seafile, clearskies), to find an opensource bitsync clone that fits our needs.

We need to be able to sync multiple dirs from different hosts using read or write permissions auth tokens that can be obtained from a central server or shared thought IM or links from your friends.
Does the multi-repos support (#32) feature fits our needs? Do you have any ETA for it?

We want to make it working on android platform (native or under chrooted linux) and we are willing to help your project by adding the native Android GUI able to work with your backend sync library.
Any suggestions are appreciated.

Refactor & test 'restore' command and operation

The sy restore command and operation (RestoreCommand and RestoreOperation) have no clear API and work not very intuitively. It is necessary to look at the command options and the actual logic.

Anyone interested?

Related: #86

Syncany website

The Syncany website needs a refresh. We need something simple and easily manageable -- most importantly however, something up-to-date.

Material / stuff to work with:

• Generated GitHub pages site: http://binwiederhier.github.io/syncany/
• Current/old website: https://github.com/syncany/syncany-website

Implement TransferManager.test() functionality

In the connect and init operation (as it is used by the GUI and the CLI), the storage must be probed / tested to figure out whether there already is a repository or not -- or if the connection fails completely.

I had something like this in mind:

public enum StorageTestResult {
   REPO_EXISTS, NO_REPO, ...?!
}

public StorageTestResult test() {
   // ...
}

I'm looking for someone to implement this. Please let me know if you're interested.

Atomic operations / transactions in TransferManagers

All operations that perform actions on the remote repository can be interrupted by connection issues. The TransferManagers are dumb in the sense that they cannot ensure transactions, idempotency or atomicity of any kind.

To avoid repository corruption, some sort of transaction concept for remote operations is needed.

Affected:

• UpOperation
• DownOperation
• CleanupOperation

Crypto: Review Syncany crypto concept (experts needed!)

I'm looking for someone to review the crypto code. Please let me know if you're interested. It has been briefly reviewed before on Stackoverflow, but we definitely need a thorough review

Related infographic:

The crypto functions are bundled in the Java package org.syncany.crypto. The two central classes in this package, MultiCipherOutputStream and MultiCipherInputStream implement Syncany's crypto file format. All other classes are more or less helper classes around these two stream-based entities:

• CipherSpec: A cipher spec defines and identifies cipher combination. It describes the cipher algorithm (e.g. AES/GCM/NoPadding), the key size (e.g. 128 bit), the block/IV size (e.g. 128 bit for AES), and whether the unlimited strength extension is needed.
• CipherSpecs: The CipherSpecs class holds a static list of supported CipherSpec objects and allows the MultiCipherInputStream to initialize a Cipher object based on the cipher spec ID in the header.
• CipherSession: The cipher session is used in MultiCipherOutputStream and MultiCipherInputStream for two reasons: (a) to reference the password, and (b) to maintain a per-session secret key cache (more below). If keys secret keys are invalid, it creates new keys using the CipherUtil class.
• MultiCipherOutputStream: This class implements the above described file format. It uses a CipherSession to retrieve/create a secret key and CipherUtil to create random IVs.
• MultiCipherInputStream: This class implements the corresponding input stream to the MultiCipherOutputStream. Like the output stream, it uses CipherSessions cache and CipherUtil to create a Cipher.
• CipherUtil: As the name suggests, the CipherUtil class implements a few static utility functions to create Cipher, SecretKey or a random array. It also provides methods to encrypt/decrypt byte arrays, as well as to programatically enabled unlimited strength.

Make chunking options configurable

Right now, the the "repo" file stores the chunking options (MultiChunker, Chunker and Transformer), but except for the transformer, none of the information is actually used. Instead, the standard chunking/multichunking mechanism is used, namely the FixedChunker and the ZipMultiChunker.

This configurability has to be implemented to allow using the MimeTypeChunker, the TttdChunker and possibly other MultiChunkers.

Points of reference:

• Init chunking and multichunking in the Config class
• Unused TTTDChunker
• Transfer object to store the chunking/multichunking/transformer options: RepoTO

I'm looking for someone to implement this. Please let me know if you're interested.

Deleting folder leads to ChangeFileSystemAction instead of DeleteFileSystemAction

Under some circumstances (always?), deleting a folder creates a ChangeFileSystemAction in the FileSystemActionReconciliator. Instead, a DeleteFileSystemAction should be created.

Example (as occurred in #58):

15-3-14 14:31:01.629 | FileSystemActio | NotifyThre | INFO :    + Comparing local version: FileVersion [version=1, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]
15-3-14 14:31:01.629 | FileSystemActio | NotifyThre | INFO :      with winning version   : FileVersion [version=2, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=DELETED, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]
15-3-14 14:31:01.629 | FileVersionComp | NotifyThre | INFO :      - [DELETED]: Local file DIFFERS from file version, expected EXISTS = true, but actual EXISTS = false, for file Untitled Folder/python_ctf_workshop
15-3-14 14:31:01.629 | FileSystemActio | NotifyThre | INFO :   + (14) Content changed: Local file differs from last version: local file = /home/pheckel/SyncanyMitGregor/Untitled Folder/python_ctf_workshop, local version = FileVersion [version=1, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null], winning version = FileVersion [version=2, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=DELETED, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]
15-3-14 14:31:01.629 | FileSystemActio | NotifyThre | INFO :     --> ChangeFileSystemAction [file1=FileVersion [version=1, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null], file2=FileVersion [version=2, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=DELETED, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]]
(...)
15-3-14 14:32:10.255 | FileSystemActio | NotifyThre | INFO :    + ChangeFileSystemAction [file1=FileVersion [version=1, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null], file2=FileVersion [version=2, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=DELETED, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]]
(...)
15-3-14 14:32:11.916 | DownOperation   | NotifyThre | FINE :    +  ChangeFileSystemAction [file1=FileVersion [version=1, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=NEW, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null], file2=FileVersion [version=2, path=Untitled Folder/python_ctf_workshop, type=FOLDER, status=DELETED, size=4096, lastModified=Sat Mar 15 14:19:29 CET 2014, linkTarget=null, checksum=null, updated=Sat Mar 15 14:30:10 CET 2014, posixPermissions=rwxrwxr-x, dosAttributes=null]]
15-3-14 14:32:11.916 | FileSystemActio | NotifyThre | INFO :      - (7) Original does NOT match, target file does not exist (and SHOUDN'T): Nothing to do!

There is not yet a test to reproduce this behavior, neither is there a fix for this issue. Feel free to take/fix this issue if you have the time.

Refactor & test 'log' command and operation

The sy log command and operation (LogCommand and LogOperation) have no clear API and work not very intuitively. It is necessary to look at the command options and the actual logic.

Anyone interested?

Related: #85

Some tests fail when if encryption enabled (with AllExceptLongRunningWithEncryption)

Reminder

SQL Error: integrity constraint violation: foreign key no parent; SYS_FK_10163 table: FILEVERSION

Error when inserting in database. Client A uploaded 16 database versions, client B downloaded them. Upon insertion in the database, the following exception occurred.

I have not found a test to reproduce this yet.
Full dump of the log here: http://pastebin.com/7Dnsizyv

6-3-14 21:40:32.521 | DownOperation   | main       | INFO :    Adding database versions to SQL database ...
6-3-14 21:40:32.521 | DownOperation   | main       | INFO :    + Applying database version (platoppheckel6404813311)
6-3-14 21:40:32.532 | DatabaseVersion | main       | SEVE : SQL Error:
java.sql.BatchUpdateException: integrity constraint violation: foreign key no parent; SYS_FK_10163 table: FILEVERSION
        at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)
        at org.syncany.database.dao.FileVersionSqlDao.writeFileVersions(FileVersionSqlDao.java:86)
        at org.syncany.database.dao.FileHistorySqlDao.writeFileHistories(FileHistorySqlDao.java:60)
        at org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:145)
        at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:106)
        at org.syncany.database.SqlDatabase.persistDatabaseVersion(SqlDatabase.java:127)
        at org.syncany.operations.DownOperation.persistDatabaseVersion(DownOperation.java:243)
        at org.syncany.operations.DownOperation.applyDatabaseVersions(DownOperation.java:207)
        at org.syncany.operations.DownOperation.applyWinnersBranch(DownOperation.java:193)
        at org.syncany.operations.DownOperation.execute(DownOperation.java:162)
        at org.syncany.operations.WatchOperation.runSync(WatchOperation.java:159)
        at org.syncany.operations.WatchOperation.execute(WatchOperation.java:112)
        at org.syncany.Client.watch(Client.java:165)
        at org.syncany.cli.WatchCommand.execute(WatchCommand.java:98)
        at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
        at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
        at org.syncany.Syncany.main(Syncany.java:57)
6-3-14 21:40:32.533 | WatchOperation  | main       | INFO : Sync FAILED, waiting 120 seconds ...
java.lang.RuntimeException: Cannot persist database.
        at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:114)
        at org.syncany.database.SqlDatabase.persistDatabaseVersion(SqlDatabase.java:127)
        at org.syncany.operations.DownOperation.persistDatabaseVersion(DownOperation.java:243)
        at org.syncany.operations.DownOperation.applyDatabaseVersions(DownOperation.java:207)
        at org.syncany.operations.DownOperation.applyWinnersBranch(DownOperation.java:193)
        at org.syncany.operations.DownOperation.execute(DownOperation.java:162)
        at org.syncany.operations.WatchOperation.runSync(WatchOperation.java:159)
        at org.syncany.operations.WatchOperation.execute(WatchOperation.java:112)
        at org.syncany.Client.watch(Client.java:165)
        at org.syncany.cli.WatchCommand.execute(WatchCommand.java:98)
        at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
        at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
        at org.syncany.Syncany.main(Syncany.java:57)
Caused by: java.sql.BatchUpdateException: integrity constraint violation: foreign key no parent; SYS_FK_10163 table: FILEVERSION
        at org.hsqldb.jdbc.JDBCPreparedStatement.executeBatch(Unknown Source)
        at org.syncany.database.dao.FileVersionSqlDao.writeFileVersions(FileVersionSqlDao.java:86)
        at org.syncany.database.dao.FileHistorySqlDao.writeFileHistories(FileHistorySqlDao.java:60)
        at org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:145)
        at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:106)
        ... 12 more

Gradle/Git: Extract non-core plugins / allow optional plugins

Right now, the master branch contains many plugins in the syncany-plugins directory, namely these plugins: syncany-plugin-ftp, syncany-plugin-sftp, syncany-plugin-webdav, syncany-plugin-rest.

Also, the plugins appear in the gradle configuration for the CLI (and the GUI): settings.gradle and in the dependencies of syncany-cli.

In the future, the number of plugins will grow, and I won't have enough time to maintain them all. That's why we need the possibility to add optional plugins from other github repos -- which can be hosted at the Syncany organization space, but maintained by others.

Something like git subtree or submodule could be used, but maybe something completely loosely coupled would be even better.

I'm looking for someone to implement this. Please let me know if you're interested.

There are no mentions of how to use with GUI

I saw screenshots of a Syncany GUI, and a Nautilus plugin. However there are no mentions in the readme, or the syncany.org website on how to use them, where are their source code hosted, or anything. The only thing I could find (letting Google help me) was that the core and the GUI separated. Can I still use the GUI?

Adjust crypto concept & implementation (no nested ciphers)

Due to security concerns raised by Internetland, crypto students and even a few well-respected cryptographers, nesting ciphers could be vulnerable to different kinds of attacks (meet-in-the-middle, etc.)

The crypto concept and implementation has to be adjusted accordingly.

Related:

• Issue #28
• http://crypto.stackexchange.com/questions/5827/can-a-nested-block-cipher-avoid-the-meet-in-the-middle-attack-by-using-a-secret
• http://crypto.stackexchange.com/questions/1737/can-nesting-different-ciphers-with-different-keys-ever-decrease-security

Uniqueness constraint violation in retrieving multichunks

MultiChunkSqlDao.getMultiChunkIdsByChecksums throws an error when passed a list with double instances of an id. The best way to fix this is likely to use a Set instead of a List.

I will pick this up

Windows installer: Add bin/ to PATH variable during setup

The Windows installer currently does not add C:\Program Files\Syncany\bin\ to the PATH variable. The user is asked to do that manually. This should be done automatically during setup.

References:

• See the innosetup folder in the repo, especially the setup.iss.skel file
• Also see: http://stackoverflow.com/questions/3304463/how-do-i-modify-the-path-environment-variable-when-running-an-inno-setup-install
• Or: http://www.jrsoftware.org/isfaq.php#env

CLI: Allow command help texts; split help.skel

The --help text is really large. I think there should be a shortened version and per-command pages. Something like this:

$ sy 
(short help text)

$ sy --help
(medium help text)

$ sy --help-all
(detailed per-command help texts)

$ sy init --help
(long help text for "init" command)

Test UploadInterruptedTest fails if run in combination with other tests

The test UploadInterruptedTest fails if ScenarioTestSuite and DatabaseTestSuite is run before, but not if OtherShortTestSuite is run in standalone.

What fails is this assertion, because the corresponding file exists:

assertFalse(new File(testConnection.getRepositoryPath()+"/databases/db-A-0000000004").exists());

The file db-A-0000000004 exists, contains correct information and is created after db-A-0000000003, so one can assume that the files are created by this test and not another...

Running AllExceptLongRunningTestSuite creates this behavior. I could not figure out why it behaves this way.

WebDAV - erroneous error - Repo already exists

Many thanks for syncany! Starting to use it with Mac OS X. When trying to set up a WebDAV based repository, the following error message comes up. I tried livedrive.com and sd2.1und1.de, the error message is identical. However, the repo does not already exist, because I had created the repo folder myself. The repo folder is empty and has standard permissions. Please find the debug output here: http://pastebin.com/ZrM4YNkJ. Can I help resolving this error? Cheers

Use TransferManager.test() in CLI

The CLI currently does not make use of the test() method in the TransferManager. To improve usability, it should be used.

Release 0.1.1-alpha

Ignore operation and .ignore file

Currently it is not possible to selectively sync files or folders. To achieve the possibility of this, we should have an IgnoreOperation, which ensures that an ignored file is not considered when compiling changesets and when Upping. In addition, there should be a .ignore file, in the same spirit of .gitignore and friends.

No one is working on this. Feel free to comment here if you want to pick this up.

Allow user-interaction/choice for file conflicts

As of today, the DownOperation determines the remote changes and maps them to local changes (a la "rename fileA to fileB", "create fileC"). These FileSystemActions always behave as follows:

• If the local file (on the disk) matches the last known file version in the local database, apply the changes.
• If the local file does not match the last known file version in the local database, a local conflict file is created at "filename (username's conflict, ...)", and the new remote file is created at "filename".

This behavior is not always desired (as the long discussions on the mailing list prove). Instead, a user might want to decide what to do, e.g.

file1 changed. What to do?
[x] create conflict file at "file1 (conflict ...)"
[ ] overwrite local file (discard local changes)
[ ] keep local file (discard remote changes)

Of course this has to be compatible with many files, so that not every file creates such a popup/dialog/user-query.

To achieve this, I believe the best way is to

• Extract a ApplyChangesOperation from the DownOperation
• Determine file change actions before the ApplyChangesOperation is called (right now, the FileSystemAction itself decides what to do!)

I'm looking for someone to implement this. Please let me know if you're interested.

"Cannot determine file content for checksum"

New exception: Produced by @pimotte and my repo right now. Can be reproduced by doing a sy down from a fresh client. This will be "fun" to debug...

@pimotte We might need to exchange log files sooner or later. I'll ask for them if I don't get anywhere.

@pimotte @vwiencek Please do not upload anything to this repo -- only do sy down. I backed it up, just in case though.

5-4-14 20:14:24.826 | FileSystemActio | main       | INFO :      - Creating file Papers and such/Bachelor-Arbeit (1.0).pdf to /home/pheckel/Syncany/.syncany/cache/temp-reconstructedFileVersion-2889740705105438377.tmp ...
Exception in thread "main" java.lang.Exception: Cannot determine file content for checksum 7563c20661e77183d25613be9587ad0c65ecb24f
    at org.syncany.operations.down.actions.FileCreatingFileSystemAction.assembleFileToCache(FileCreatingFileSystemAction.java:103)
    at org.syncany.operations.down.actions.FileCreatingFileSystemAction.createFile(FileCreatingFileSystemAction.java:85)
    at org.syncany.operations.down.actions.FileCreatingFileSystemAction.createFileFolderOrSymlink(FileCreatingFileSystemAction.java:49)
    at org.syncany.operations.down.actions.NewFileSystemAction.execute(NewFileSystemAction.java:42)
    at org.syncany.operations.down.DownOperation.applyFileSystemActions(DownOperation.java:421)
    at org.syncany.operations.down.DownOperation.applyWinnersBranch(DownOperation.java:207)
    at org.syncany.operations.down.DownOperation.execute(DownOperation.java:173)
    at org.syncany.Client.down(Client.java:144)
    at org.syncany.Client.down(Client.java:140)
    at org.syncany.cli.DownCommand.execute(DownCommand.java:44)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)

[WebDAV] Allow additional (untrusted) certificates

Currently syncany fails when trying to connect to a WebDAV storage via https when the certificate is not in the Java trust store.
Furthermore the reason why the connection fails is only obvious when looking at the CLI output.
Until this is implemented it would be nice to output a more meaningful error message in the GUI.

No endless database/history: Implement cleanup (delete after X days, keep Y versions)

As of today, the local database (and the databases exchanged by users) are ever-growing databases -- meaning that all versions of all files (even deleted files) are kept forever. In particular, there is no file version purging after X days or Y versions, and no deletion of multichunks on the remote storage -- which leads to ever-growing disk space on the remote storage.

In short, we need to implement the following:

• Allow database / file version purging after X days or Y versions
• Remove unreferenced chunks/multichunks/filecontents from the local database
• Lock the remote storage, remove unreferenced multichunks and upload "purge" database
• Repackage multichunks (removed unused chunks) if more than X% of the multichunk contain unreferenced chunks.

This is one of the few things missing for release 0.1, I'm on top of this; but feel free to comment or discuss here

CleanupOperation (old file removal) needs min/max allowed versions

The current implementation of the CleanupOperation removes old file versions if there are more than N versions for a file -- so file histories are only N versions long. Default for N is 5.

The CleanupOperation is called in every UpOperation (unless cleanup is disabled in the UpOperationOptions). If a file "somefile.txt" is updated very often (save, edit, save, edit, save, edit, ...), new file versions for this file are added every time, and if the max. amount of N file versions is reached, the older file versions are deleted.

These deletions are communicated to other clients in PURGE database versions, which is basically a normal database version, but interpreted differently. In the above case, this leads to two database version files (db-clientname-0000X and db-clientname-0000[X+1]) are uploaded every time -- meaning that every time you click "Save", two database files have to be uploaded and then downloaded/processed by the other clients.

Long story short: There should a min. and a max file history length parameter (like in the "merge databases") case. The behavior should be as follows: If a file history is MAX file versions long, delete all but MIN file versions of it. E.g. if it is 15 file versions long, delete all but 5. That would mean that the cleanup must be called only every 10 "up" calls.

Windows wrong test paths: creates folders a la "C:\UsersDanielAppDataLocalTemp..."

As reported by a user via e-mail, the tests have issues with backslashes. Instead of creating temp files in C:\Users\Daniel\AppData\LocalTemp..., Syncany creates C:\UsersDanielAppDataLocalTemp...

See here:
http://i.imgur.com/x80Ittb.png

Creation of a SFTP synchronization repository

Hi,

After a fresh installation of syncany with Ubuntu 12.04, I can't initalize the synchronization folder with sftp plugin.

I can connect to the server manually with sftp or ssh, I give the right access.
Is there any special configuration for the server part (sftp)?

Here the output of the terminal :

Choose a storage plugin. Available plugins are: ftp, local, s3, sftp, webdav
Plugin: sftp

Connection details for SFTP connection:
- Hostname: @IP
- Username: bob
- Password (not displayed): 
- Path: /data
- Port (optional, default is 22): 

The password is used to encrypt data on the remote storage.
Wisely choose you must!

Password: 
Confirm: 

WARNING: The password is a bit short. Less than 12 chars are not future-proof!
Are you sure you want to use it (y/n)? y
Exception in thread "main" java.lang.Exception: Unable to connect to repository.
    at org.syncany.operations.InitOperation.repoFileExistsOnRemoteStorage(InitOperation.java:151)
    at org.syncany.operations.InitOperation.execute(InitOperation.java:77)
    at org.syncany.Client.init(Client.java:174)
    at org.syncany.cli.InitCommand.execute(InitCommand.java:66)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:230)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)
Caused by: org.syncany.connection.plugins.StorageException: 2: No such file
    at org.syncany.connection.plugins.sftp.SftpTransferManager.list(SftpTransferManager.java:278)
    at org.syncany.operations.InitOperation.repoFileExistsOnRemoteStorage(InitOperation.java:147)
    ... 6 more
Caused by: 2: No such file
    at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2833)
    at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2185)
    at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2202)
    at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1566)
    at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1527)
    at org.syncany.connection.plugins.sftp.SftpTransferManager.list(SftpTransferManager.java:255)
    ... 7 more

Thank you.

Split off current daemon code from GUI

As described in the mailing list, the current daemon implementation should be separated from the GUI branch so that we can firstly create a clean daemon before we focus on the GUI.

Also, the daemon needs to be branched off of the 'develop' branch.

Travis CI & Coveralls.io integration

Set up Travis-CI (continous integration) and Coveralls.io (continuous coverage tool, like EclEmma) for Syncany.

Must-have:

• Automatic tests via Travis CI
• Coverage via coveralls.io

Nice to have:

• Automatic creation and upload of distributables tar.gz/zip/exe/deb/rpm

After DIRTY: integrity constraint violation: unique constraint or index violation; SYS_PK_10180 table: MULTICHUNK

Yet another unique constraint: Not yet reproducable, although I am confident this shouldn't be too hard.

• Copy archive.zip to A, let it sync to B
• Extract archive.zip at A and at the same time extract archive.zip at B

java.sql.SQLIntegrityConstraintViolationException: integrity constraint violation: unique constraint or index violation; SYS_PK_10180 table: MULTICHUNK
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
    at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(Unknown Source)
    at org.syncany.database.dao.MultiChunkSqlDao.writeMultiChunks(MultiChunkSqlDao.java:62)
    at org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:143)
    at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:106)
    at org.syncany.database.SqlDatabase.persistDatabaseVersion(SqlDatabase.java:132)
    at org.syncany.operations.down.DownOperation.persistDatabaseVersion(DownOperation.java:258)
    at org.syncany.operations.down.DownOperation.applyDatabaseVersions(DownOperation.java:222)
    at org.syncany.operations.down.DownOperation.applyWinnersBranch(DownOperation.java:208)
    at org.syncany.operations.down.DownOperation.execute(DownOperation.java:173)
    at org.syncany.operations.watch.WatchOperation.runSync(WatchOperation.java:171)
    at org.syncany.operations.watch.WatchOperation.pushNotificationReceived(WatchOperation.java:194)
    at org.syncany.operations.watch.NotificationListener$SocketThread.processIncomingMessage(NotificationListener.java:272)
    at org.syncany.operations.watch.NotificationListener$SocketThread.run(NotificationListener.java:224)
Caused by: org.hsqldb.HsqlException: integrity constraint violation: unique constraint or index violation; SYS_PK_10180 table: MULTICHUNK
    at org.hsqldb.error.Error.error(Unknown Source)
    at org.hsqldb.Constraint.getException(Unknown Source)
    at org.hsqldb.index.IndexAVL.insert(Unknown Source)
    at org.hsqldb.persist.RowStoreAVL.indexRow(Unknown Source)
    at org.hsqldb.persist.RowStoreAVLDisk.indexRow(Unknown Source)
    at org.hsqldb.TransactionManager2PL.addInsertAction(Unknown Source)
    at org.hsqldb.Session.addInsertAction(Unknown Source)
    at org.hsqldb.Table.insertSingleRow(Unknown Source)
    at org.hsqldb.StatementDML.insertSingleRow(Unknown Source)
    at org.hsqldb.StatementInsert.getResult(Unknown Source)
    at org.hsqldb.StatementDMQL.execute(Unknown Source)
    at org.hsqldb.Session.executeCompiledStatement(Unknown Source)
    at org.hsqldb.Session.execute(Unknown Source)
    ... 14 more

Cannot persist database: "sy down" after cleanup

New issue with "sy down" after cleanup. No real way to reproduce it yet. I have a repo that re-throws this error every time though. So it's easy to debug right now.

Full log here: http://uploaded.net/file/1bhuyjuz
(no pastebin link; size limit is 500k)

$ sy down
Exception in thread "main" java.lang.RuntimeException: Cannot persist database.
    at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:114)
    at org.syncany.database.SqlDatabase.persistDatabaseVersion(SqlDatabase.java:127)
    at org.syncany.operations.DownOperation.persistDatabaseVersion(DownOperation.java:246)
    at org.syncany.operations.DownOperation.applyDatabaseVersions(DownOperation.java:210)
    at org.syncany.operations.DownOperation.applyWinnersBranch(DownOperation.java:196)
    at org.syncany.operations.DownOperation.execute(DownOperation.java:162)
    at org.syncany.Client.down(Client.java:136)
    at org.syncany.cli.DownCommand.execute(DownCommand.java:44)
    at org.syncany.cli.CommandLineClient.runCommand(CommandLineClient.java:236)
    at org.syncany.cli.CommandLineClient.start(CommandLineClient.java:109)
    at org.syncany.Syncany.main(Syncany.java:57)
Caused by: java.lang.NullPointerException
    at org.syncany.database.dao.DatabaseVersionSqlDao.writeDatabaseVersion(DatabaseVersionSqlDao.java:139)
    at org.syncany.database.dao.DatabaseVersionSqlDao.persistDatabaseVersion(DatabaseVersionSqlDao.java:106)
    ... 10 more