appnexus / sicksync Goto Github PK

Each change to the version of package.json and npm publish should correspond to a pushed git tag -a v1.0.0 -m 'Releasing 1.0.0' to line things up cleanly with github.

You could also use the npm version [ major | minor | patch ] function for a shortcut. It does the package.json modification and git tagging for you in one step. You'll still have to push the tag and publish the package to npm though.

Would it be desirable to have an option to execute a command on remote after a successful sync?

Like I'm currently syncing /etc/nginx and would like to have nginx -s reload after each sync.

File deletions are causing errors

Here the file is simply deleted:

[APP] [PC] > unlink C:\.../client/dockerfile
[APP] [REMOTE]  < ~/projects/.../client/dockerfile
/home/.../sicksync/dist/remote/fs-helper.js:67../client/dockerfile
            _fsExtra2['default']['delete']((0, _untildify2['default'])(message.destinationpath), function (err) {
                                        ^

TypeError: _fsExtra2.default.delete is not a function
    at FSHelper.removePath (/home/.../sicksync/dist/remote/fs-helper.js:67:43)
    at WSServer.<anonymous> (/home/.../sicksync/dist/remote/index.js:72:26)
    at emitOne (events.js:96:13)
    at WSServer.emit (events.js:188:7)
    at WSServer.handleMessage (/home/.../sicksync/dist/remote/ws-server.js:69:29)
    at emitTwo (events.js:106:13)
    at WebSocket.emit (events.js:191:7)
    at Receiver.ontext (/home/.../sicksync/node_modules/ws/lib/WebSocket.js:816:10)
    at /home/.../sicksync/node_modules/ws/lib/Receiver.js:477:18
    at /home/.../sicksync/node_modules/ws/lib/Receiver.js:361:7

Here the file is npm-debug.log.1806592492 which I gets created momentarily and then automatically deleted.

change client\npm-debug.log.1806592492
fs.js:640
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                ^

Error: ENOENT: no such file or directory, open 'C:\...\client\npm-debug.log.1806592492'
    at Error (native)
    at Object.fs.openSync (fs.js:640:18)
    at Object.fs.readFileSync (fs.js:508:33)
    at FSHelper.onFileChange (C:/.../sicksync/src/local/fs-helper.js:51:66)
    at Timeout._onTimeout (C:/.../sicksync/src/util.js:104:23)
    at tryOnTimeout (timers.js:232:11)
    at Timer.listOnTimeout (timers.js:202:5)

Don't know if this is a windows only issue. Does it cause issues on *nix?

Right now config related to all projects are stored in ~/.sicksync/config.json combined

I use BashOnWindows so the home ~/ in bash is not my home in windows, so getting to the bash's ~/.sicksync/conf.json is sometimes tricky, and there's a duplication of configs that way.

How about if configs related to projects be stored in the project directory itself?

Deleted files cause the server to disconnect.

related #53

right now i have to do sicksync -o; sicksync in order to do the 'big sync' first. it would be nice if it could be done in a single command

sicksync pretty much needs "absolute" hostname URI's, which kind of stink since most folks operate under aliases to do their work.

It looks like the initial setup expects the json file exists before its created.

Error: Cannot find module '/.sicksync-config.json'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/sicksync/lib/rsync.js:3:14)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

Windows support would be awesome. ssh isn't natively available on Windows, it's usually available through Git Bash or cygwin or babun. There's also https://github.com/mscdex/ssh2

Thoughts?

Seeing some files being synced more than once, probably because of my editor or some other tool in the pipeline.

Would be nice to add a debounced function before actual sync

[remote] [locl] Connected rmt1 not using encryption
[remote] [locl] > change webpack.config.babel.js
[remote] [rmt1]  < /home/rmt1/remote/proj/webpack.config.babel.js
[remote] [locl] > change .gitignore
[remote] [locl] > change .gitignore
[remote] [rmt1]  < /home/rmt1/remote/proj/.gitignore
[remote] [rmt1]  < /home/rmt1/remote/proj/.gitignore
[remote] [locl] > change .gitignore
[remote] [locl] > change .gitignore
[remote] [rmt1]  < /home/rmt1/remote/proj/.gitignore
[remote] [rmt1]  < /home/rmt1/remote/proj/.gitignore

I only modified webpack.config.babel.js here, didn't touch .gitignore at all

I've also changed the debounce timeout manually to FILE_CHANGE_COOLDOWN_MS: 2000, and file change does seem to respect that but after 2000ms I still get this repeated .gitignore file syncs

If you have node_modules excluded but the dir node_modules hasn't actually been created yet and you've started the sync and then do npm i which creates the node_modules dir, it fires the adddir event which causes the Big Rsync to be fired again. But it doesn't do anything much which is good and expected, so this probably isn't of immediate concern but still leaving this here so it might be addressed.

sicksync add chooses the same default websocketPort for all projects which causes problems if two projects are run simultaneously (#55). It should choose different ports based on what ports have already been used in other saved projects.

A have a few questions, as I have been looking for a more reliable code sync solution for some time (mostly for use between a local OS X file system, and a VM running inside VirtualBox on the same machine).

1 & 2: What is your implementation plan for Live Reload? Is this because the remote mirror does not fire inotify events for watchers like Grunt and Gulp to pick up?

3: Does this, or could this ever, support two-way sync? ie. I am SSH'd in to the dev box, and I do an NPM install --save-dev (which updates the package.json). Could a change on the remote be mirrored back to the local?

4: Can I ignore files to be mirrored?

Great work on the package, and I look forward to more to come.

It seems ignored paths aren't 100% ignored. Like if I change something in the .git directory, surely it doesn't actually get uploaded but it still triggers a file change event, meaning it's still being monitored which is unnecessary. Having a huge node_modules or otherwise ignored folders cause it to take up considerable time to start up.

This can be tested by putting a console log here: src\local\fs-helper.js and changing any file in one of the ignored folders, .git/info/excludes

Getting a lot of "Lost connection"s.

It should reconnect

npm install -g sicksync is sort of gross to do updates for a CLI. sicksync should check if there is an update and then update itself or inform the user that there is an update.

After installing sicksync, I tried to run sicksync -s for the setup wizard but got the following error:

module.js:340
    throw err;
    ^
Error: Cannot find module '/Users/jdelamotte/.sicksync-config.json'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/sicksync/lib/rsync.js:3:14)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

I was able to fix the issue with echo '{}' > ~/.sicksync-config.json which should probably happen by default when you run the setup command.

Even after #86 the file changes that were lost due to lost connection still aren't synced. A queue could help ensuring the file changes are synced in this case

Since both the client and remote machines rely on a single config object, and we expose editing this config with sicksync -c, it'd be nice to have a "listen" on that save event and sync the config in a bi-directional fashion.

Commit 60feb71 appears to have broken the ability to .git in your excludes and have it work properly. I don't think minimatch handles it the same way the previous code did. Incoming PR.

Seems that the single *'s aren't working properly, which might have to do with relative pathing.

I'm using sicksync on windows with mingw. I installed sicksync globally using node 6.9.5 installed with nvm-windows.
When I run the update command I see a successful update of the remote machine but then I get an error about my local machine. After that, sicksync is not installed anymore. Error is:

[Function: getHostname] '\u001b[31mUpdate failed! Please run manually\u001b[39m' { Error: Command failed: npm i -g sicksync
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules\sicksync\node_modules\chokidar\node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "i" "-g" "sicksync"
npm ERR! node v6.9.4
npm ERR! npm v3.10.10
npm ERR! path C:\Program Files\nodejs\sicksync
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall open

npm ERR! Error: EPERM: operation not permitted, open 'C:\Program Files\nodejs\sicksync'
npm ERR! at Error (native)
npm ERR! { Error: EPERM: operation not permitted, open 'C:\Program Files\nodejs\sicksync'
npm ERR! at Error (native)
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! syscall: 'open',
npm ERR! path: 'C:\Program Files\nodejs\sicksync' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

npm ERR! Please include the following file with any support request:
npm ERR! xxx\npm-debug.log

at ChildProcess.exithandler (child_process.js:206:12)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:877:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)

killed: false,
code: 4294963248,
signal: null,
cmd: 'npm i -g sicksync' }

Steps to reproduce:

1. clone the hbapi repo on your local and remote machines
2. cd library && git submodule init && git submodule update
3. Start sicksync watching on the hbapi repo
4. change a file in the submodule
5. sicksync will see the change and say that it synced, but the file doesn't change on the server

We need's the test.

when a user drops a connection, it would be nice if sicksync would retry establishing a connection instead of killing the process

I have a server with a full linux installation, with the usual file structure.

Locally I have a directory structure with only a very few files of interest and in the same places (paths) as they would be on the server. For example /etc/nginx/...

Right now if I sync this to root on the server / it'll delete everything but the /etc/nginx/... (as I found out x_x to my own cost).

How about mode where local files are only uploaded to the server, but any extra files on the server aren't deleted? I know it wouldn't technically be "sync"-ing but still useful.

The stubbed methods in the rsyncSpies object are never reset. The reason that the tests still pass is because each new call to Rsync inits a completely new object.

This problem should be fixed so that spies can be properly used on the Rsync object(s).

In lots of places paths are created by using '/' slashes.

// src/local/fs-helper.js:21
this._baseDir = path.parse(this._sourceLocation).base + '/';

path.sep and path.join should be used instead.

It doesn't cause any major issues right now, but something to consider in a future refactoring

Jest has been a pleasure to use on other projects, and treats things like mocks/coverage as first-class citizens rather than an afterthought.

I tried this after a long time, I know it was fixed before (#42), so I'm not sure (yet) if there's some other elements causing this.

The problem currently is that it's trying to sync from the root:

x@DELL C:\Users\x\test\sicksync-test
> sicksync start
[sicksync-test] [dell] sending incremental file list

[sicksync-test] [dell] file has vanished: "/cygdrive/c/Users/x/test/sicksync-test/\/$Recycle.Bin"

[sicksync-test] [dell] file has vanished: "/cygdrive/c/Users/x/test/sicksync-test/\/$WINRE_BACKUP_PARTITION.MARKER"

[sicksync-test] [dell] file has vanished: "/cygdrive/c/Users/x/test/sicksync-test/\/Apps"

I'm syncing from C:\Users\x\test\sicksync-test but it's trying to sync files like /cygdrive/c/Users/x/test/sicksync-test/\/$Recycle.Bin

Sometimes I would like to monitor node_modules for some particular file I might change locally and want only that file to be uploaded.

But including node_modules to file watcher causes a big rsync.

There should be an option to disable it with a flag.

Would it be desirable to have it load the exclude from a file like .gitignore?

Right now sicksync is one-way. It pushes files from the local to the remote. In some development cases the need for bi-directional syncing is needed.

Justification:

Doing NPM installs on a local (OS X) that is a different architecture than the remote (Linux) is a bad idea, since modules that have native extensions require being compiled on the arch where they will be ran. I am often SSH'd into the remote during development and will perform my npm install MODULE --save there. This updates the package.json. That update should be mirrored back to the local to be committed into source control.

I think config should be refactored. It'll probably be required to be done in #78, so here's a couple ideas that could be helpful to implement while refactoring

• combine project config and CLI args.

  The two options I added in #84 (disableDeletion & disableRsync) can be configured by both project config as well as CLI args but it looks ugly if (!(params.disableDeletion || project.disableDeletion)) {. They should be combined before this check.

• CLI params should override project config.

• There should a default master config that should be applied for all projects. Currently all projects have their own separate configs and there's a lot of duplication in config itself that way. A master config could help a lot. The project config could then only be used to override the default one. (and CLI params still override them both)

The module responsible for the initialSync and bigSync is not sending deletions. This maybe configurable, but it should be investigated as it's not effectively mirroring if that's the case.

It'd be nice to have multiple folders sicksynced independently. This would likely require some config modifications...

If you have two sicksync instances they seem to interfere with one another's operation, when one of them sees a file change another one gets disconnected.

There's also that re-connection issue (related #51) that it re-connects multiple times so after a while you see loads of connections:

[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
[REMOTE] [LOCAL] Connected server not using encryption
… (about >50)
^CTerminate batch job (Y/N)? ^C

PS: Both instances are to the same remote. Haven't checked with different remotes.

Sometimes it's too restrictive that I cannot start syncing any current folder.

How about if it could just let you sync current dir by just asking you which path on server you want to sync to.

Like this:

sicksync --here --server <server-name> --path /some/path/on/server 

The dependency on rsync for syncing had me conform to a uniform API as far as ignores go. It'd be nice to use nodes's globbing patterns for ignores as they're pretty ubiquitous.

Need to investigate this as well it's potential impact on the rsync module.