Instaclone is a simple, configurable command-line tool to publish and later install snapshots of files or directories in S3 (or another store). It keeps a cache of downloaded snapshots so switching between previously cached snapshots is instant -- just a symlink to the local cache.
It's good for files you want to version but not check them into Git, due to size, sensitivity, platform dependence, etc. You can git-ignore the original files, publish them with Instaclone, and instead check in the Instaclone configuration file that references them.
This tool isn't only for use with Node, but this is a good motivating use case.
While npm is amazingly convenient during development, managing the workflow around npm install
can be a pain point in terms of speed, reliability, and reproducibility as you scale out builds and in production. If you use Instaclone to publish after committing your npm shrinkwrap file, you can switch back and forth between Git branches and run instaclone install
instantly instead of npm install
and waiting 3 minutes. Your colleagues can do this too -- after you publish, they can run instaclone install
and get a byte-for-byte exact copy of your node_modules
cached on their machines. Finally, your CI builds will speed up most of the time -- possibly by a lot! See below for more info on this.
- Upload/download is via configurable shell commands, using whatever backing storage system desired, so you don't have to worry about configuring credentials just for this tool, and can publish to S3 or elsewhere. I'd recommend using
s4cmd
for high-performance multi-connection access to S3. - Snapshots have version strings, which can be:
- Explicit (you just say what version to use in the config file)
- SHA1 of a file (you say another file that is hashed to get a unique string)
- Command (you have Instaclone execute an arbitrary command, like
uname
, which means you can have different versions per platform type automatically)
- Simple, clean internal format.
- By default it's in
~/.instaclone
, but you can set theINSTACLONE_DIR
environment variable to set this directory to something else. - The file cache is just a simple file tree that you can look at and clean up as you wish.
- Directories are archived as .zip files, stored next to the full directory, which is read-only.
- By default it's in
- Good hygiene: All files, directories, and archives are created atomically, so that interruptions or problems never leave files in a partially complete state.
- You can install items as symlinks (probably what you want), hardlinks (works for files but not directories), or fully copy (slower but still better than a download).
- Some conveneint details regarding handling of symlinks and symlink installs:
- The file permissions on items in the cache is read-only, so that if you inadvertently try to modify the contents of the cache by following the symlink and changing a file, it will fail.
- The target of the symlink (in the cache) has the same name as the source, so installed symlinks will play nice paths like
../target/foo
(wheretarget
is the symlink). - Internally within an archive, relative symlinks are preserved. But instaclone is smart enough to check for and abort if it sees symlinks to absolute paths or to relative paths outside the source directory (which would usually be a mistake).
Requires Python 2.7+. Then (with sudo if desired):
pip install instaclone
It also requires some tools in your path:
zip
andunzip
s3cmd
,aws
,s4cmd
, or any similar tool you put into yourupload_command
anddownload_command
settings
Instaclone requires two things to run:
- A config file, which can be called
instaclone.yml
orinstaclone.json
in the current directory. (YAML or JSON syntax is fine.) This configuration file says how resources will be published and installed. - A main directory to store the cache in, which defaults to
$HOME/.instaclone
but can be overridden by theINSTACLONE_DIR
environment variable. If you want, you can put a globalinstaclone.{yml,json}
file there instead.
As an example, here is a marginally self-explanatory instaclone.yml
configuration, which you would drop anywhere you want and probably should check into Git. You'd create as many files like this as desired in different directories, taking care you give them distinct remote_path
s are unique.
---
# You can have as many items as you like and all will be installed.
# You'll want to git-ignore the local_paths below.
items:
# A big file lives in this directory. It takes a while to generate, so we're going to
# reference it in this file by version, instaclone publish, and anyone can
# instaclone install it. We update the version string manually when we regenerate it.
- local_path: my-big-and-occasionally-generated-resource.bin
remote_prefix: s3://my-bucket/instaclone-resources
remote_path: some/big-resources
upload_command: s4cmd put -f $LOCAL $REMOTE
download_command: s4cmd get $REMOTE $LOCAL
copy_type: symlink
# This is the version of the file. It can be any string.
version: 42a
- local_path: node_modules
remote_prefix: s3://my-bucket/instaclone-resources
remote_path: my-app/node-stuff
upload_command: s4cmd put -f $LOCAL $REMOTE
download_command: s4cmd get $REMOTE $LOCAL
copy_type: symlink
# We generate the version string as a hash of the npm-shrinkwrap.json plus the architecture we're on:
version_hashable: npm-shrinkwrap.json
version_command: uname
See below for more on the node_modules
one.
Once Instaclone is configured, run:
instaclone publish
: upload configured items (and add to cache)instaclone install
: download configured items (and add to cache)instaclone configs
: sanity check configurationinstaclone purge
: delete entire cache (leaving resources uploaded)
This use case deserves a little more explanation.
Having fast and reproducible runs of npm install
is a challenge for developers, CI systems, and deployment:
- As we
all
know,
the state of the
node_modules
is not inherently reproducible from thepackage.json
file, so you should usenpm shrinkwrap
. - While
npm shrinkwrap
mostly locks down exact package versions, even this doesn't guarantee byte-for-byte repeatable installations. Think about it: Reliability requires controlling change. If you change some single piece of code somewhere unrelated to your dependencies, and your build system rerunsnpm install
, what if one of your hundreds of packages was unpublished, or you have an issue connecting to npmjs.org? In addition, shrinkwrap doesn't prevent churn in peer dependencies. It's impossible to ensure exact repeatability unless you just make an exact copy and use it everywhere. - Operationally, you also want a more scalable solution to distributing packages than hitting npmjs.org every time. You don't want lots of servers or build machines doing this continuously.
- You can set up a local npm repository or a local npm cache server to help, but this is more infrastructure for devops to maintain and scale. And incidentally, it also is likely to be a single point of failure: Not being able to push new builds reliably is a Bad Thing (precisely when you don't need it). Plus, if you use private modules and pay npm, Inc. to host private code for you, you probably don't otherwise need another local repository.
- Finally, downloading from the global server and even installing from the local cache, take a lot of time, e.g if you want to do rapid CI builds from a clean install. With all these solutions,
npm install
still takes minutes for large projects even when you haven't changed anything.
A simpler and more scalable solution to this is to archive the entire node_modules
directory, and put it somewhere reliable, like S3. But it can be large and slow to manage if it's always published and then fetched every time you need it. It's also a headache to script, especially in a continuous integration environment, where you want to re-install fresh on builds on all branches, every few minutes, and reinstall only when the checked-in npm-shrinkwrap.json
file changes. Oh, and also the builds are platform-dependent, so you need to publish separately on MacOS and Linux.
Instaclone does all this for you. If you already have an npm shrinkwrap
workflow, it's pretty easy. It lets you specify where to store your node_modules
in S3, and version that entire tree by the SHA1 hash of the npm-shrinkwrap.json
file togetehr with the architecture. You can then work on multiple branches and swap them in and out -- a bit like how nvm
caches Node installations.
Copy and edit the example config file to try it. On your CI system, you might want to have some sort of automation that tries to reuse pre-published versions, but if not, publishes automatically:
echo "Running instaclone install and publish..."
instaclone install || (rm -rf ./node_modules && npm install && instaclone publish)
Mostly a one-day hack. It works in at least one continuous build environment, but still under development.
- You have to clean up the cache manually by running
instaclone purge
or deleting files in~/.instaclone/cache
-- there's no automated process for this yet, so if you publish a lot it will begin to accumulate. - There is no
unpublish
functionality -- if you publish something by mistake, go find it in S3 (or wherever you put it) and delete it. - If you are obsessed with Node, you'll somehow have to accept that this is written in Python.
- See issues and the TODOs list for further work.
Tests require s4cmd
:
$ TEST_BUCKET=my-s3-bucket tests/run.sh
This is a bash-based harness that runs the test script at tests/tests.sh
. Its output can then be git diff
ed with the previous output.
Yes, please! File issues for bugs or general discussion. PRs welcome as well -- just figure out how to run the tests and document any other testing that's been done.
Apache.