we need at least one :)

The dir ~/.srcd comes from the one we used in Engine, but since the command is sourced I think it makes sense to rename it to ~/.sourced too.

The command must be renamed from sourced-ce to sourced.

Right now you can change your active workdir. But you can't list all the dirs for which you already have data.

We need to define if this will be a command to list existing workdirs, or just a matter of adding to the docs that you can peek into ~/.srcd/workdirs.

Related to this, you can prune existing data for the current active workdir, but this will not clean the data for other workdirs.

I downloaded the latest sourced v0.14.0-beta.1 and when installing sandbox I got a weird error message.

(but the UI was opened in Chrome, and everything seems to work)

$ sourced install /projects/src/github.com/src-d
docker-compose working directory set to /home/david/.srcd/workdirs/projects/src/github.com/src-d
Creating network "srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_default" with the default driver
Creating volume "srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_gitbase_indexes" with default driver
Creating volume "srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_postgres" with default driver
Creating volume "srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_redis" with default driver
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_redis_1    ... done
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_postgres_1 ... done
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_bblfsh_1   ... done
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_gitbase_1    ... done
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_bblfsh-web_1 ... done
Creating srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_sourced-ui_1 ... done

[21958:21981:0603/125639.629908:ERROR:browser_process_sub_thread.cc(217)] Waited 7 ms for network service
Opening in existing browser session.

$ sourced status
                           Name                                         Command                  State               Ports         
-----------------------------------------------------------------------------------------------------------------------------------
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_bblfsh-web_1   /bin/bblfsh-web -addr :808 ...   Up             0.0.0.0:9999->8080/tcp
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_bblfsh_1       /tini -- bblfshd                 Up             0.0.0.0:9432->9432/tcp
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_gitbase_1      ./init.sh                        Up             0.0.0.0:3306->3306/tcp
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_postgres_1     docker-entrypoint.sh postgres    Up             0.0.0.0:5432->5432/tcp
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_redis_1        docker-entrypoint.sh redis ...   Up             0.0.0.0:6379->6379/tcp
srcd-7ebb7b45703b71262e89f73d16f70f66328f2858_sourced-ui_1   /entrypoint.sh                   Up (healthy)   0.0.0.0:8088->8088/tcp

When this was discussed the docker-compose.yml and the sourced binary releases were assumed to be independent.
But now that both are in the same repository and will be released together, I think the reasonable functionality is to assume an update means "update sourced" exclusively.

Any sourced command should check the latest release tag from github, and show a warning to the user with a link to the release.

For local mode we keep the original path from the root as a path to the workdir.
For org mode the path is base64 from orgs names.

Originally it was proposed to move all local workdirs in local subdirectory and orgs in orgs.
Though we have concerns about it and it was also proposed to use base64 for consistency and to simplify the logic inside cli.

There is no final decision yet, we need to discuss it.

we support passing GITBASE_REPOS_DIR env var from cli environment into docker

There are few reasons to remove the ability to pass through vars completely it in my opinion:

1. The env var was renamed in one of the PRs already
2. .env file grows. Allowing a user changing it from outside can break things.
3. A user can be not aware of set variables in his environment but they will override values from .env which would cause very strange bug reports.

We should at least provide a way to see the logs. cd to the active workdir and docker-compose logs.

If we know any other common problems, let's also add them here.

Parent #47

Currently we have init and orgs init.
Most probably we should have local init and orgs init.
But better to confirm with @marnovo before making the change to avoid renaming again to something else later.

From #20 (review)
@smacker said:

But we definitely need to think a bit more about how to re-create containers.
It is easy with only update command. But we also have sourced compose download and sourced compose set which right now don't work as expected. (At least not as I expect)

Our wrapper for docker-compose up is install/init. But it requires an argument with the working directory.
So right now we don't have a command for "run docker-compose up without changing the current workdir".

This should be discussed in the next team meeting.
Maybe we just need to add a sourced up command in addition to the current sourced install/init.

Since the same images can be used by many workdirs, sourced prune --images can show many docker-compose errors in such installations

e.g.

Removing image srcd/gitbase:v0.20.0
ERROR: Failed to remove image for service gitbase: 409 Client Error:
  Conflict ("conflict: unable to remove repository reference "srcd/gitbase:v0.20.0"
  (must force) - container 90f006083a79 is using its referenced image e6029594062e")

I see that it can be expected behavior, but I wonder if we'd like to improve the DX is these situations, like:

• if there are already prepared workdirs, skip image deletion, and notice the user about that.

This is the error:

./sourced init /home/antonio/bblfsh-org/
docker-compose working directory set to /home/antonio/.sourced/workdirs/home/antonio/bblfsh-org
Traceback (most recent call last):
  File "/snap/docker/384/bin/docker-compose", line 11, in <module>
    load_entry_point('docker-compose==1.22.0', 'console_scripts', 'docker-compose')()
  File "/snap/docker/384/lib/python2.7/site-packages/compose/cli/main.py", line 71, in main
    command()
  File "/snap/docker/384/lib/python2.7/site-packages/compose/cli/main.py", line 124, in perform_command
    project = project_from_options('.', options)
  File "/snap/docker/384/lib/python2.7/site-packages/compose/cli/command.py", line 26, in project_from_options
    environment = Environment.from_env_file(project_dir)
  File "/snap/docker/384/lib/python2.7/site-packages/compose/config/environment.py", line 61, in from_env_file
    instance = _initialize()
  File "/snap/docker/384/lib/python2.7/site-packages/compose/config/environment.py", line 57, in _initialize
    return cls(env_vars_from_file(env_file_path))
  File "/snap/docker/384/lib/python2.7/site-packages/compose/config/environment.py", line 35, in env_vars_from_file
    with contextlib.closing(codecs.open(filename, 'r', 'utf-8-sig')) as fileobj:
  File "/snap/docker/384/usr/lib/python2.7/codecs.py", line 896, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 13] Permission denied: u'./.env'
exit status 1

Sometimes gitbase container doesn't stop with sourced stop, then sourced start fails due to port conflict.
Then you have to stop the container manually using Docker, and sourced start works as expected.

Not sure how I quit it last time, maybe this is related? However surely Docker was quit in between.

Versions:

• OS: macOS Mojave 10.14.5 (18F132)
• Docker: 2.0.0.3
• sourced binary: v0.14.0-beta.1

Log:

$ ./sourced start
Starting bblfsh     ... done
Starting gitbase    ... error
Starting bblfsh-web ... done
Starting redis      ... done
Starting postgres   ... done
Starting sourced-ui ... error

ERROR: for gitbase  Cannot start service gitbase: driver failed programming external connectivity on endpoint srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_gitbase_1 (ba83c96b32371a45024f4ccdc9a9079304db1aa202c0b1f6928c9fda2972ddcb): Bind for 0.0.0.0:3306 failed: port is already allocated
Initializing source{d}... ⠸^C

$ docker ps -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                      PORTS                    NAMES
7de285894081        srcd/sourced-ui:v0.1.0           "/entrypoint.sh"         13 days ago         Exited (137) 11 days ago                             srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_sourced-ui_1
a5bcb5bcf5d5        bblfsh/web:v0.11.0               "/bin/bblfsh-web -ad…"   13 days ago         Up 21 seconds               0.0.0.0:9999->8080/tcp   srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh-web_1
e068969470a7        srcd/gitbase:v0.20.0             "./init.sh"              13 days ago         Exited (255) 11 days ago                             srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_gitbase_1
b76975d14aff        redis:3.2                        "docker-entrypoint.s…"   13 days ago         Up 21 seconds               0.0.0.0:6379->6379/tcp   srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_redis_1
3ec8761c7afa        postgres:10                      "docker-entrypoint.s…"   13 days ago         Up 22 seconds               0.0.0.0:5432->5432/tcp   srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_postgres_1
6ed58b31d5f6        bblfsh/bblfshd:v2.14.0-drivers   "/tini -- bblfshd"       13 days ago         Up 22 seconds               0.0.0.0:9432->9432/tcp   srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh_1
cf96a5f9dee5        srcd/sourced-ui:v0.1.0           "/entrypoint.sh"         13 days ago         Exited (137) 13 days ago                             srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_sourced-ui_1
c89b7ef19b92        srcd/gitbase:v0.20.0             "./init.sh"              13 days ago         Up 22 hours                 0.0.0.0:3306->3306/tcp   srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_gitbase_1
dea48225e14a        bblfsh/web:v0.11.0               "/bin/bblfsh-web -ad…"   13 days ago         Exited (2) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_bblfsh-web_1
c82fd2a398d3        redis:3.2                        "docker-entrypoint.s…"   13 days ago         Exited (0) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_redis_1
70d5a2a4edb0        postgres:10                      "docker-entrypoint.s…"   13 days ago         Exited (0) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_postgres_1
0a4d034d71a6        bblfsh/bblfshd:v2.14.0-drivers   "/tini -- bblfshd"       13 days ago         Exited (255) 22 hours ago   0.0.0.0:9432->9432/tcp   srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_bblfsh_1

$ ./sourced stop
Stopping srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh-web_1 ... done
Stopping srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_redis_1      ... done
Stopping srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_postgres_1   ... done
Stopping srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh_1     ... done

$ ./sourced stop

sourced-ce/build on  master > docker container list -a
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                      PORTS                    NAMES
7de285894081        srcd/sourced-ui:v0.1.0           "/entrypoint.sh"         13 days ago         Exited (137) 11 days ago                             srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_sourced-ui_1
a5bcb5bcf5d5        bblfsh/web:v0.11.0               "/bin/bblfsh-web -ad…"   13 days ago         Exited (2) 29 seconds ago                            srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh-web_1
e068969470a7        srcd/gitbase:v0.20.0             "./init.sh"              13 days ago         Exited (255) 11 days ago                             srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_gitbase_1
b76975d14aff        redis:3.2                        "docker-entrypoint.s…"   13 days ago         Exited (0) 29 seconds ago                            srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_redis_1
3ec8761c7afa        postgres:10                      "docker-entrypoint.s…"   13 days ago         Exited (0) 29 seconds ago                            srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_postgres_1
6ed58b31d5f6        bblfsh/bblfshd:v2.14.0-drivers   "/tini -- bblfshd"       13 days ago         Exited (1) 29 seconds ago                            srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh_1
cf96a5f9dee5        srcd/sourced-ui:v0.1.0           "/entrypoint.sh"         13 days ago         Exited (137) 13 days ago                             srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_sourced-ui_1
c89b7ef19b92        srcd/gitbase:v0.20.0             "./init.sh"              13 days ago         Up 22 hours                 0.0.0.0:3306->3306/tcp   srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_gitbase_1
dea48225e14a        bblfsh/web:v0.11.0               "/bin/bblfsh-web -ad…"   13 days ago         Exited (2) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_bblfsh-web_1
c82fd2a398d3        redis:3.2                        "docker-entrypoint.s…"   13 days ago         Exited (0) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_redis_1
70d5a2a4edb0        postgres:10                      "docker-entrypoint.s…"   13 days ago         Exited (0) 13 days ago                               srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_postgres_1
0a4d034d71a6        bblfsh/bblfshd:v2.14.0-drivers   "/tini -- bblfshd"       13 days ago         Exited (255) 22 hours ago   0.0.0.0:9432->9432/tcp   srcd-d88690e30386069096c7c33915c6dbbbc084c9e7_bblfsh_1

$ docker container stop c89b7ef19b92
c89b7ef19b92

$ ./sourced status
                           Name                                         Command                State     Ports
--------------------------------------------------------------------------------------------------------------
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh-web_1   /bin/bblfsh-web -addr :808 ...   Exit 2
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_bblfsh_1       /tini -- bblfshd                 Exit 1
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_gitbase_1      ./init.sh                        Exit 255
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_postgres_1     docker-entrypoint.sh postgres    Exit 0
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_redis_1        docker-entrypoint.sh redis ...   Exit 0
srcd-807c4ceaea185dbe21b14b6f7955d1722ddca312_sourced-ui_1   /entrypoint.sh                   Exit 137

$ ./sourced start
Starting bblfsh     ... done
Starting gitbase    ... done
Starting bblfsh-web ... done
Starting redis      ... done
Starting postgres   ... done
Starting sourced-ui ... done

We should add something similar to the engine install instructions: https://github.com/src-d/engine/#2-install-sourced-engine

The cli is going to be the same as already implemented for ghcollector:
#51

init for org should start ghsync as well:

1. run migration
2. pass a token and list of organizations
3. pass correct env variables to srcd-ui (use the image from src-d/sourced-ui#112 )
4. validate everything works together as expected

$ go run cmd/sourced/main.go prune
Stopping srcd-z28ty2hp_sourced-ui_1 ... done
....
$ go run cmd/sourced/main.go prune
lstat /Users/smacker/.sourced/workdirs/__active__: no such file or directory
exit status 1

we fixed it before by showing nice error message but now it's broken again.

Probably it's best to hide it.
https://docs.sourced.tech/community-edition

What is the best way right now to get query results in a jupyter notebook?

Example use case that was done with the source{d} Engine:
https://github.com/src-d/engine-analyses/blob/master/k8s_analysis.ipynb
(see In [3]: and In [43]: cells from the notebook)

The Initialize sourced community edition doc is missing the local argument in the local repositories section

$ sourced init </path/to/repositories>

should be

$ sourced init local </path/to/repositories>

This issue is to find a way to limit the containers resources so they don't freeze the host.
Ideally this can be done with settings in the compose file.

Otherwise gitcollector v0.0.1-beta.6 already supports GITCOLLECTOR_HALF_CPU=true, other containers may have similar confs.

To integrate https://github.com/src-d/gitcollector we need:

• Add gitcollector to docker-compose file, use env vars for config.
• Create new sub commands. We don't have a final decision on this, but this is easy to rename at any time.
  • sourced local init /path/to/repos: the current init. It will need to run docker-compose up excluding the service for gitcollector
  • sourced orgs init name-a,name-b,name-c: new sub command to launch gitcollector and ghsync
• Manage the workdir for orgs. It's probably better to separate local and orgs, so we could have:
  • ~/.sourced/workdirs/fs/ + path/to/dir
  • ~/.sourced/workdirs/orgs/ + <base64(org-a,org-b,org-c)>

Side note: we may want to get the organizations argument in sourced and order them alphabetically. This way we'll have the same workspace for src-d,bblfsh and bblfsh,src-d

Most of the idea(s) can be found in this Slack thread, but basically, I would like to be able to manage and especially exclude repositories with ease. I know this can also be done by adding filters on superset, but as a user I want something easier.

Feature proposals

• Add an --exclude flag on the command line. This would be especially useful when importing repos from organizations, but also when repo are stored locally, as one may just have centralized all repos but only want to analyze a part, and doesn't want or is not allowed to move them. This could take in a txt file, or just repo names
• Have this option available as well on the UI. The reasoning is that one may have already launched sourced-ce, it took a bit time to compute metrics, and suddenly he sees he forgot to remove some repo.
• Add an --exclude-forks boolean flag, that would exclude by default all forks. There is already another issue for this.
• Have a setting that would enable analysis of forked repositories data only after the moment it was forked

Right now the orgs command seems to only support, well, Github orgs, but definitely a common (given most people don't own orgs) and interesting (given you might want to check your Github "profile" in-depth) use case is to try it on your own Github user. Is it easy enough to extend orgs to also cover individual users?

The env vars do not apply anymore, and instead of documenting how to combine GITBASE_VOLUME_TYPE and GITBASE_VOLUME_SOURCE it's just better to assume the users will just use sourced.

carlosms-test-org,bblfsh and bblfsh,carlosms-test-org should be the same workdir.

sourced workdirs
  carlosms-test-org,bblfsh
* bblfsh,carlosms-test-org
  src-d,bblfsh

After #2 the install sub command is not only used for the first time you run source{d}

Now it's also used as a command to come back to a dir that was used previously, and maybe renaming it to init would be more intuitive. Or up, to make it clear that it's the equivalent of docker-compose up.

1. Document new command orgs init
2. Document how to obtain the token

Make CLI command list description more intuitive and consistent for new & existing users. Suggestion:

After:

source{d} Community Edition & Enterprise Edition CLI client

Help Options:
  -h, --help  Show this help message

Available commands:
  completion  Print a script that allows bash completion
  compose     Manage source{d} docker compose files
  init        Initialize source{d} to work on local or Github orgs datasets
  prune       Stop and remove components and resources
  sql         Open a MySQL client connected to a SQL interface for Git
  start       Start any stopped components
  status      Show the status of all components
  stop        Stop any running components
  version     Print the current version of the CLI client
  web         Open the web interface in your browser
  workdirs    List all working directories

Before:

source{d} CE and EE installer

Help Options:
  -h, --help  Show this help message

Available commands:
  completion  print bash completion script
  compose     Manage docker compose files
  init        Install and initialize containers
  prune       Stop and remove containers and resources
  sql         Open a MySQL client connected to gitbase
  start       Start stopped containers
  status      Shows status of the components
  stop        Stop running containers
  version     print version
  web         Open the web interface in your browser
  workdirs    List working directories

Originally posted by @EgorBu in src-d/sourced-ui#172 (comment)

A bit of feedback from empathy session
https://github.com/src-d/empathy-sessions/issues/41

We should not mix localhost & 127.0.0.1
https://github.com/src-d/sourced-ce#quick-start contains localhost when UI contains 127.0.0.1 - it requires to login one more time

We could have an automated way to upgrade sourced once it has been locally installed.

To update sourced, the steps to be (manually) done, are docmmented by
https://docs.sourced.tech/community-edition/learn-more/faq-troubleshooting#how-can-i-update-my-version-of-source-d-ce

(from @smacker slack thread)

At least version would be useful.

We should test the dashboard from src-d/sourced-ui#110 works.

After we implemented different workdirs prune command cleans only current workdir.
There should be a way to clean everything.
Better block this issue until #13 is resolved.

from https://github.com/src-d/empathy-sessions/issues/40

I got an unhandled error when opening the browser.
It was opened, but the error is still there.

$ uname -a
Linux 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux

$ sourced init orgs bblfsh --token=***

docker-compose working directory set to bblfsh
Starting srcd-ymjsznno_bblfsh_1     ... done
Starting srcd-ymjsznno_postgres_1   ... done
Starting srcd-ymjsznno_metadatadb_1   ... done
Starting srcd-ymjsznno_redis_1        ... done
Starting srcd-ymjsznno_bblfsh-web_1   ... done
Starting srcd-ymjsznno_gitbase_1      ... done
Starting srcd-ymjsznno_gitcollector_1 ... done
Starting srcd-ymjsznno_ghsync_1       ... done
Starting srcd-ymjsznno_sourced-ui_1   ... done

Initializing source{d}... ⠼

[13861:13884:0627/115634.989293:ERROR:browser_process_sub_thread.cc(217)] Waited 6 ms for network service
Opening in existing browser session.

I also got:

[25113:25127:0627/093059.362587:ERROR:browser_process_sub_thread.cc(221)] Waited 8 ms for network service

with my

$ uname -a
Linux sourced 4.4.0-146-generic #172-Ubuntu SMP Wed Apr 3 09:00:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

The docs should be clear enough that can be copy-pasted.
It should include details on how to run:

• a modified docker-compose file, optional if needed
• a sourced-ui built locally, optional if needed

Parent: #47

Keep in mind at the time of writing this issue neither gitcollecter nor ghsync support multi-orgs.

There are some limitations with how superset handles long-running sql queries, and it affects the dashboard charts.

This should be documented in a new section of the readme, explaining the kind of errors that can be expected, and how to refresh data when they are found.

One of the solutions might be to use env vars.
Related discussion: #54 (comment)

When analyzing sourced's codebase with the dashboard, huge repos that have been forked under src-d have made some visuals become irrelavant and even confuse the overview of the codebase/company's activity. For example,

Here, we include the src-d/or-tools huge C repo that somebody forked one day, in the analysis, and I think by default we should not. Also, another consequence is that, all the contributors of those forsks are going to be analyzed in the dashboard and thus be presented as the analyzed company's contributors whereas they never did one contribution to any the company's repos.

I try to filter them with gitbase one day in vain. So, to have a relevant analysis of a company, I fetch the data myself excluding forks with the GitHub API

#!/bin/bash

output_directory="/home/waren/tmp/"
repos=$(curl -s "https://api.github.com/orgs/src-d/repos?page=1&per_page=100" | jq '.[].url' | tr -d '"')
for repo in $repos; do
    is_fork=$(curl -s $repo | jq '.fork')
    if [ $is_fork == "false" ]; then
        html_url=$(curl -s $repo | jq '.html_url' | tr -d '"')
        git clone $html_url $output_directory
    fi
done

Handle empty .srcd when other commands are executed before install:

$ rm -rf ~/.srcd
$ sandbox-ce stop
chdir /home/cmartin/.srcd/workdirs/__active__: no such file or directory

Discussion started here: https://github.com/src-d/sourced-ce/pull/75/files#r295922705

We don't have any solution for it now but I hope we could discuss it later and improve the documentation.

From the design doc:

As a user I want to choose the dataset that I will be working with so that I can switch between sets of repositories for different usages

From a previous conversation:

@carlosms

One thing to discuss. If you change the repositories dir, should the Superset DB also change?

i.e. if I start the sandbox with the repos in bblfsh org and I create a dashboard with charts that show certain metrics.
Then I change the repositories dir to work with src-d repos.
When I log in back to Superset, is it expected to have the same dashboard with updated information, or is it supposed to be a clean installation?

@mcuadros

The same dashboards, you should keep the postgresql

But in a later informal discussion it was mentioned that the other scenario might be preferred.
Needs input from @src-d/product @mcuadros @smola

Parent: #47

We need to validate incorrect input early because we can't catch errors when containers are started already.

Currently, there is no way to delete workdir except manually rm -rf directory.
I think prune should delete everything including workdir.

Also right now it's possible to run prune multiple times. On the second time it prints:

Removing network srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_default
WARNING: Network srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_default not found.
Removing volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_gitbase_indexes
WARNING: Volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_gitbase_indexes not found.
Removing volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_postgres
WARNING: Volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_postgres not found.
Removing volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_redis
WARNING: Volume srcd-2ac536d6a6bb44bab48c58b1797bcafe76bd6ef0_redis not found.

Yes/No?

This issue needs more details, consider this an open conversation.

So far, my understanding from informal conversations and chats is that the sandbox-ce command should be able to download docker-compose.yml files. The goal is to be able to update the sandbox deployment by having new docker-compose files uploaded, without having to release a new binary of the tool.

Without more details I think it makes sense to implement a first basic version, the command should be able to:

• Download the latest docker-compose file from the master branch
• Store the compose file somewhere. The user might want to run docker-compose commands themselves, so we need to make the .yml file available.
• For maximum flexibility we probably want to allow the user to specify the .yml file to use. For example, somebody might want to customize the yml file, but still be able to use our sandbox command for initialization and start.