kobotoolbox / kobo-install Goto Github PK

A command-line installer for setting up and running KoboToolbox on a remote server or local computer, using kobo-docker.

Python 86.35% Smarty 12.50% Shell 1.15%

kobo-install's People

Contributors

Stargazers

Watchers

Forkers

nanusdad stopi arnasan diegoasencio96 yubarajpoudel neraunzaran mouhammade frankm1 djmmax steve-it iyusa tomislavr dzangpo avi-moskoff dmitrisichinava d0ng4667 soumares tiffanyneel manuhmutua adam704a localsnet jonathanf ellyleting ceduardodch samirnoman dosanjosfilho aspiryg internetofwater molecode dememan ww2zyl simasafa2020 grupocato ciarwanda odoema sunainagupta2020 vishalreddysarasani manu-20-mvp incentro-africa noxreborn deoafema djangrang phil-99 tobiasmedia cbareiro theabstact237 admariner geodatahuasi raeesaroj sg3847 digitalgreenorg djabonou chilz6 serrangel fkhorchani zite-io obwoya drfaustusfade joshuaberetta skrooloose toelintun ubpd ohari5336 aldrin24 nexionarchived urjashee buralp dolpotech salanesi devhypergeo digireg-id devgateway sabrisabah bhavikpatel luciekimotho dozie9 alexsierrarodriguez israaar stephencoduor saifkhan-m kobid589 anwar31dec bhavyakamboj reachnetworks md2i-madagascar simatei maheha azizur77 owenbeers-agriai sandeshsays cocoemily mamadou96 wasimalsabbouh simatei-tech franklwambo jiehoes santitamariz emmanuelnegash arnoldnjogu contribute4ever

kobo-install's Issues

Nginx configure

I use Plesk panel software to manage webhosting, and it include own Nginx frontend. So I need correct configure one with backend Nginx from kobo. When I add domain to my frontend Nginx and proxypass it to backend Nginx, in browser I've got

Bad Request (400)

How correctly setup Nginx in my current environment? Please suggest me in right direction.

Warn that upgrade to two databases could be slow

See #66 (review).

Explain that it could be an extended outage for a large database
Note that this can be delayed by downgrading to 2.019.52-final-shared-database
Prompt to continue

Check for non-zero exit codes

When run_command() calls something and it returns a non-zero exit code, consider failing quickly instead of waiting for the 10-minute timeout.

See #5, #6, for examples of situations where this would be helpful.

ImportError: Missing redis library (pip install redis).

Hi all,

I use kobo-install to deploy Kobotoolbox on a fresh vm with the following setting.

Ubuntu 16.04 server
Public and local DNS configured properly

But finally it throws the following error.

Something went wrong! Please look at docker logs.

when i see the log detail for KPI it shows the following error

 ImportError: Missing redis library (pip install redis)
 Traceback (most recent call last):
   File "/usr/local/bin/celery", line 11, in <module>
     sys.exit(main())
   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
     _main()
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
     cmd.execute_from_commandline(argv)
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
     super(CeleryCommand, self).execute_from_commandline(argv)))
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 275, in execute_from_commandline
     return self.handle_argv(self.prog_name, argv[1:])
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in handle_argv
     return self.execute(command, argv)
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 420, in execute
     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 223, in run_from_argv
     return self(*args, **options)
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 238, in __call__
     ret = self.run(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 257, in run
     **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 101, in __init__
     self.setup_instance(**self.prepare_args(**kwargs))
   File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 124, in setup_instance
     self.should_use_eventloop() if use_eventloop is None
   File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 243, in should_use_eventloop
     self._conninfo.transport.implements.asynchronous and
   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 844, in transport
     self._transport = self.create_transport()
   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 576, in create_transport
     return self.get_transport_cls()(client=self)
   File "/usr/local/lib/python2.7/dist-packages/kombu/transport/redis.py", line 1016, in __init__
     raise ImportError('Missing redis library (pip install redis)')
 ImportError: Missing redis library (pip install redis)

Can anyone please help.

Thank you
Nuredin

Could not write configuration file error

I am getting the Could not write configuration file error when I am trying to create a deployment of Kobo on a Linux server. Do you have any idea of what could be causing this error? I tried manually providing permissions to the folder using chmod but hasn't worked. How can I go deeper into the error?

Only pull `kobo-docker` when doing an `--upgrade`

@noliveleger, you asked me to note this. Right now, run.py always pulls kobo-docker, but this probably shouldn't be done unless the user explicitly wants to upgrade.

When I run $ python run.py I get ERROR: Couldn't connect to Docker daemon - you might need to run 'docker-machine start default'

Hi I am running Ubunt VM using HyperV on Windows 10 to test kobotoolbox installation
I have installed docker and docker compose and started and enabled docker
When I run $ python run.py I get
ERROR: Couldn't connect to Docker daemon - you might need to run 'docker-machine start default'

What does this mean. Is it a relevant error. What must I do next please?

make it possible to use existing postgresql database server

It'd be great if we could optionally use an existing postgresql database server instead of always setting up a docker container for it

Error during installation : OSError: [Errno 2] No such file or directory

MacOS: 10.14.5 Beta (18F131a)

Cloning into '/Users/john/projects/kobo-install/kobo-docker'...
remote: Enumerating objects: 70, done.
remote: Counting objects: 100% (70/70), done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 2823 (delta 33), reused 38 (delta 17), pack-reused 2753
Receiving objects: 100% (2823/2823), 2.08 MiB | 562.00 KiB/s, done.
Resolving deltas: 100% (1725/1725), done.
Already on 'master'
From https://github.com/kobotoolbox/kobo-docker
 * branch            master     -> FETCH_HEAD
Traceback (most recent call last):
  File "run.py", line 50, in <module>
    run(force_setup=True)
  File "run.py", line 36, in run
    Command.start()
  File "/Users/john/projects/kobo-install/helpers/command.py", line 217, in start
    cls.stop(output=False, frontend_only=frontend_only)
  File "/Users/john/projects/kobo-install/helpers/command.py", line 311, in stop
    CLI.run_command(backend_command, config.get("kobodocker_path"))
  File "/Users/john/projects/kobo-install/helpers/cli.py", line 88, in run_command
    stdout = subprocess.check_output(command, universal_newlines=True, cwd=cwd)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Need to raise Enketo's Text field character limit

Hi,

I need to raise the character length limits in text fields from the default 2000 to something much bigger (see https://groups.google.com/forum/#!topic/enketo-users/5qYlmC7yxNc).

I've been running Kobo from docker for field projects over the past 3 years now, and we never ran into the character length issue before. A few points:
(1) text character length seems like a really important config that it would great to specify in the kobo-install options perhaps a TODO on this?
(2) [SOLVED -- See below] I now have a working deployment from kobo-install, with people entering data now. I need to update this now without risking my current deployment.

Thanks in advance!

EDIT: I figured out where to edit the config in /kobo-deployments/enketo_express/config.json and I used the python run.py --stop and then python run.py command to stop and restart the docker containers. I still think it will be useful to make this important bit of configuration easier to edit. Thanks!

Timeout waiting for host : mongo

Hi.

I tried kobo-install from fresh clone and installation. Have been on this all day.

Whenever the installation is finished, it said something like check the docker logs. And here is what I found:

kpi_1             | *** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
kpi_1             | *** Running /etc/my_init.d/01_wait_for_mongo.bash...
kpi_1             | Waiting for container `mongo`.
kpi_1             | 2019/03/27 14:03:12 Waiting for host: tcp://mongo.kobo.private:27017
kobocat_1         | 2019/03/27 14:03:42 Timeout after 40s waiting on dependencies to become available: [tcp://mongo.kobo.private:27017]
kobocat_1         | *** /etc/my_init.d/01_wait_for_mongo.bash failed with status 1
kobocat_1         |
kobocat_1         | *** Killing all processes...

Is there anything I missed?

For the record, I'm installing with following option:

Workstation
Developer mode
Tried both python2.7 and python3.7

Error with "select_one_from_file" type

Hi All,

I just used kobo-install, and it was a much less painful process than my deployment of Kobotoolbox last year using just the Kobo-Docker project, so THANKS!

I'm having an issue however with errors in forms that have the "select_one_from_file" type field. When I preview my form I get a "invalid data_type: select_one_from_file wine.csv". Here's some background:

I followed the instructions here: https://community.kobotoolbox.org/t/extremely-long-list-of-choices/3259/2
My form validates with https://opendatakit.org/xlsform/
I had a similar issue last year with Kobotoolbox, but the issue is marked as resolved: kobotoolbox/kobo-docker#183

So, does kobo-install deploy an older version of kobo-docker? If so, can I change my installation so I can get my "select_one_from_file" field types working? Or is there another issue?

Thanks!

Firewall can block frontend containers from connecting to backend containers

@duvld said:

make sure if running on Linux to either disable firewall or make a rule to allow kpi, if firewall blocks kpi from connecting to backend strange errors will come up.

Maybe we should write something about this in a readme or in the run.py -s prompts.

Originally posted by @duvld in #76 (comment)

Secure backend ports

By default PostgreSQL, Redis, Mongo ports are listening on 0.0.0.0.
It should be restricted to private ip.

For references:
kobotoolbox/kobo-docker#249

IP change is not detected in `server` mode

Steps to Reproduce

Install in server mode (wait to start). python run.py -s
Change local_interface_ip in .run.conf to 1.1.1.1
Rerun config mode python run.py -s

Expected behavior

It should detect that ip has changed.

Actual behavior

Starts without asking which interface to choose.

Unable to initiate kobotoolbox in an EC2 instance

I am trying to install the kobotoolbox in a fresh EC2 instance of AWS. I follow the below steps:

apt-get update
apt-get upgrade
3. install docker
4. install docker-composer
git clone https://github.com/kobotoolbox/kobo-install.git
sudo python3 run.py

The service is never initiated (attached screenshot) and in the log file I see the below error:

ERROR: .FileNotFoundError: [Errno 2] No such file or directory: './docker-compose.frontend.yml'

Fresh install not working

Hi, I just tried to install this and it's not working.
Here is my config :

Alpine Linux 3.8.2 (last version, up to date) running on HyperV
docker 18.09.1-r0
docker-compose 1.23.2

Here the output :

kobo [/home/kobo-install]# python run.py
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to `kobo-install`!                                    ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/home/kobo-docker]:
Do you want to see advanced options?
        1) Yes
        2) No
[2]:
What kind of installation do you need?
        1) On your workstation
        2) On a server
[2]:
Public domain name [kobo.local]: mydomain.tld
KPI sub domain [kf]: koboform
KoBoCat sub domain [kc]: kobocat
Enketo Express sub domain name [ee]: enketo
Do you use a reverse proxy or a load balancer?
        1) Yes
        2) No
[1]:
Use HTTPS?
Please note that certificate has to be installed on the load balancer!
        1) Yes
        2) No
[1]:
Internal port used by reverse proxy?
[80]:
SMTP server: mail.mydomain.tld
SMTP port [25]: 587
SMTP user: [email protected]
SMTP password: xxxx
Use TLS?
        1) True
        2) False
[1]:
From email address [[email protected]]: [email protected]
Super user's username [super_admin]:
Super user's password [^H$kNO-Tdi1Aq%EB]: xxxx
Do you want to activate backups?
        1) Yes
        2) No
[2]:
Cloning into '/home/kobo-docker'...
remote: Enumerating objects: 2546, done.
remote: Total 2546 (delta 0), reused 0 (delta 0), pack-reused 2546
Receiving objects: 100% (2546/2546), 1.84 MiB | 1.04 MiB/s, done.
Resolving deltas: 100% (1580/1580), done.
Switched to a new branch 'kobo-install'
From https://github.com/kobotoolbox/kobo-docker
 * branch            kobo-install -> FETCH_HEAD
Removing network kobo-docker_default
WARNING: Network kobo-docker_default not found.
Removing network kobo-docker_kobo-fe-network
WARNING: Network kobo-docker_kobo-fe-network not found.
Launching environment
Creating network "kobo-docker_default" with the default driver
Pulling mongo (mongo:3.4)...
Pulling redis_main (redis:3.2)...
Pulling postgres (mdillon/postgis:9.5)...
Pulling rabbit (kobotoolbox/rabbit:latest)...
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_redis_main_1  ... done
Creating kobo-docker_redis_cache_1 ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_redis_cache_1, kobo-docker_rabbit_1, kobo-docker_redis_main_1, kobo-docker_postgres_1, kobo-docker_mongo_1) for this project. If you removed or renamed this service in your compose file, you
can run this command with the --remove-orphans flag to clean it up.
Pulling nginx (kobotoolbox/nginx:latest)...
Pulling kobocat (kobotoolbox/kobocat:2.018.48)...
Pulling enketo_express (kobotoolbox/enketo-express-extra-widgets:1.72.2)...
Pulling kpi (kobotoolbox/kpi:2.018.48)...
Creating kobo-docker_enketo_express_1 ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_kobocat_1        ... done
Waiting for environment to be ready. It can take a few minutes.
............................................................
`KoBoToolbox` has not started yet, wait for another 600 minutes!
............................................................
`KoBoToolbox` has not started yet, sometimes frontend containers can not communicate with backend containers.
Let's restart frontend containers.

Stopping kobo-docker_nginx_1          ... done
Stopping kobo-docker_kpi_1            ... done
Stopping kobo-docker_kobocat_1        ... done
Stopping kobo-docker_enketo_express_1 ... done
WARNING: Found orphan containers (kobo-docker_redis_cache_1, kobo-docker_rabbit_1, kobo-docker_redis_main_1, kobo-docker_postgres_1, kobo-docker_mongo_1) for this project. If you removed or renamed this service in your compose file, you
can run this command with the --remove-orphans flag to clean it up.
Removing kobo-docker_nginx_1          ... done
Removing kobo-docker_kpi_1            ... done
Removing kobo-docker_kobocat_1        ... done
Removing kobo-docker_enketo_express_1 ... done
Removing network kobo-docker_kobo-fe-network
Launching frontend containers
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_redis_cache_1, kobo-docker_rabbit_1, kobo-docker_redis_main_1, kobo-docker_postgres_1, kobo-docker_mongo_1) for this project. If you removed or renamed this service in your compose file, you
can run this command with the --remove-orphans flag to clean it up.
Creating kobo-docker_enketo_express_1 ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_kobocat_1        ... done
...........................................................
Something went wrong! Please look at docker logs

The docker logs can be accessed here : http://dl.pasteur.la/?t=8755da5a95e39d223a3109829c5ad5cd
The syslog output /var/log/docker.log is here : http://dl.pasteur.la/?t=f4e53dd70f9c1cf9e7f51b12ab89054f
Any help would be appreciated, thanks.

Check working directory

If I do this (i.e. I do not cd kobo-install):

john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobo-install
Cloning into 'kobo-install'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 549 (delta 19), reused 25 (delta 11), pack-reused 513
Receiving objects: 100% (549/549), 129.61 KiB | 3.32 MiB/s, done.
Resolving deltas: 100% (364/364), done.
john@scrappy:/tmp/kit$ python kobo-install/run.py 
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to `kobo-install`!                                    ║

Things will eventually fail:

Do you want to review your /etc/hosts file before overwriting it?
	1) Yes
	2) No
[2]: 
ERROR: .IOError: [Errno 2] No such file or directory: './docker-compose.backend.master.override.yml'

ERROR: .IOError: [Errno 2] No such file or directory: './docker-compose.frontend.override.yml'

Launching environment
ERROR: .IOError: [Errno 2] No such file or directory: './docker-compose.backend.master.override.yml'

ERROR: .IOError: [Errno 2] No such file or directory: './docker-compose.frontend.override.yml'

Waiting for environment to be ready. It can take a few minutes.
..............

It doesn't appear like anything is happening at this point: no containers have been created, no process is using much of the CPU, etc.

User Registration

Hello,

I have trying to locate that is there a way we can remove "Create an Account" link from home page/login panel and somehow create that link inside admin panel? In my case, I don't want everyone to get registered with my Kobo but I want to add the user from Admin Panel and assign them roles accordingly?

Let me know if there is a way or any workaround for this problem.

Thanks,

Backups cannot be activated independently

When user activate backups, all backups kobomedia, postgresql, mongo, redis are planned to run.

Users should be able to choose whether they want to run each backup or not.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 689: ordinal not in range(128)

Install error afetr commit e450d53:
python run.py -s

root_:/home/kubrick/www/kobo-install/templates/kobo-deployments/enketo_express,
destination_directory_:/home/kubrick/www/kobo-deployments/enketo_express,
filenames_:[u'config.json.tpl'],
--------------------
t:<string.Template object at 0x7f9fd5ba7590>,
template_variables:{u'PRIVATE_DOMAIN_NAME': u'kobo.private', u'GOOGLE_API_KEY': u'', u'SMTP_PASSWORD': u'', u'SMTP_USE_TLS': True, u'REDIS_CACHE_PORT': u'6380', u'POSTGRES_PASSWORD': u'Y~GbI+sS!pyOEG+x', u'KOBOCAT_SUBDOMAIN': u'kc', u'SMTP_HOST': None, u'MASTER_BACKEND_IP': u'192.168.31.217', u'MONGO_BACKUP_SCHEDULE': u'0 1 * * 0', u'USE_KPI_DEV_MODE': u'#', u'AWS_BACKUP_MONTHLY_RETENTION': u'12', u'SOFT_LIMIT': 134217728, u'KPI_PATH': u'', u'KPI_RAVEN_DSN': u'', u'SMTP_USER': u'', u'KOBO_SUPERUSER_USERNAME': u'super_admin', u'NGINX_EXPOSED_PORT': u'80', u'USE_BACKUP': u'#', u'AWS_REDIS_BACKUP_MINIMUM_SIZE': u'5', u'AWS_MONGO_BACKUP_MINIMUM_SIZE': u'50', u'KOBOFORM_SUBDOMAIN': u'kf', u'WSGI_SERVER': u'uWSGI', u'SMTP_PORT': u'25', u'USE_AWS': u'#', u'LETSENCRYPT_EMAIL': None, u'USE_DNS': u'', u'KPI_RAVEN_JS_DSN': u'', u'MAX_REQUESTS': u'512', u'POSTGRES_BACKUP_SCHEDULE': u'0 2 * * 0', u'ENKETO_ENCRYPTION_KEY': u'060bc0642c6d97041b8532f0f1a9d6356f1715241e112ab89522e47a31064420e8eb5c8a65752d16d8d26342b91bc881d5c04b815ee5ed719f0f058e', u'KPI_DEV_BUILD_ID': u'', u'PUBLIC_DOMAIN_NAME': u'kobo.local', u'USE_KC_DEV_MODE': u'#', u'INTERNAL_DOMAIN_NAME': u'kobo.internal', u'KOBO_SUPERUSER_PASSWORD': u'VgF9vbX4KPF+zu', u'AWS_BACKUP_BUCKET_DELETION_RULE_ENABLED': u'False', u'POSTGRES_APP_PROFILE': u'Mixed', u'MONGO_PORT': u'27017', u'USE_X_FORWARDED_HOST': u'#', u'KC_PATH': u'', u'PROTOCOL': u'http', u'ENKETO_SUBDOMAIN': u'ee', u'NGINX_PUBLIC_PORT': u'80', u'OVERRIDE_POSTGRES_SETTINGS': u'#', u'AWS_BACKUP_WEEKLY_RETENTION': u'4', u'AWS_BACKUP_BUCKET_NAME': u'', u'POSTGRES_RAM': u'2', u'POSTGRES_DB': u'kobotoolbox', u'REDIS_MAIN_PORT': u'6379', u'WORKERS_START': u'1', u'POSTGRES_REPLICATION_PASSWORD': u'u739sxpt+fGGaxJ', u'USE_AWS_BACKUP': u'#', u'AWS_BUCKET_NAME': u'', u'POSTGRES_SETTINGS': u'', u'AWS_POSTGRES_BACKUP_MINIMUM_SIZE': u'50', u'DEBUG': False, u'WORKERS_MAX': u'2', u'AWS_ACCESS_KEY_ID': u'', u'USE_HTTPS': u'#', u'USE_MEDIA_BACKUP': u'#', u'ENKETO_API_TOKEN': u'9d7b980debf40401a190f0e610f659aa52b3563a58465070480550c3fa8bb699403da84bbfcaa043bc2b65042e33c14b5ffcb711e43b0f5f46545455', u'POSTGRES_USER': u'kobo', u'USE_PUBLIC_DNS': u'#', u'KOBOCAT_MEDIA_BACKUP_SCHEDULE': None, u'GOOGLE_UA': u'', u'LOCAL_INTERFACE_IP': u'192.168.31.217', u'AWS_BACKUP_YEARLY_RETENTION': u'2', u'POSTGRES_PORT': u'5432', u'KC_DEV_BUILD_ID': u'', u'DJANGO_SECRET_KEY': u'9860f89729b9304bcf26507ed946f5eca0eb0fa05e0b01fb', u'AWS_BACKUP_DAILY_RETENTION': u'30', u'AWS_BACKUP_UPLOAD_CHUNK_SIZE': u'15', u'POSTGRES_BACKUP_FROM_SLAVE': u'#', u'DEFAULT_FROM_EMAIL': u'[email protected]', u'AWS_SECRET_ACCESS_KEY': u'', u'REDIS_BACKUP_SCHEDULE': u'0 3 * * 0', u'USE_PRIVATE_DNS': u'', u'KOBOCAT_RAVEN_DSN': u''},
++++++++++++++++++++
Traceback (most recent call last):
  File "run.py", line 55, in <module>
    run(force_setup=True)
  File "run.py", line 28, in run
    Template.render(config)
  File "/home/kubrick/www/kobo-install/helpers/template.py", line 159, in render
    write_templates(root, destination_directory, filenames)
  File "/home/kubrick/www/kobo-install/helpers/template.py", line 148, in write_templates
    f.write(t.substitute(template_variables))
  File "/usr/lib/python2.7/string.py", line 176, in substitute
    return self.pattern.sub(convert, self.template)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 689: ordinal not in range(128)

Installation doesn't work

The environment is a fresh DigitalOcean Droplet with ubuntu 18.04.3 | 2 vCPU | 2 Gb RAM and the setup up doesn't finish:

python3 run.py --setup
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to KoBoInstall!                                       ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/opt/kobo-docker]: 
Please confirm path [/opt/kobo-docker]
	1) Yes
	2) No
[1]: 
Do you want to see advanced options?
	1) Yes
	2) No
[2]: 
What kind of installation do you need?
	1) On your workstation
	2) On a server
[2]: 
Public domain name [public.domain]: 
KPI sub domain [kf]: 
KoBoCat sub domain [kc]: 
Enketo Express sub domain name [ee]: 
Do you want to use HTTPS?
	1) Yes
	2) No
[1]: 
╔════════════════════════════════════════════════════════════════════╗
║ Please note that certificates must be installed on a reverse-proxy ║
║ or a load balancer.                                                ║
║ KoBoInstall can install one, if needed.                            ║
╚════════════════════════════════════════════════════════════════════╝
Auto-install HTTPS certificates with Let's Encrypt?
	1) Yes
	2) No - Use my own reserve-proxy/load-balancer
[1]: 
╔════════════════════════════════════════════════╗
║ Domain names must be publicly accessible.      ║
║ Otherwise Let's Encrypt won't be able to valid ║
║ your certificates.                             ║
╚════════════════════════════════════════════════╝
Email address for Let's Encrypt [email]: 
Please confirm [email]
	1) Yes
	2) No
[1]: 
SMTP server [public.domain]: 
SMTP port [465]: 
SMTP user [[email protected]]: 
SMTP password [public.domain.pass]: 
Use TLS?
	1) True
	2) False
[1]: 
From email address [[email protected]]: 
Super user's username [super_admin]: 
Super user's password [random.pass]: 
Do you want to activate backups?
	1) Yes
	2) No
[2]: 
Already on 'master'
From https://github.com/kobotoolbox/kobo-docker
 * branch            master     -> FETCH_HEAD
Existing data found for kf.public.domain,kc.public.domain,ee.public.domain. Continue and replace existing certificate? (y/N) 
Stopping kobo-docker_mongo_1       ... done
Stopping kobo-docker_redis_cache_1 ... done
Stopping kobo-docker_redis_main_1  ... done
Stopping kobo-docker_postgres_1    ... done
WARNING: Found orphan containers (kobo-docker_kpi_1, kobo-docker_nginx_1, kobo-docker_kobocat_1, kobo-docker_enketo_express_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing kobo-docker_mongo_1       ... done
Removing kobo-docker_redis_cache_1 ... done
Removing kobo-docker_redis_main_1  ... done
Removing kobo-docker_postgres_1    ... done
Removing network kobo-docker_default
Stopping kobo-docker_kpi_1            ... done
Stopping kobo-docker_nginx_1          ... done
Stopping kobo-docker_kobocat_1        ... done
Stopping kobo-docker_enketo_express_1 ... done
Removing kobo-docker_kpi_1            ... done
Removing kobo-docker_nginx_1          ... done
Removing kobo-docker_kobocat_1        ... done
Removing kobo-docker_enketo_express_1 ... done
Removing network kobo-docker_kobo-fe-network
Stopping nginx-certbot_certbot_1 ... done
Stopping nginx-certbot_nginx_1   ... done
Removing nginx-certbot_certbot_1 ... done
Removing nginx-certbot_nginx_1   ... done
Removing network nginx-certbot_default
Launching environment
Creating network "kobo-docker_default" with the default driver
Creating kobo-docker_redis_main_1  ... done
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_redis_cache_1 ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_redis_cache_1, kobo-docker_mongo_1, kobo-docker_postgres_1, kobo-docker_redis_main_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_enketo_express_1 ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_kobocat_1        ... done
Creating network "nginx-certbot_default" with the default driver
Creating nginx-certbot_nginx_1   ... done
Creating nginx-certbot_certbot_1 ... done
Waiting for environment to be ready. It can take a few minutes.
............................................................
`KoBoToolbox` has not started yet. This is can be normal with low CPU/RAM computers.

Wait for another 600 seconds?
	1) Yes
	2) No
[1]: ......................................

And then after minutes a message ask for 600 minutes more and never end.

Obvioslly public.domain is a domain with DNS entry for kf. kc and ee.

In /etc/hosts I put these line:
x.x.x.x kf.domain.internal kf.docker.internal

The firewall is disabled and all ports must be availables.

I try:
curl -I to all nginx containers at http and https or to the publicIP and always get the same response:
curl: (7) Failed to connect to XXXX port XX: Connection refused

docker ps

CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS                         PORTS                              NAMES
9159c2181cf7        certbot/certbot                                                        "/bin/sh -c 'trap ex…"   14 minutes ago      Up 14 minutes                  80/tcp, 443/tcp                    nginx-certbot_certbot_1
ed1d18dbd8ff        nginx:1.15-alpine                                                      "/bin/sh -c 'while :…"   14 minutes ago      Restarting (1) 7 seconds ago                                      nginx-certbot_nginx_1
39f939975498        kobotoolbox/enketo-express-extra-widgets:1.77.0-jnm-grunt-workaround   "/sbin/my_init"          14 minutes ago      Up 14 minutes                  8005/tcp                           kobo-docker_enketo_express_1
cbf456186f81        kobotoolbox/kobocat:2.019.34                                           "/sbin/my_init"          14 minutes ago      Up 3 minutes                   8000/tcp                           kobo-docker_kobocat_1
f9288095cde8        kobotoolbox/nginx:latest                                               "/sbin/my_init"          14 minutes ago      Up 14 minutes                  443/tcp, 0.0.0.0:8080->80/tcp      kobo-docker_nginx_1
e5dca395327a        kobotoolbox/kpi:2.019.35b-hotfix2                                      "/sbin/my_init"          14 minutes ago      Up 3 minutes                   8000/tcp                           kobo-docker_kpi_1
60267d4af28b        redis:3.2                                                              "docker-entrypoint.s…"   14 minutes ago      Up 14 minutes                  6379/tcp, 0.0.0.0:6380->6380/tcp   kobo-docker_redis_cache_1
9666686f8745        mongo:3.4                                                              "docker-entrypoint.s…"   14 minutes ago      Up 14 minutes                  0.0.0.0:27017->27017/tcp           kobo-docker_mongo_1
39c49cadda2f        mdillon/postgis:9.5                                                    "docker-entrypoint.s…"   14 minutes ago      Up 14 minutes                  0.0.0.0:5432->5432/tcp             kobo-docker_postgres_1
d55eb75e9b21        redis:3.2                                                              "docker-entrypoint.s…"   14 minutes ago      Up 14 minutes                  0.0.0.0:6379->6379/tcp             kobo-docker_redis_main_1```

I don't know what is wrong.

I really appreciate your help.

Fresh install on AWS EC2 ubuntu always giving message wait for 600 seconds

Hello, I am trying for almost 3 days now to install Kobo Toolbox but everytime regardless of the settings I choose, it always ending up as below:

_Waiting for environment to be ready. It can take a few minutes.
............................................................
KoBoToolbox has not started yet. This is can be normal with low CPU/RAM computers.

Wait for another 600 seconds?
1) Yes
2) No
[1]: 2

Sometimes frontend containers can not communicate with backend containers.
Restarting the frontend containers usually fixes it.

Do you want to try?
1) Yes
2) No
[1]: 1_

And then another wait for 600 seconds, I am not really sure what to do. running python3 run.ph --logs doesn't show much either.

I am trying to install it on AWS EC2 ubuntu Instance having 50 GB of Hard Disk and 4 GB RAM.

Any help would be highly appreciated.

Error nginx-certbot_default - Ubuntu 18.04

Creating network "nginx-certbot_default" with the default driver
Pulling certbot (certbot/certbot:)...
Pulling nginx (nginx:1.15-alpine)...
Creating nginx-certbot_nginx_1 ... done
An error has occurred

If they do not exist, clone `kpi` and `kobocat` for developer mode

Trying to use developer mode without having already cloned kpi and kobocat results in a failure:

john@scrappy:/tmp/kit$ find -delete
john@scrappy:/tmp/kit$ ls -la
total 1080
drwxr-xr-x  2 john john    4096 Jan 19 18:11 .
drwxrwxrwt 26 root root 1097728 Jan 19 18:11 ..
john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobo-install
Cloning into 'kobo-install'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 549 (delta 19), reused 25 (delta 11), pack-reused 513
Receiving objects: 100% (549/549), 129.61 KiB | 2.59 MiB/s, done.
Resolving deltas: 100% (364/364), done.
john@scrappy:/tmp/kit$ cd kobo-install/
john@scrappy:/tmp/kit/kobo-install$ python run.py 
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to `kobo-install`!                                    ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/tmp/kit/kobo-docker]: 
Do you want to see advanced options?
	1) Yes
	2) No
[2]: 1
What kind of installation do you need?
	1) On your workstation
	2) On a server
[2]: 1
Please choose which network interface you want to use?
	other) Other
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
[wlan0]: other
	lo) 127.0.0.1
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
	docker0) 172.17.0.1
[wlan0]: docker0
SMTP server: 
SMTP port [25]: 
SMTP user: 
From email address [[email protected]]: 
Super user's username [super_admin]: 
Super user's password [F!~mb7lroBAblJ]: 
Docker Compose prefix? (leave empty for default): 
Web server port?
[80]: 
Developer mode?
	1) Yes
	2) No
[2]: 1
╔═══════════════════════════════════════════════════════════╗
║ Where are the files located locally? It can be absolute   ║
║ or relative to the directory of the installation.         ║
║ Leave empty if you don't need to overload the repository. ║
╚═══════════════════════════════════════════════════════════╝
KoBoCat files location: /tmp/kit/kobocat
KPI files location: /tmp/kit/kpi
Enable DEBUG?
	1) True
	2) False
[2]: 1
Postgres database [kobotoolbox]: 
Postgres user [kobo]: 
Postgres password [H6%ne-B2$@$D]: 
Do you want to tweak PostgreSQL settings?
	1) Yes
	2) No
[2]: 
Do you want to customize service ports?
	1) Yes
	2) No
[2]: 
Do you want to use AWS S3 storage?
	1) Yes
	2) No
[2]: 
Google Analytics Identifier: 
Google API Key: 
Intercom App ID: 
Do you want to use Sentry?
	1) Yes
	2) No
[2]: 
Do you want to tweak uWSGI settings?
	1) Yes
	2) No
[2]: 
Do you want to activate backups?
	1) Yes
	2) No
[2]: 
Cloning into '/tmp/kit/kobo-docker'...
remote: Enumerating objects: 2546, done.
remote: Total 2546 (delta 0), reused 0 (delta 0), pack-reused 2546
Receiving objects: 100% (2546/2546), 1.84 MiB | 6.31 MiB/s, done.
Resolving deltas: 100% (1580/1580), done.
Switched to a new branch 'kobo-install'
From https://github.com/kobotoolbox/kobo-docker
 * branch            kobo-install -> FETCH_HEAD
╔═══════════════════════════════════════════════════════════════════╗
║ Administrative privileges are required to update your /etc/hosts. ║
╚═══════════════════════════════════════════════════════════════════╝
Do you want to review your /etc/hosts file before overwriting it?
	1) Yes
	2) No
[2]: 
Removing network kobo-docker_default
WARNING: Network kobo-docker_default not found.
ERROR: build path /tmp/kit/kobocat either does not exist, is not accessible, or is not a valid URL.

Launching environment
Creating network "kobo-docker_default" with the default driver
Pulling mongo (mongo:3.4)...
Pulling redis_main (redis:3.2)...
Pulling postgres (mdillon/postgis:9.5)...
Pulling rabbit (kobotoolbox/rabbit:latest)...
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_redis_main_1  ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_redis_cache_1 ... done
ERROR: build path /tmp/kit/kobocat either does not exist, is not accessible, or is not a valid URL.

Waiting for environment to be ready. It can take a few minutes.
............

How to export the data safely?

Hi there!

Last year I've installed KoboToolbox using kobo-install.
After some time (and the help of https://github.com/noliveleger - see #9), I managed to make it run.

After some time, I read that it wasn't possible to upgrade, because the database changed.
So we are suppose to restart the installation from scratch.

???!!!

Okaaayyy...

Despite the awkward feeling, I've asked my users to think about exporting all their data and their forms, because I was planning to install the last version, then figure out a way to insert the old data back in the system.

But for some reason, the interface now refuses to export the data of all projects in any format.
These are last few months of works by our scientists and technicians over all our departments, we really cannot afford to lose the data.

Is their a command I can simply run to extract a dump of the database from docker?
It's not the best solution I would like to go with. But if it allows us to save the data, then I will try to browse the dump and to rebuild a proper extract.

Currently, the KoboToolbox system seems to work, except that we cannot export data.
When I run the python run.py --logs, it's like an infinite loop of errors:

nginx_1           | Overwrite default nginx configuration
nginx_1           | Clearing out any default configurations.
nginx_1           | Proxying to `kobocat` through uWSGI.
nginx_1           | Serving static files for container kobocat from Nginx.
nginx_1           | Proxying to `kpi` through uWSGI.
nginx_1           | Serving static files for container kpi from Nginx.
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] still could not bind()
nginx_1           | Overwrite default nginx configuration
nginx_1           | Clearing out any default configurations.
nginx_1           | Proxying to `kobocat` through uWSGI.
nginx_1           | Serving static files for container kobocat from Nginx.
nginx_1           | Proxying to `kpi` through uWSGI.
nginx_1           | Serving static files for container kpi from Nginx.
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx_1           | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
kpi_1             | ERROR: Pidfile (/tmp/celery_sync_kobocat_xforms.pid) already exists.
kpi_1             | Seems we're already running? (pid: 86)
kpi_1             | ERROR: Pidfile (/tmp/celery.pid) already exists.
kpi_1             | Seems we're already running? (pid: 87)
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:08 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:14 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:17 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:18 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:23 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:25 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:26 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:27 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:28 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:29 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:30 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:31 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:32 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:33 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:34 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:35 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:36 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:37 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:38 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:39 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:40 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:41 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:42 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:43 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:44 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:45 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:46 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:47 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:48 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:49 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:50 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:51 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:52 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:53 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:54 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:55 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:56 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:57 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:58 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:00:59 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:00 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:02 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:03 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:04 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:05 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:06 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:07 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:08 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:09 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:10 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:11 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:12 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:13 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:14 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:15 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:16 --:--:--     0
enketo_express_1  | >> 0     0    0     0    0     0      0      0 --:--:--  0:01:17 --:--:--     0

At this point, I'm scared of even stopping the VM, because I'm not sure if it will start again.
Please help...

Can't build frontend from local machine with npm run watch

Description

On MacOS after fresh kobo-install run, calling npm run watch command from local machine will throw error Local package.json exists, but node_modules missing, did you mean to install?.

Guess: this happens because node_modules and jsapp/fonts directories are symlinked.

Note that it is possible to run npm run watch from within the docker container, but as my short experiments showed, it will be much slower than running the same command locally.

Temp hack fix

After fresh install, I run these commands to fix the problem:

rm -rf node_modules && rm -rf jsapp/fonts && npm i && npm run copy-fonts

Additional details

Related flowdock threads:

Install KPI's dev_requirements.txt when dev mode selected

@noliveleger's idea :)

Warn before modifying pre-existing `kobo-deployments`

I used to have a directory structure like this:

kobo-stuff
- kobo-docker
- kobo-deployments

I added kobo-install and renamed kobo-docker to move it out of the way:

kobo-stuff
- kobo-docker__old
- kobo-deployments
- kobo-install

I then ran kobo-install's run.py, and I noticed afterwards that kobo-deployments had been modified. I should've moved that out of the way as well, but I forgot! It would be nice to have a warning if run.py encounters an existing kobo-deployments directory on its first run.

TypeError: must be char, not unicode: Python 2.7.5

S.O. Centos 7

Traceback (most recent call last):
File "run.py", line 66, in
run()
File "run.py", line 26, in run
current_config = config.build()
File "/opt/kobo-install/helpers/config.py", line 211, in build
self.__detect_network()
File "/opt/kobo-install/helpers/config.py", line 404, in __detect_network
interfaces = Network.get_local_interfaces()
File "/opt/kobo-install/helpers/network.py", line 55, in get_local_interfaces
names = array.array('B', MAX_BYTES * FILL_CHAR)
TypeError: must be char, not unicode

SSL Not working, continual http

Sorry in advance if this is a super simple issue, and to be clear this is not a bug, we are just at wits end with SSL setup and need help from those more experienced.

We have been able to full and successfully set up kobotoolbox on a Microsoft Azure VM. We have set up the server version with frontend and backend on the same machine. All other parameters for postgres and mongo setup were left to default. Internal and external DNS routing is working in our environment.

My main question since documentation regarding SSL setup is thin is: Is it really as simple to setup as dropping in the crt and key files into the kobo-docker/secrets folder and restarting the nginx container? Did we miss some settings somewhere?

If this is the case, then our certs are not working properly for some reason and we can focus there (but which combo of fullchain, crt, key is required?) *we have a wildcard cert that covers our domain and all subdomains.

We have pored over the forums and other issues, and there is no mention of changing other SSL settings in other env files. In digging through the env files I see variables related to https and public_request_headers. We have tried various combinations of those settings and certs to no avail. I would love to write up a little documentation on this when we get it running, but we need to run it first! '

Hopefully someone out there has some advice or little tidbits that can push us across the finish line, we are so close! Thanks in advance!

How i can setup fresh install without domain only public IP

Hi folks , anyone of you know, How i can configure a new kobo server without registered domain, i just has my public ip addresss, could you please help on this

Invalid cross-device link

OS: Ubuntu 18.04.3 LTS
Python Version: Python 3.6.8

Additional context

Traceback (most recent call last):
File "run.py", line 55, in
run(force_setup=True)
File "run.py", line 27, in run
Setup.run(current_config)
File "/home/user/kobo/kobo-install/helpers/setup.py", line 28, in run
os.path.join(tmp_dirpath, Config.UNIQUE_ID_FILE))
OSError: [Errno 18] Invalid cross-device link: '/home/user/kobo/kobo-docker/.uniqid' -> '/tmp/tmpgwvfg6wk/.uniqid'

Port is hard-coded in Nginx template of maintenance container

It should be the config value.

Is there any API that can be used to get data from the app so we can connect it to another app

Description

Steps to Reproduce

Expected behavior

Desktop

OS:
Python Version:
Docker Version:
Docker Compose Version:

Additional context

Use `runserver[_plus]` for KoBoCAT in developer mode

We support it already for KPI:

kobo-install/templates/envfiles/nginx.txt.tpl

Lines 1 to 4 in 1e21e88

    
           # Options for the following are "uWSGI" or "runserver_plus" (for debugging). 
        
           KPI_WEB_SERVER=${WSGI_SERVER} 
        
           # django extensions are not installed on KoBoCat. So only `uWSGI` option is available. 
        
           KOBOCAT_WEB_SERVER=uWSGI

It sure would be nice 😁

https://www.flowdock.com/app/kobotoolbox/kobo/threads/HkqZYxP8ZFh-lNTOKR7JwXcprLD

Improve coding style

Comply with KoBo team's coding rules.

Reorganize imports
Use single quotes for string
Use kobotoolbox/kpi#2307 as much as possible
Use 80 characters per line
~~[ ] Use function annotations~~ Not until Python2 is supported

Usage of messages colors

Sometimes, COLOR_SUCCESS is used instead of COLOR_INFO.

How to test front-end changes?

I would like to test front-end changes to KPI code, e.g. while reviewing PRs. I do not want to use npm run watch, because no production build uses this. I would prefer not to have to rebuild and recreate all containers each time I change something in KPI, because this is slow (in excess of 3.5 minutes on my machine).

Here's what I tried, to no avail:

Cloned fresh copies of the KPI and KoBoCAT repositories;
Cloned a fresh kobo-install;
Set up my environment with run.py, instructing it to use developer mode and giving it the paths to the KPI and KoBoCAT source directories I'd just cloned;
Waited for everything to start up and verified that it worked;
Made a single change to an ES6 file in KPI;
Ran run.py --build-kpi;
Checked to see if my changes were visible in the browser or had made it into the compiled static files (they hadn't);
After that completed, ran run.py again, which recreated all containers;
Checked again for my changes in the browser and in the static files: they had not taken effect.

❓ Am I doing something wrong, or am I trying to do something we haven't made possible yet?

Here is the (very long) transcript of my shell session:

Starting fresh and cloning the source repositories:

john@scrappy:/tmp/kit$ sudo find -delete
john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kpi
Cloning into 'kpi'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 35185 (delta 0), reused 0 (delta 0), pack-reused 35184
Receiving objects: 100% (35185/35185), 11.94 MiB | 7.96 MiB/s, done.
Resolving deltas: 100% (26851/26851), done.
john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobocat
Cloning into 'kobocat'...
remote: Enumerating objects: 216, done.
remote: Counting objects: 100% (216/216), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 55268 (delta 132), reused 164 (delta 114), pack-reused 55052
Receiving objects: 100% (55268/55268), 23.14 MiB | 7.35 MiB/s, done.
Resolving deltas: 100% (39126/39126), done.

Cloning and running kobo-install:

john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobo-install
Cloning into 'kobo-install'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 549 (delta 19), reused 25 (delta 11), pack-reused 513
Receiving objects: 100% (549/549), 129.61 KiB | 3.50 MiB/s, done.
Resolving deltas: 100% (364/364), done.
john@scrappy:/tmp/kit$ cd kobo-install/
john@scrappy:/tmp/kit/kobo-install$ python run.py
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to `kobo-install`!                                    ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/tmp/kit/kobo-docker]: 
Do you want to see advanced options?
	1) Yes
	2) No
[2]: 1
What kind of installation do you need?
	1) On your workstation
	2) On a server
[2]: 1
Please choose which network interface you want to use?
	other) Other
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
[wlan0]: other
	lo) 127.0.0.1
	br-206827b65b4d) 172.24.0.1
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
	docker0) 172.17.0.1
[wlan0]: docker0
SMTP server: 
SMTP port [25]: 
SMTP user: 
From email address [[email protected]]: 
Super user's username [super_admin]: super
Super user's password [hdshIONMPviP]: duper
Docker Compose prefix? (leave empty for default): 
Web server port?
[80]: 
Developer mode?
	1) Yes
	2) No
[2]: 1
╔═══════════════════════════════════════════════════════════╗
║ Where are the files located locally? It can be absolute   ║
║ or relative to the directory of the installation.         ║
║ Leave empty if you don't need to overload the repository. ║
╚═══════════════════════════════════════════════════════════╝
KoBoCat files location: /tmp/kit/kobocat
KPI files location: /tmp/kit/kpi
Enable DEBUG?
	1) True
	2) False
[2]: 1
Postgres database [kobotoolbox]: 
Postgres user [kobo]: 
Postgres password [YJnEdBFnm5]: 
Do you want to tweak PostgreSQL settings?
	1) Yes
	2) No
[2]: 
Do you want to customize service ports?
	1) Yes
	2) No
[2]: 
Do you want to use AWS S3 storage?
	1) Yes
	2) No
[2]: 
Google Analytics Identifier: 
Google API Key: 
Intercom App ID: 
Do you want to use Sentry?
	1) Yes
	2) No
[2]: 
Do you want to tweak uWSGI settings?
	1) Yes
	2) No
[2]: 
Do you want to activate backups?
	1) Yes
	2) No
[2]: 
Cloning into '/tmp/kit/kobo-docker'...
remote: Enumerating objects: 2546, done.
remote: Total 2546 (delta 0), reused 0 (delta 0), pack-reused 2546
Receiving objects: 100% (2546/2546), 1.84 MiB | 6.38 MiB/s, done.
Resolving deltas: 100% (1580/1580), done.
Switched to a new branch 'kobo-install'
From https://github.com/kobotoolbox/kobo-docker
 * branch            kobo-install -> FETCH_HEAD
╔═══════════════════════════════════════════════════════════════════╗
║ Administrative privileges are required to update your /etc/hosts. ║
╚═══════════════════════════════════════════════════════════════════╝
Do you want to review your /etc/hosts file before overwriting it?
	1) Yes
	2) No
[2]: 
Stopping kobo-docker_redis_cache_1 ... done
Stopping kobo-docker_redis_main_1  ... done
Stopping kobo-docker_rabbit_1      ... done
Stopping kobo-docker_mongo_1       ... done
Removing kobo-docker_redis_cache_1 ... done
Removing kobo-docker_redis_main_1  ... done
Removing kobo-docker_postgres_1    ... done
Removing kobo-docker_rabbit_1      ... done
Removing kobo-docker_mongo_1       ... done
Removing network kobo-docker_default
Removing network kobo-docker_kobo-fe-network
WARNING: Network kobo-docker_kobo-fe-network not found.
Launching environment
Creating network "kobo-docker_default" with the default driver
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_redis_cache_1 ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_redis_main_1  ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_postgres_1, kobo-docker_rabbit_1, kobo-docker_redis_main_1, kobo-docker_redis_cache_1, kobo-docker_mongo_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Pulling nginx (kobotoolbox/nginx:latest)...
Building kobocat
WARNING: Image for service kobocat was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling enketo_express (kobotoolbox/enketo-express-extra-widgets:1.72.2)...
Building kpi
WARNING: Image for service kpi was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_kobocat_1        ... done
Creating kobo-docker_enketo_express_1 ... done
Waiting for environment to be ready. It can take a few minutes.
..
`KoBoToolbox` has not started yet, wait for another 600 minutes!
......................
╔════════════════════════════╗
║ Ready                      ║
║ URL: http://kf.kobo.local/ ║
║ User: super                ║
║ Password: duper            ║
╚════════════════════════════╝

Looking for the string creating your first project, which I will later change:

john@scrappy:/tmp/kit/kobo-install$ cd ../kobo-docker/
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first project' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
./.vols/static/kpi/js/components/searchcollectionlist.es6
./.vols/static/kpi/compiled/app-5a551488c6647ae95445.js
./.vols/static/kpi/compiled/app-05390ab1917b6af887af.js
grep: ./.vols/db: Permission denied

I make the following change to KPI, outside this shell:

diff --git a/jsapp/js/components/searchcollectionlist.es6 b/jsapp/js/components/searchcollectionlist.es6
index 965a251e..449bd3fd 100644
--- a/jsapp/js/components/searchcollectionlist.es6
+++ b/jsapp/js/components/searchcollectionlist.es6
@@ -249,7 +249,7 @@ class SearchCollectionList extends Reflux.Component {
                         return (
                           <bem.Loading>
                             <bem.Loading__inner>
-                              {t('Let\'s get started by creating your first project. Click the New button to create a new form.')}
+                              {t('Let\'s get started by creating your first Martian. Click the New button to create a new form.')}
                               <div className='pro-tip'>
                               {t('Advanced users: You also drag and drop XLSForms here and they will be uploaded and converted to projects.')}
                               </div>

Rebuilding KPI and seeing if the change made it through:

john@scrappy:/tmp/kit/kobo-docker$ cd -
/tmp/kit/kobo-install
john@scrappy:/tmp/kit/kobo-install$ python run.py --build-kpi
Building kpi
john@scrappy:/tmp/kit/kobo-install$ cd -
/tmp/kit/kobo-docker
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first Martian' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
grep: ./.vols/db: Permission denied

Since no results were found after rebuilding KPI, I'm now using run.py with no arguments to recreate all containers:

john@scrappy:/tmp/kit/kobo-docker$ cd -
/tmp/kit/kobo-install
john@scrappy:/tmp/kit/kobo-install$ time python run.py
Stopping kobo-docker_postgres_1    ... done
Stopping kobo-docker_rabbit_1      ... done
Stopping kobo-docker_redis_main_1  ... done
Stopping kobo-docker_redis_cache_1 ... done
Stopping kobo-docker_mongo_1       ... done
WARNING: Found orphan containers (kobo-docker_kobocat_1, kobo-docker_kpi_1, kobo-docker_nginx_1, kobo-docker_enketo_express_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing kobo-docker_postgres_1    ... done
Removing kobo-docker_rabbit_1      ... done
Removing kobo-docker_redis_main_1  ... done
Removing kobo-docker_redis_cache_1 ... done
Removing kobo-docker_mongo_1       ... done
Removing network kobo-docker_default
Stopping kobo-docker_kobocat_1        ... done
Stopping kobo-docker_kpi_1            ... done
Stopping kobo-docker_nginx_1          ... done
Stopping kobo-docker_enketo_express_1 ... done
Removing kobo-docker_kobocat_1        ... done
Removing kobo-docker_kpi_1            ... done
Removing kobo-docker_nginx_1          ... done
Removing kobo-docker_enketo_express_1 ... done
Removing network kobo-docker_kobo-fe-network
Launching environment
Creating network "kobo-docker_default" with the default driver
Creating kobo-docker_redis_cache_1 ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_redis_main_1  ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_postgres_1, kobo-docker_redis_main_1, kobo-docker_rabbit_1, kobo-docker_mongo_1, kobo-docker_redis_cache_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_enketo_express_1 ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_kobocat_1        ... done
Waiting for environment to be ready. It can take a few minutes.
..................
╔════════════════════════════╗
║ Ready                      ║
║ URL: http://kf.kobo.local/ ║
║ User: super                ║
║ Password: duper            ║
╚════════════════════════════╝

real	3m33.192s
user	0m2.287s
sys	0m0.376s

Now looking again for my modification in the compiled static files:

john@scrappy:/tmp/kit/kobo-install$ cd -
/tmp/kit/kobo-docker
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first Martian' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-20_024825.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
grep: ./.vols/db: Permission denied

Since there are still no results, here's a sanity check in the source directory:

john@scrappy:/tmp/kit/kobo-docker$ cd ../kpi
john@scrappy:/tmp/kit/kpi$ grep -lFR 'creating your first Martian' .
grep: ./jsapp/fonts: No such file or directory
./jsapp/js/components/searchcollectionlist.es6
grep: ./jsapp/compiled: No such file or directory
grep: ./node_modules: No such file or directory

In case it's helpful: run.conf (formatted).zip

`UnicodeEncodeError: codec can't encode characters`

Not sure we should change any code here, but for some reason, I needed to export LANG=en_US.UTF-8 on a particular Ubuntu 18.04.2 LTS server to avoid this traceback:

Traceback (most recent call last):
  File "run.py", line 66, in <module>
    run()
  File "run.py", line 26, in run
    current_config = config.build()
  File "/tmp/kobo-install/helpers/config.py", line 206, in build
    self.__welcome()
  File "/tmp/kobo-install/helpers/config.py", line 1222, in __welcome
    CLI.colored_print("\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557", CLI.COLOR_WARNING)
  File "/tmp/kobo-install/helpers/cli.py", line 46, in colored_print
    print(cls.colorize(message, color))
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 5-69: ordinal not in range(256)

This was with Python 3.6.7 (traceback above) and Python 2.7.15+ (same error but showed the box-drawing characters instead of the \u escape codes). If nothing else, we can could close this issue and let it serve as a reference for anyone searching for this error.

Server mode can't install in LAN

Hi, everybody
I'm trying to install with server mode.But can't access in LAN.
Can you help me please?

Kobo Toolbox not storing backups to S3

I have below Backup settings which I configured while Kobo Toolbox setup using Kobo install but the backups are not initiating and also not placed on S3.

Do you want to activate backups?
	1) Yes
	2) No
[1]: 
╔═════════════════════════════════════════════════════════════════╗
║ Schedules use linux cron syntax with UTC datetimes.             ║
║ For example, schedule at 12:00 AM E.S.T every Sunday would be:  ║
║ 0 5 * * 0                                                       ║
║                                                                 ║
║ Please visit https://crontab.guru/ to generate a cron schedule. ║
╚═════════════════════════════════════════════════════════════════╝
PostgreSQL backup schedule?
[*/2 * * * *]: 
MongoDB backup schedule?
[*/4 * * * *]: 
Redis backup schedule?
[*/6 * * * *]: 
AWS Backups bucket name [bucket-name]: 
How many yearly backups to keep?
[2]: 
How many monthly backups to keep?
[12]: 
How many weekly backups to keep?
[4]: 
How many daily backups to keep?
[10]: 
MongoDB backup minimum size (in MB)?
Files below this size will be ignored when rotating backups.
[5]: 
PostgresSQL backup minimum size (in MB)?
Files below this size will be ignored when rotating backups.
[5]: 
Redis backup minimum size (in MB)?
Files below this size will be ignored when rotating backups.
[5]: 
Chunk size of multipart uploads (in MB)?
[5]: 
Use AWS LifeCycle deletion rule?
	1) Yes
	2) No
[1]:

Similarly, as the setup asked for AWS bucket name for backup, it is not asking for Key and Secret of the bucket, can I know how it is going to store the backup? Can someone from Kobo team can check this?

'KoBoToolbox` has not started yet.

I get KoBoToolbox` has not started yet
Wait for another 600 seconds?
1) Yes
2) No
[1]: 1. even after waiting for 4 hours,

My server is 8GB RAM, Storage is 25GB
what could be the problem?

When I check log by python run.py --logs I get logs below

kpi_1 | execute_from_command_line(sys.argv) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line kpi_1 | utility.execute() kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute kpi_1 | self.fetch_command(subcommand).run_from_argv(self.argv) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv kpi_1 | self.execute(*args, **cmd_options) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute kpi_1 | output = self.handle(*args, **options) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/syncdb.py", line 25, in handle kpi_1 | call_command("migrate", **options) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 120, in call_command kpi_1 | return command.execute(*args, **defaults) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute kpi_1 | output = self.handle(*args, **options) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 91, in handle kpi_1 | connection.prepare_database() kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/contrib/gis/db/backends/postgis/base.py", line 36, in prepare_database kpi_1 | if self.template_postgis is None: kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__ kpi_1 | res = instance.__dict__[self.name] = self.func(instance) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/contrib/gis/db/backends/postgis/base.py", line 28, in template_postgis kpi_1 | with self._nodb_connection.cursor() as cursor: kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor kpi_1 | cursor = self.make_cursor(self._cursor()) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor kpi_1 | self.ensure_connection() kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection kpi_1 | self.connect() kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__ kpi_1 | six.reraise(dj_exc_type, dj_exc_value, traceback) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection kpi_1 | self.connect() kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect kpi_1 | self.connection = self.get_new_connection(conn_params) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection kpi_1 | connection = Database.connect(**conn_params) kpi_1 | File "/usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py", line 130, in connect kpi_1 | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) kpi_1 | django.db.utils.OperationalError: FATAL: password authentication failed for user "kobo" kpi_1 | kpi_1 | *** /etc/my_init.d/10_init_kpi.bash failed with status 1

Running $kobo-install> python run.py --setup to update Public domain name

Everything seems to be working (great job guys and gals on that), but when rerean the setup to change the public domain things got weird. I am still seeing the old domain reference in different places in kf. This happens when I try to redeloy a form and when I go into Settings -> Media. I was just hoping to know what files I can update to fix this problem. Thanks.

Detect upgrade from single to two databases

If the KPI database is empty and the KoBoCAT database is not, we should run the clone/migration script to copy the KPI tables from the KoBoCAT database (formerly shared by KPI) to the new KPI-only database.

Related to kobotoolbox/kpi#2543

Add option `--maintenance`

It should stop containers and start maintenance mode.

Kobo not starting due to error in kpi container

When I run the setup using kobo-install all the containers start successful except the kpi container that ends with this error.

raceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 303, in execute settings.INSTALLED_APPS File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 48, in __getattr__ self._setup(name) File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 44, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 92, in __init__ mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/srv/src/kpi/kobo/settings.py", line 184, in <module> 'default': dj_database_url.config(default="sqlite:///%s/db.sqlite3" % BASE_DIR), File "/usr/local/lib/python2.7/dist-packages/dj_database_url.py", line 51, in config config = parse(s, engine, conn_max_age) File "/usr/local/lib/python2.7/dist-packages/dj_database_url.py", line 98, in parse 'PORT': url.port or '', File "/usr/lib/python2.7/urlparse.py", line 113, in port port = int(port, 10) ValueError: invalid literal for int() with base 10: 'SgX*P' *** /etc/my_init.d/10_init_kpi.bash failed with status 1 2018-11-01T04:39:51.846640800Z *** Killing all processes...

Do you have any idea what could be causing this and if not would what additional information would help? Please let me know thanks.

Step by step installation on a fresh VPS running Ubuntu 18.04 (LTS)

Hi Guys. Is there any step by step guide available to install Kobotoolbox on a fresh Ubuntu server? I would like to know the nginx configuration and other packages I need to install to successfully run kobo-install.

Any help is much appreciated. Thanks

Installing kobotoolbox without Docker

Hi there,

Actually this is not a problem but a question about how to install kobotoolbox without using the docker? This is due to kobo image has not been created for armhf. Perhaps is something not feasible but as a PoC I would like to try to have a kobo server in a Raspberry Pi3 as a low cost infrastructure in places with limited resources.

Thank you very much in advance.

Best regards,

Fresh install on debian 9 does not start-up: certbot issue

Configuring kobo-installer on debian 9 does leave me with a system not fully starting, as far as docker ps shows me

CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS                          PORTS                              NAMES
24c101861782        certbot/certbot                                   "/bin/sh -c 'trap ex…"   18 hours ago        Up 18 hours                     80/tcp, 443/tcp                    nginx-certbot_certbot_1
505a15ddec87        nginx:1.15-alpine                                 "/bin/sh -c 'while :…"   18 hours ago        Restarting (1) 24 seconds ago                                      nginx-certbot_nginx_1
f8ecea6fb0ee        kobotoolbox/kobocat:2.019.22                      "/sbin/my_init"          18 hours ago        Up 18 hours                     8000/tcp                           kobo-docker_kobocat_1
8e5ec7412bf7        kobotoolbox/enketo-express-extra-widgets:1.77.0   "/sbin/my_init"          18 hours ago        Up 18 hours                     8005/tcp                           kobo-docker_enketo_express_1
1b662e52c617        kobotoolbox/kpi:2.019.23                          "/sbin/my_init"          18 hours ago        Up 18 hours                     8000/tcp                           kobo-docker_kpi_1
ce5496b718c0        kobotoolbox/nginx:latest                          "/sbin/my_init"          18 hours ago        Up 18 hours                     443/tcp, 0.0.0.0:8080->80/tcp      kobo-docker_nginx_1
b4325dd8fbc0        redis:3.2                                         "docker-entrypoint.s…"   18 hours ago        Up 18 hours                     6379/tcp, 0.0.0.0:6380->6380/tcp   kobo-docker_redis_cache_1
805c9c1fe1fe        mdillon/postgis:9.5                               "docker-entrypoint.s…"   18 hours ago        Up 18 hours                     0.0.0.0:5432->5432/tcp             kobo-docker_postgres_1
9730ce486809        redis:3.2                                         "docker-entrypoint.s…"   18 hours ago        Up 18 hours                     0.0.0.0:6379->6379/tcp             kobo-docker_redis_main_1
dfc96d48f1d2        mongo:3.4                                         "docker-entrypoint.s…"   18 hours ago        Up 18 hours                     0.0.0.0:27017->27017/tcp           kobo-docker_mongo_1

the problem seems to lie within

505a15ddec87        nginx:1.15-alpine                                 "/bin/sh -c 'while :…"   18 hours ago        Restarting (1) 24 seconds ago                                      nginx-certbot_nginx_1

the following output does always repeat

docker logs nginx-certbot_nginx_1
...
2019/06/19 06:23:48 [emerg] 1#1: cannot load certificate "/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
2019/06/19 06:24:49 [emerg] 1#1: cannot load certificate "/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/kf.kobo.medevit.at/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
...

Run arbitrary docker-compose commands

@noliveleger, what do you think of this idea?
d44344e

I've been using that change locally for convenience. I just wanted to post it here before I forgot about it.

	# Options for the following are "uWSGI" or "runserver_plus" (for debugging).
	KPI_WEB_SERVER=${WSGI_SERVER}
	# django extensions are not installed on KoBoCat. So only `uWSGI` option is available.
	KOBOCAT_WEB_SERVER=uWSGI