Checking that repo migration worked Contains information and scripts to get a development machine up and running very quickly
Python (not 3, 2.7)
VirtualEnv (Like rvm for ruby, manages versions of Python)
Django 1.3
Postgresql 9.1
Rackspace
Chef
Newrelic
First install git if you don't already have It, and setup a github account and gravatar image and get access to our team. if you are using git for the first time, follow these instructions to let you push to the ureport repo https://help.github.com/articles/generating-ssh-keys
Create somewhere to store all the stuff, e.g.
$ mkdir -p ~/Code/unicef/ureport
First clone this repository so that you can update this doc or the developer-faq with anything you learn!
$ cd ~/Code/unicef/ureport
$ git clone [email protected]:ureport/bootstrap.git
Some prerequisites:
Rackspace account (we have to this set up for you)
Newrelic account
Homebrew (https://github.com/mxcl/homebrew/wiki/Installation)
Python (not 3, 2.7)
XCode Developer Tools 4.6.1 (from your app store)
Then your pip
$ sudo easy_install pip
$ pip install virtualenv
If $ pip install virtualenv fails with the message "Permission Denied", try
$ sudo pip install virtualenv
Go to some folder where you want to store the code and create a folder called “virtualenv”, e.g.
$ mkdir -p virtualenv
Go in there and run virtualenv
$ cd virtualenv
$ virtualenv --no-site-packages ureport
Go back and clone the repos:
$ cd ..
$ git clone [email protected]:unicefuganda/ureport.git
<nimrod: ask us about submodules!!>
$ git clone [email protected]:ureport/provisioning.git
$ git clone [email protected]:ureport/performance.git
For each repository:
$ git submodule init
$ git submodule update
Activate your virtualenv so ureport can run on the python instance in the virtualenv:
$ source ./virtualenv/ureport/bin/activate
Configure your python environment
$ cd ureport
$ pip install -r pip-requires.txt
You can see if this works by
$ pip freeze
It should show you all the libraries that are in pip-install.txt
Postgres
Download the postgresapp from http://postgresapp.com/
Start the app. A little elephant should appear in your task bar.
Go to the elephant and make sure the first message in the drop down is Running on Port 5432
If the message is Failed to start on port 5432, it is likely the case that your pre-installed postgres automatically starts on that port. Make sure the pre installed postgres is not running on that port and that it does not restart itself when you kill it.
Check if there is a pre-existing installation of psql in /usr/bin/
by running
which psql
. If it exists, rename it using mv /usr/bin/psql /usr/bin/pqsl-old
so it does not
conflict with Postgresapp's psql. Check that it has been removed by running which pqsl
.
Check that there is no psql location reported
Then, add the Postgresapp psql to your path. It is prefered that you do add it to your
~/.bashrc
. The full path to the Postgresapp psql is
Users/Nimrod/Downloads/Postgres\ 2.app/Contents/MacOS/bin/psql
.
Login into psql as yourself by running psql
Create the role postgres
with CREATE ROLE postgres WITH LOGIN CREATEDB;
In your terminal, run psql -U postgres
. This should take you into a postgresql prompt
Then create the ureport DB by running CREATE DATABASE ureport;
. A success indicator is that
psql prints CREATE DATABASE
after you run the command.
Connect to the ureport
database by running \connect ureport;
and make sure you get the
message You are now connected to database "postgres" as user "xxx".
where xxx is local your username
Open a new terminal window, and navigate to the repo by running /path/to/ureport/original-repo/ureport/
.
While in that directory, activate your virtual environment by running
source /path/to/ureport/virtualenv/ureport/bin/activate
. Your prompt should now change to start with
(ureport)
With your virtual environment activated but while still in the repo, run ./manage.py syncdb --noinput
.
You should see something like this if it succeeds.
Syncing...
Creating tables ...
Creating table django_site
Creating table auth_permission
.
.
Creating table south_migrationhistory
Synced:
> mptt
> uni_form
.
.
> south
Not synced (use migrations):
- ureport
- django_extensions
.
.
-(use ./manage.py migrate to migrate these)
Then run the migrations using the command ./manage.py migrate
.
Now make sure you have postgress installed and running...
$ lsof -i:5432
You should see something like:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 72677 XXXX 5u IPv6 0x1e7355b40b608055 0t0 TCP
localhost:postgresql (LISTEN)
postgres 72677 XXXX 6u IPv4 0x1e7355b417851705 0t0 TCP
localhost:postgresql (LISTEN)
postgres 72677 XXXX 7u IPv6 0x1e7355b410d27c75 0t0 TCP
localhost:postgresql (LISTEN)
You should also be able to do :
$ psql postgres
And it should give you a db console.
Install Chef Client
$ gem install chef
Verify that chef has installed by running
$ chef-client -v
You should see something like this:
Chef: 11.4.4
Check that the following folder stucture is present on your local machine
/opt
/chef
/bin
/embedded
/bin
/include
/lib
/share
/ssl
$ gem install knife-rackspace
In your browser, Go to the Chef Server at http://95.138.169.81:4040/clients and create a client using the "Create" tab.
Enter a client name (your name), tick the box marking your client as an admin and click create. This will generate a private/public key pair that will be associated with the client you create. Save it as:
~/.chef/<client_name>.pem
Add the following to ~/.bash_profile
(or appropriate other file if you're using e.g. zsh).
# Rackspace credentials
export RACKSPACE_USERNAME="xxx"
export RACKSPACE_API_KEY="xxx"
export NEWRELIC_LICENSE_KEY="xxx"
Substitute the xxx with the rackspace details you obtain from one of the developers. Run the following command to generate the knife configuration file
$ knife configure --initial
When prompted for chef server url, enter
'http://95.138.169.81:4000'
Edit ~/.chef/knife.rb
and make the following changes:
client_key '~/.chef/<client_name>.pem'
Copy the following 5 knife attributes into ~/.chef/knife.rb
:
knife[:rackspace_api_username] = "#{ENV['RACKSPACE_USERNAME']}"
knife[:rackspace_api_key] = "#{ENV['RACKSPACE_API_KEY']}"
knife[:rackspace_version] = 'v2'
knife[:rackspace_api_auth_url] = "lon.auth.api.rackspacecloud.com"
knife[:rackspace_endpoint] = "https://lon.servers.api.rackspacecloud.com/v2"
Add:
encrypted_data_bag_secret '~/.chef/encrypted_data_bag_secret'
To verify:
$ knife node list
All devs should know how to and be capable of (i.e. have tools / keys installed) spinning up a new environment.
To have launchd start nginx at login: ln -sfv /usr/local/opt/nginx/*.plist ~/Library/LaunchAgents Then to load nginx now: launchctl load ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
brew install tomcat
Edit the file /usr/local/Cellar/tomcat/<version/libexec/conf/tomcat-users.xml
and add the following user:
<role rolename="manager-gui"/>
<user username="<choose a username>" password="<choose a password>" roles="manager-gui" />
Make a convenient shortcut for stopping and starting its
ln -s /usr/local/Cellar/tomcat/7.0.39/bin/catalina /usr/local/bin/tomcat7
ln -s /usr/local/Cellar/tomcat/7.0.39/libexec/ /usr/local/etc/tomcat7
This is the program that will import shape files to postgres. It lives wherever your Postgres.app file is
ln -s /Applications/Postgres.app/Contents/MacOS/bin/ogr2ogr /usr/local/bin/ogr2ogr
You will need a few libraries...
brew install libtiff
mkdir -p ~/tmp/geoserver/war
cd ~/tmp/geoserver/war
wget -O geoserver-2.2.5.zip http://downloads.sourceforge.net/project/geoserver/GeoServer/2.2.5/geoserver-2.2.5-war.zip?r=http%3A%2F%2Fgeoserver.org%2Fdisplay%2FGEOS%2FGeoServer%2B2.2.5
unzip geoserver-2.2.5.zip
tomcat7 stop
cp geoserver.war /usr/local/etc/tomcat7/webapps
tomcat7 start
open http://localhost:8080/geoserver
You can see if its worked by looking in
tail -f /usr/local/etc/tomcat7/logs/catalina.out
If it doesnt work, try tailing that log and restarting tomcat.
First get the zip files
mkdir -p ~/tmp/geoserver/data
cd ~/tmp/geoserver/data
wget -O geoserver-data.tgz https://github.com/ureport/provisioning/blob/master/chef/cookbooks/geoserver_app/files/default/geoserver-data.tgz?raw=true
mkdir -p /usr/local/var/lib/geoserver_data
cd /usr/local/var/lib/geoserver_data
tar xfvz ~/tmp/geoserver/data/geoserver-data.tgz
Now we need to add some machine specific config
sudo ln -s /usr/local/var/lib/geoserver_data /var/lib/geoserver_data
cd /var/lib/geoserver_data/workspaces/unicef/geoserver
wget -O datastore.xml https://raw.github.com/ureport/provisioning/master/chef/cookbooks/geoserver_app/templates/templates/default/datastore.xml.erb
We need to replace some variables
sed "s/<\%= @db_port \%>/put_db_port_here/" datastore.xml >> datastore.xml.merged
mv datastore.xml.merged datastore.xml
sed "s/<\%= @db_host \%>/put_db_host/" datastore.xml >> datastore.xml.merged
mv datastore.xml.merged datastore.xml
sed "s/<\%= @user \%>/put_db_user_name_here/" datastore.xml >> datastore.xml.merged
mv datastore.xml.merged datastore.xml
We now need to create our geoserver data base
mkdir -p ~/tmp/geoserver/db
cd ~/tmp/geoserver/db
wget -O geoserver-db.tgz https://github.com/ureport/provisioning/raw/master/chef/cookbooks/geoserver_db/files/default/geoserver-db.tgz
tar xfvz geoserver-db.tgz
You now have a script in this folder which should setup a geoserver database from scratch. Be warned, it will overwrite the DB if it exists!
./configure-geoserver-db.sh
This script will create a databse as your local user on localhost with the default postgres port. You can pass in these parameters.
Install the postgis extension. You will need this if your shape files are to be mapped to the correct coorinate systems:
$ psql geoserver
geoserver=# create extension postgis.
Check that it created a table called spatial_ref_sys by running geoserver=# \dt
Now we need to tell geoserver to pick up this data dir...
cd ~/tmp/geoserver
wget -O web.xml https://raw.github.com/ureport/provisioning/master/chef/cookbooks/geoserver_app/templates/templates/default/web.xml.erb
tomcat7 stop
cp web.xml /usr/local/etc/tomcat7/webapps/geoserver/WEB-INF
tomcat7 start
tail -f /usr/local/etc/tomcat7/logs/catalina.out
And now you should see everthing start up!
If you go into http://localhost:8080/geoserver you should see that there is a unicef workspace and that some layers are there. Also there should be no errors in the log file.
If you have existing data for polls, you will need to export the poll data...
To get this to work, unfortunately we have to patch django!
patch -p2 -d ${UREPORT_VIRTUAL_ENV_HOME}/lib/python2.7/site-packages/django < ${UREPORT_HOME}/ureport_project/rapidsms_ureport/12890.Django-1.3.diff
It will complain about some files which are actually test files. When it does this, keep pressing and it will ask if you want to skip this patch, just say 'y'.
Now we can do this:
cd ${UREPORT_HOME}/ureport_project
Ensure your virtualenv is activated
./manage.py export_poll_data --settings=xxx_settings
Where xxx_settings
is which ever settings file you want to use.
If you don't have the patch applied, you will see this
django.db.utils.DatabaseError: missing FROM-clause entry for table "t7"
LINE 1: SELECT (T7.name) AS "location_name", (locations_point.longit...
^