mattmakai / underwear

Dead simple LAMP-stack deployments for Python-powered web applications.

Underwear

This project is no longer under active development. Sorry folks, my other open source projects take too much of my time away from working on a library like this one. Fork away!

Underwear is a library for easily deploying any Python-powered web application to one or more Linux servers. Underwear is configurable by a YAML template and takes care of installing packages, configuring web/WSGI servers, and securing the server.

What Problem Does Underwear Solve?

Despite the advent of configuration management tools such as Puppet, Chef, Ansible, and Salt, it remains difficult to deploy a web application because you have to first learn one of those tools and then write scripts in the tool's domain-specific language.

Underwear makes deploying to a traditional Linux server stack as easy as deploying to Heroku by providing a pre-packaged, easily configurable library. Deployments can be executed simply by installing Underwear with pip, specifying the IP addresses of the server(s) to deploy to, then running a couple of commands.

Quick Start

After these ten steps your app should be completely configured, secured, and deployed to remote servers.

These instructions will be simplied in future releases as the library is further developed.

1. Install Underwear to your virtualenv:

   pip install underwear
   

2. Add 'underwear' to INSTALLED_APPS in your Django project. Underwear currently only supports Django (Flask and Bottle will be added in the 0.5 release).

3. Create a deploy directory under your Django project:

   mkdir deploy; cd deploy
   

4. Download the Fabric file for uploading SSH keys, YAML configuration template, and hosts file to your project under the deploy/ directory:

   wget https://raw.github.com/makaimc/underwear/master/underwear/deploy/fabfile.py
   wget https://raw.github.com/makaimc/underwear/master/underwear/deploy/underwear.yml
   wget https://raw.github.com/makaimc/underwear/master/underwear/deploy/hosts
   

5. Fill in blank variables fields in fabfile.py.

6. Execute the Fabric script (the script will prompt you for the non-root password):

   fab bootstrap_ansible
   

7. Fill in blank fields and modify desired settings in the underwear.yml file so that Underwear knows where to install your web application.

8. Fill in the hosts file with your servers' IP addresses. For example, the file may look like the following:

   [webservers]
   192.168.1.1
   
   [dbservers]
   192.168.1.2
   

8. Deploy your application:

   python manage.py deploy
   

9. Underwear will output each step in the deployment process to standard output as it happens.

10. Access your application from the domain name you provided in the underwear.yml file.

License and Documentation

Underwear is free software under the MIT license.

Additional Underwear documentation can be found on Read the Docs.

About the Name "Underwear"

This library goes underneath your WSGI application to help you quickly deploy it to traditional servers. Otherwise no one knows you're using it, and you can swap it out at any time for another deployment process when you outgrow the standard LAMP-stack setup.

There also weren't any projects on GitHub by the name of underwear that had more than two stars and "underwear" was an open library name on PyPi :)

Features

• Automated WSGI application deployments to a web & database server
• Configurable via a simple YAML template
• Support for ad hoc deployment needs without SSH-ing into remote servers

underwear's Issues

Fix simple typo: simplied -> simplified

Issue Type

[x] Bug (Typo)

Steps to Replicate

1. Examine README.rst.
2. Search for simplied.

Expected Behaviour

1. Should read simplified.

Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR. Alternatively if the fix is undesired please
close the issue with a small comment about the reasoning.

https://github.com/timgates42/underwear/pull/new/bugfix_typo_simplified

Thanks.

invalid syntax at python manage.py deploy

I've followed the installation process and everything was prepared right on the target system with a clean Debian 9 machine. I've started the script out of my Mac OS X with Python 3.7.1 from my virtualenv.
After the fab bootstrap_ansible went through I've started the command python manage.py deploy and got the following error message:

(.env) ➜  src git:(master) ✗ python manage.py deploy 
Traceback (most recent call last):
  File "manage.py", line 13, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/django/core/management/__init__.py", line 216, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/django/core/management/__init__.py", line 36, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/underwear/management/commands/deploy.py", line 3, in <module>
    from underwear.run_underwear import deploy
  File "/Users/andrej/Documents/Projekte/sms-api-gateway/.env/lib/python3.7/site-packages/underwear/run_underwear.py", line 64
    print "number of hosts: %s" % str(len(inventory.list_hosts()))
                              ^
SyntaxError: invalid syntax

I'm just a beginner with python, but after some research how to print a string with parameter it looks like the line:
print "number of hosts: %s" % str(len(inventory.list_hosts()))

should be look like the following line:
print("number of hosts: %(s)" % str(len(inventory.list_hosts())))

Assumes Postgres 9.1

Server being deployed to is Ubuntu 15.10, which by default installs Postgres 9.4

TASK: [db | update postgres access and listen for external hosts] ************* 
failed: [<scrubbed>] => {"failed": true}
msg: Destination directory /etc/postgresql/9.1/main does not exist

# test -e /etc/postgresql/9.4/main; echo $?                                                                                                                                                      
0

install Supervisor with pip instead of outdated apt package

Supervisor installed through aptitude is out of date. pip should be used to do the installation instead.

fabric script and a vagrant VM

Hi,

Great tool. I really like the idea of being able to deploy django easily.
However I had a couple of problems to use the Fabric script with a Vagrant VM. The box I use is a "precise64" (http://files.vagrantup.com/precise64.box)

First, I did not know what the "Login password for root" was at the beginning.

(dev)vagrant@precise64:/underwear$ fab bootstrap_ansible
[10.0.0.51] Executing task 'bootstrap_ansible'
[localhost] local: ssh-keygen -R 10.0.0.51
/home/vagrant/.ssh/known_hosts updated.
Original contents retained as /home/vagrant/.ssh/known_hosts.old
[10.0.0.51] Login password for 'root':
[10.0.0.51] Login password for 'root':
[10.0.0.51] Login password for 'root':
Sorry, you can't enter an empty password. Please try again.
[10.0.0.51] Login password for 'root':

I think there is no password for root on a Vagrant VM.
Then I did on the Vagrant VM: sudo passwd
I entered a password and I could enter this new password on the prompt below to keep going. So that works.

Then it asked to enter a new UNIX password. What is this one for? I have the feeling it replaces the one I just entered above right?

[10.0.0.51] run: /usr/bin/passwd deployer
[10.0.0.51] out: Enter new UNIX password:
[10.0.0.51] out: Retype new UNIX password:
[10.0.0.51] out: passwd: password updated successfully
[10.0.0.51] out:

Then it asked for the [email protected]'s password:
I am not sure what this password is, but it worked when I entered the new UNIX password I just set above. So it is not the password of the deployer user right?

I am a bit confused with all these passwords.

Also, why the script wants "authorized_keys2" instead of "authorized_keys". Is there a reason? What is the difference?
I have read online that authorized_keys2 is deprecated.

Thanks

No handler for paramiko, incompatible ssh server

Hello !

I'm very very very much excited about your project, I think it's very much needed to ease deployments like that. Well done !

But I'm stuck at point 6 with the fab command. Also note that after installation of underwear I didn't have the fab command (in zsh, even after rehash), so I installed fabric manually.

$ fab bootstrap_ansible  
[5.xxx.70.6] Executing task 'bootstrap_ansible'
[localhost] local: ssh-keygen -R 5.xxx.70.6
/home/vince/.ssh/known_hosts updated.
Original contents retained as /home/vince/.ssh/known_hosts.old
No handlers could be found for logger "paramiko.transport"

Fatal error: Incompatible ssh server (no acceptable macs)

Underlying exception:
    Incompatible ssh server (no acceptable macs)

Aborting.

I already worked a bit in my server. We can't login as root, so I filled in the right env.user in the fabfile, but same error. Does my server need to be in some predefined state ?

Any idea on how I can carry on ?

Thanks !

[optparse] Conflicting Option String

My Environment:

Python 2.7.10
Ubuntu 15.10
pip 8.1.0

Installed underwear via pip install underwear

Traceback:

$ python manage.py deploy
Traceback (most recent call last):
  File "manage.py", line 13, in <module>
    execute_from_command_line(sys.argv)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "<scrubbed>/local/lib/python2.7/site-packages/underwear/management/commands/deploy.py", line 14, in handle
    '--extra-vars', '@%s' % './deploy/underwear.yml'])
  File "<scrubbed>/local/lib/python2.7/site-packages/underwear/run_underwear.py", line 55, in deploy
    help="set additional variables as key=value or YAML/JSON", default=[])
  File "/usr/lib/python2.7/optparse.py", line 1021, in add_option
    self._check_conflict(option)
  File "/usr/lib/python2.7/optparse.py", line 996, in _check_conflict
    option)
optparse.OptionConflictError: option -e/--extra-vars: conflicting option string(s): -e, --extra-vars

New deptree:

underwear==0.4.5
  - ansible [installed: 1.9.4]
    - paramiko [installed: 1.16.0]
      - ecdsa [required: >=0.11, installed: 0.13]
      - pycrypto [required: >=2.1, installed: 2.6.1]
    - setuptools
    - PyYAML [installed: 3.11]
    - pycrypto [required: >=2.6, installed: 2.6.1]
    - jinja2 [installed: 2.8]
      - MarkupSafe [installed: 0.23]

Error can be resolved by removing this line

Possibilities with Python 3

Hi Matt! Decided to give this a run after hearing your plug for it on the TPTM podcast. But it looks like it depends on Fabric which is apparently Python 2 only. Wondering if there were any plans to move this to invoke or create Py3 compatibility otherwise?

Ansible: No module named template

My Environment:

Python 2.7.10
Ubuntu 15.10
pip 8.1.0

Installed underwear via pip install underwear

Traceback:

$ python manage.py deploy                                                                                                                                              1 ↵
Traceback (most recent call last):
  File "manage.py", line 13, in <module>
    execute_from_command_line(sys.argv)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 195, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "<scrubbed>/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 39, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "<scrubbed>/local/lib/python2.7/site-packages/underwear/management/commands/deploy.py", line 3, in <module>
    from underwear.run_underwear import deploy
  File "<scrubbed>/local/lib/python2.7/site-packages/underwear/run_underwear.py", line 17, in <module>
    import ansible.utils.template
ImportError: No module named template

underwear==0.4.5
  - ansible [installed: 2.0.1.0]
    - paramiko [installed: 1.16.0]
      - ecdsa [required: >=0.11, installed: 0.13]
      - pycrypto [required: >=2.1, installed: 2.6.1]
    - setuptools
    - PyYAML [installed: 3.11]
    - pycrypto [required: >=2.6, installed: 2.6.1]
    - jinja2 [installed: 2.8]
      - MarkupSafe [installed: 0.23]