rehabstudio / rehabstudio-gae-scaffold

rehabstudio-gae-scaffold

---

NOTE: This scaffold is a fork of Google's gae-secure-scaffold-python used under the terms of the Apache 2.0 license. Whilst we've tried to stay close to the original project, we've modified the layout extensively and made numerous updates (which are outlined in detail below).

Introduction

---

This scaffold aims to make it as easy as possible to get started with Google Appengine, providing a secure application template including example handlers, routes and tests. An example makefile is provided, containing sensible defaults for management commands (running the development server, tests, deploying, etc.).

This template uses Docker to provide a standard development environment for all developers on a project, this is the preferred method of installation/development.

Security

---

The scaffold provides the following basic security guarantees by default through a set of base classes found in app/base/handlers.py. These handlers:

1. Set assorted security headers (Strict-Transport-Security, X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, Content-Security-Policy) with strong default values to help avoid attacks like Cross-Site Scripting (XSS) and Cross-Site Script Inclusion. See _SetCommonResponseHeaders() and SetAjaxResponseHeaders().
2. Prevent the XSS-prone construction of HTML via string concatenation by forcing the use of a template system (Jinja2 supported). The template systems have non-contextual autoescaping enabled by default. See the render(), render_json() methods in BaseHandler and BaseAjaxHandler. For contextual autoescaping, you should use Closure Templates in strict mode (https://developers.google.com/closure/templates/docs/security).
3. Test for the presence of headers that guarantee requests to Cron or Task endpoints are made by the AppEngine serving environment or an application administrator. See the dispatch() method in BaseCronHandler and BaseTaskHandler.
4. Verify XSRF tokens by default on authenticated requests using any verb other that GET, HEAD, or OPTIONS. See the _RequestContainsValidXsrfToken() method for more information.

In addition to the protections above, the scaffold monkey patches assorted APIs that use insecure or dangerous defaults (see app/base/api_fixer.py).

Obviously no framework is perfect, and the flexibility of Python offers many ways for a motivated developer to circumvent the protections offered. Under the assumption that developers are not malicious, using the scaffold should centralize many security mechanisms, provide safe defaults, and structure the code in a way that facilitates security review.

Sample implementations can be found in app/handlers.py. These demonstrate basic functionality, and should be removed / replaced by code specific to your application.

Differences from gae-secure-scaffold-python

---

We've made some changes from the scaffold that Google provide. The major changes are listed below, expect this list to grow over time as the scaffold matures.

• Removed frontend-related code (to be re-added very soon).
• Added full docker setup to allow running in a consistent environment.
• Switched to a make based task runner for the overall application.
  • Original grunt and bash based build tools have been removed
  • A frontend build tool (grunt or gulp) will be used to manage frontend code but will be managed via the same makefile interface.
• Removed src/ and out/ directory.
  • Moved application code to app/ directory.
  • Removed copy/build step for python code (javascript will still get built as required).
• Change how third-party code is used in the app.
  • Third-party (python) directory is now added to sys.path at runtime instead of copying to the application root.
• Enabled warmup requests in app.yaml.
• Removed support for Django templates, since we only use Jinja2.
• Tests run using nose and coverage with better test discovery.
• All tests and test related utils moved to a seperate folder.
• Routes can get messy, so now live in their own routes.py module for easier maintenance.
• PEP8 everywhere (sort of).

Installing Docker

---

NOTE: The minimum required version of docker is 1.3. Docker/boot2docker 1.3.0 added support for mounted volumes when using boot2docker on OSX.

<<<<<<< HEAD

Linux

Docker is best supported on Linux, you can probably find packages for your preferred distribution here.

Dependency Setup

---

From the root of the repository:

1. git submodule init
2. git submodule update
3. cd closure-compiler - refer to closure-compiler/README.md on how to build the compiler. Feel free to use this GWT-skipping variant: mvn -pl externs/pom.xml,pom-main.xml,pom-main-shaded.xml
4. cd ../closure-templates && mvn && cd ..
5. npm install
6. mkdir $HOME/bin; cd $HOME/bin
7. npm install grunt-cli
   • Alternatively, sudo npm install -g grunt-cli will install system-wide and you may skip the next step.
8. export PATH=$HOME/bin/node_modules/grunt-cli/bin:$PATH
   • It is advisable to add this to login profile scripts (.bashrc, etc.).
9. Visit https://cloud.google.com/appengine/docs/python/download, and choose the alternative option to "download the original App Engine SDK for Python." Choose the "Linux" platform (even if you use OS X). Unzip the file, such that $HOME/bin/google_appengine/ is populated with the contents of the .zip.

064e07bb8176fe028f332e1c9d3344fa7d0487fa

OSX

<<<<<<< HEAD Install Docker and boot2docker following the instructions on this page.

Next, we need to forward the appropriate ports so that we can reach the running appengine development server directly from the host OS:

$ VBoxManage controlvm boot2docker-vm natpf1 "aesdk,tcp,127.0.0.1,8080,,8080"
$ VBoxManage controlvm boot2docker-vm natpf1 "aesdkadmin,tcp,127.0.0.1,8000,,8000"

Windows

Not supported yet (we just haven't tried, give it a go, it might work). Pull requests very welcome.

=======

Scaffold Development

---

064e07bb8176fe028f332e1c9d3344fa7d0487fa

Getting the scaffold

---

This part is easy with git:

$ git clone https://github.com/rehabstudio/rehabstudio-gae-scaffold.git

You probably want to repoint the git remote origin to your own repository:

<<<<<<< HEAD $ git remote set-url origin [email protected]:me/my-repo.git

Using the scaffold

---

With Docker installed, running your application should be as simple as:

Note that the development appserver will be running on a snapshot of code at the time you run it. If you make changes, you can run the various Grunt tasks in order to propagate them to the local appserver. For instance, grunt copy will refresh the source code (local and third party), static files, and templates. grunt closureSoys and/or grunt closureBuilder will rebuild the templates or your provided Javascript and the updated versions will be written in the output directory.

064e07bb8176fe028f332e1c9d3344fa7d0487fa

$ make run

Then visit the running application http://localhost:8080

To run your application's tests, use the command:

<<<<<<< HEAD $ make test

To deploy your application to appengine, use:

$ make deploy

If you wish to deploy to an application other than the default (defind in the root-level Makefile) you can pass one of (or both) app or version arguments to make deploy like so:

$ make deploy app=someapp
$ make deploy version=1
$ make deploy app=someotherapp version=someotherversion

To open a bash shell inside the container environment, use:

$ make shell

To run an IPython shell inside an appengine environment (with both your app and all SDK modules importable), use:

$ make pyshell

Note: When you want to run the python shell you must have already started make run in another shell so we can access the remote API.

Building frontend code

The src-server/static/ folder should be empty in git, if you need to run the app locally or deploy to a new server, you need to first use the included tools to build the application. To build the frontend/client application, use:

$ make client-build

You can also use gulp-watch to trigger rebuilds automatically using:

$ make client-watch

Installing Libraries

---

See the Third party libraries page for libraries that are already included in the SDK. To include SDK libraries, add them in your app.yaml file. Other than libraries included in the SDK, only pure python libraries may be added to an App Engine project.

Any third-party Python modules added to the app/third_party/py/ directory will be added to Python's sys.path at runtime.

Licensing

---

Apache 2.0. See LICENSE

=======

Notes

---

Files in js/ are compiled by the Closure Compiler (if available) and placed in out/static/app.js. Included in this compilation pass is the the output of the closureSoys:js task (intermediate artifacts: out/generated/js/*.js).

Closure Templates that you provide are also compiled using the Python backend, and are available using the constants.CLOSURE template strategy (the default). The generated source code is stored in out/generated/*.py. To use them, pass the callable template as the first argument to render(), and a dictionary containing the template values as the second argument, e.g.:

from generated import helloworld

[...]

self.render(helloworld.helloWorld, { 'name': 'first last' })

The /static and /template directories are replicated in out/, and the files in src/ are rebased into out/ (so src/base/foo.py becomes out/base/foo.py).

064e07bb8176fe028f332e1c9d3344fa7d0487fa

rehabstudio-gae-scaffold's Issues

Update documentation and examples

I have some docs on adding i18n support which should hopefully save people some time in the future if they need to implement it.

We should also move the example handlers into an examples folder to make it faster to get up and running when cloning the scaffold.

merge latest commits from original scaffold

There have been a few small updates to the original scaffold that need merged in, notably this commit google@f12a7b6

Sync with latest updates in gae-secure-scaffold-python

Some security updates which need to be added

Appengine SDK URL needs updated with each new release

A script should be provided that will download the latest version of the SDK on an initial docker build regardless of what the latest released version is.

Still undecided if the version, once downloaded, should be locked for the lifetime of the project (with a ruby gems style lock file that can be edited manually if necessary.

remote_api should be disabled on deploy

We currently use the remote_api setting to enable the use of a python shell when running locally. We should come up with a nice method of disabling this on deploy (but not locally as we still need it).

Datastore in-context cache is not being cleared between tests

This seems to be a side-effect of how the session key is generated and saved to the datastore in GetApplicationConfiguration. To reproduce, remove any existing test code in app/tests/base and run the following test case on its own:

from google.appengine.ext import ndb
from tests.testcases import BaseTestCase

# This import (unused in this particular test case) is found in many
# of the test modules already present in app/tests/base. It appears
# that the call to GetApplicationConfiguration within app/config.py
# causes the bug. Remove this import to see that the problem is
# resolved.
from app import config


class MyModel(ndb.Model):
    value = ndb.StringProperty()


class ExampleTestCase(BaseTestCase):

    def test_1_model_is_created(self):
        """ Persist an object in the datastore with and ID of 1234.
        """
        m = MyModel(id=1234, value='foobar')
        m.put()
        self.assertEqual(len(MyModel.query().fetch()), 1)

    def test_2_model_no_longer_exists(self):
        """ Assert that the datastore is reset from the previous test
        run and that object 1234 is no longer present in the datastore.
        """
        self.assertEqual(len(MyModel.query().fetch()), 0)

        # This assertion will fail even though the datastore is shown
        # to be reset in the previous assertion. This hints that the
        # datastore's in-context cache is not being cleared.
        self.assertIsNone(MyModel.get_by_id(1234))