Serverless Python
Home Page: https://blog.zappa.io/
License: MIT License
Project moved to https://github.com/zappa/Zappa. Versions after 0.52.0 are published from there. Thank you Rich Jones for all your work on creating Zappa and maintaining it for years!
Needs lots of consideration and testing.
Resources:
http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.html
https://medium.com/@john.titus/moving-a-simple-api-to-amazon-s-api-gateway-680d025e0921#.mldfhxtgd
I'm tired of typing 'dev' every time.
No matter what region I set it will always end up in us-east-1. One thing I noticed straight away is that upload_to_s3 is missing locationconstraint on create_bucket:
s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={
'LocationConstraint': self.aws_region
})
Will dig around and see what else is required.
I'm most excited to use Blitz.io (or something similar) to start load-testing Zappa applications.
If we do it right, we should see a perfectly flat line!
This probably depends on profiling and optimizing (as well as improving unit test coverage) before we invest in this, but I think it will really really help to sell the benefits of Zappa.
Not sure if parts of this deployment tool could be pulled out to pull this off but here we go!
Could the general idea of deployment to lambda and setting up the API gateway be pulled out in order to generalize it enough to allow the language you deploy with not matter?
So each lambda function could be deployed in whatever language you choose and as long as it understands the template mappings you setup then the lambda endpoints to work with that setup.
If any of this doesn't make sense or isn't feasible let me know!
I encountered the following error while playing with zappa:
Packaging project as zip...
Uploading zip (15.1MiB)...
0%| | 0.00/15.9M [00:00<?, ?it/s]Failed to upload zappa-beta-1461586412.zip to scoop-lambda/zappa-beta-1461586412.zip: Need to rewind the stream <boto3.s3.transfer.ReadFileChunk object at 0x1070a2690>, but stream is not seekable.
I manually tried to upload a file to the same bucket using AWS CLI tool and it worked without any issue. Is this a known problem?
Is there any roadmap for flask support?
And is there anything I can do to help?
Based on this forum post and some experimentation I believe it's now possible to set the Location header for redirects. For example, I have a lambda function as follows:
exports.handler = function(event, context) {
if (/^[^.]+\.mydomain.com$/.test(event.Host)) {
context.succeed({'Location': event.Location});
}
else {
context.fail('Invalid Host');
}
};This can be set as a header on the 301 with the following mapping:
A log looks like this:
2016-02-20T13:15:03.68Z f38ea05d-d7d3-11e5-9e38-8167d23e7af7 127.0.0.1 - - [20/Feb/2016:13:15:03 +0000] "GET / HTTP/1.1" 200 138321 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:44.0) Gecko/20100101 Firefox/44.0" 0/126.601
Where 127001 should in fact be my real IP.
With a traditional HTTP server, the server is online 24/7, processing requests one by one as they come in. If the queue of incoming requests grows too large, some requests will time out. With Zappa, each request is given its own virtual HTTP "server" by Amazon API Gateway. Each request then calls your application from a memory cache in AWS Lambda and returns the response via Python's WSGI interface. After your app returns, the "server" dies.
I thought this was not a good thing? Didn’t we dump CGI precisely because it used one process per request? Why would this be better?
I ran into problems caused by ssl failing (SNI (Server Name Indication), Insecureplatform warnings etc) in the requests / urllib modules.
Using latest python 2.7. (2.7.11 in my case) helped.
A very good howto to do that only for a venv is here: http://mbless.de/blog/2016/01/09/upgrade-to-python-2711-on-ubuntu-1404-lts.html#example-virtualenv
This one would be huge. Not sure if it's possible, but a way to automatically have the SSL certificate update every month. Game changer.
The project is still small and manageable right now, but we've already got 3 different projects, and we may need a way to communicate outside of GitHub issues.
I personally like GitHub issues, but it might be easier to collaborate if we had a real-time forum as well. Mailing list? IRC? Gitter? Web chat? Slack? I don't know what the community would like the most.
There is a requirement for docopt but argparse is used for the cli. Can it be removed from requirements.txt?
I've been playing a lot with zappa lately, and consequently it's generated a lot of temporary files. Unfortunately it appears that these temporary files aren't cleaned up. In my case they were in /var/folders/4s/xxxxxxxx/T, and took up several gb of space. Here is what it looks like in a disk analyser:
I tried installing Werkzeug but it seems like it is already installed
~/dev/testzappa% sudo pip install Werkzeug Requirement already satisfied (use --upgrade to upgrade): Werkzeug in /usr/lib/python2.7/site-packages
I guess it isn't packaged into the lambda.
I get the following error in my lambda cloudwatch logs
Unable to import module 'handler': No module named werkzeug.wrappers
Maybe you could try something else ?
If i try a POST or DELETE i end up with a 400. Any ideas? Its a pretty straightforward flask app
Traceback (most recent call last): File "/var/task/handler.py", line 137, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 51, in lambda_handler return cls().handler(event, context) File "/var/task/handler.py", line 132, in handler raise Exception(exception) Exception:
<!DOCTYPE html>400<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Why all this logic to load credentials? Boto does all of this: http://boto.cloudhackers.com/en/latest/boto_config_tut.html#boto-config
I have no idea why one of the tests is suddenly failing on Travis. It worked before, it works locally, but apparently 0d9d710 broke the ability to load AWS credentials from a string? I'm confused.
Could be out of scope, but would be great to be able to easily see how much a deployment has cost, ex https://github.com/Miserlou/django-aws-billing
Given a configuration with a region other than default, and a bucket that hasn't yet been created:
{
"dev": {
"aws_region": "eu-west-1",
...
"s3_bucket": "not-yet-existing-bucket"
}
}when running zappa deploy dev, I was seeing this error:
An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: Error occurred while GetObject. S3 Error Code: PermanentRedirect. S3 Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
The error occurs because of the bucket being created in a different region - I checked and not-yet-existing-bucket was created in US/Standard instead of in eu-west-1.
Creating the bucket in the same region as in the configuration beforehand solved the problem, though it would be nice if this was fixed.
Using zappa 0.17.6.
First of all, awesome work, folks! Really liked it and intend to use.
This is more of a reminder for the README, because, when reading the code, I saw "python2.7" hardcoded and I figured it would be best to write that in the README
It might be sliiiiiiightly too soon to actually start developing this, but let's use this issue as a way to discuss the interface that Zappa might have as a CLI application that can be used to deploy/update/etc any arbitrary WSGI application.
I guess the main question that I have left is - how do we define the handler function?
I think we just need to throw init.py in there if there isn't one.
"hey, fun fact. python namespace packages don't work by default in AWS Lambda
so far, that means all zope._, repoze._, and google.* packages don't work by copying them out of a virtualenv"
These won't work, so fail out early.
Related to the comment here: Miserlou/django-zappa#42 (comment)
Would it be too crazy for us to maintain and publish Zappa-compatible versions of MySQL, Postgres, Scikit, Scipy, Numpy, etc?
I was hoping that this could be operated in a serverless fashion, but I don't think that's quite feasible yet. This seems like the closest workable solution.
First off, thanks for an amazing project! I've just started trying it out and I've run in to a problem.
This programming works locally:
`import flask
app = flask.Flask(name)
@app.route('/client/', methods=['GET'])
def hello(id):
return 'Hello ' + id
if name == 'main':
app.run()`
But when I deploy I and try with something like
https://107uk4y3sh.execute-api.ap-northeast-1.amazonaws.com/prod4/client/VVV
I get {"message": "Missing Authentication Token"}
Is there anything I need to do in zappa_settings.json to get this work. Changing the code to just have /client as path works fine.
Thanks
Hi,
I'm developing a small API using zappa with pycnic (https://github.com/nullism/pycnic) but I have a problem: When I do an api call with PUT method and with some json data in the body, like:
curl -i -X PUT -H "Authorization: xxxxyyyy662362632" -H "Content-Type:application/json" https://99999999999.execute-api.us-east-1.amazonaws.com/v1/uuid/cin2711b5000001qdk2fd5l3v -d '{"config": {"ADMIN_EMAIL": "[email protected]"}}'
zappa is not setting the environ['CONTENT_LENGTH'] in file wsgi.py and because this picnic generate an exception - it's needed the environ['CONTENT_LENGTH'] to read the wsgi's PUT body.
Zappa only set environ['CONTENT_LENGTH'] when the HTTP POST method is called, as you can see in line 72 bellow of wsgi.py.
# Input processing
if method == "POST":
if 'Content-Type' in event_info["headers"]:
environ['CONTENT_TYPE'] = event_info["headers"]['Content-Type']
# Multipart forms are Base64 encoded through API Gateway
if 'multipart/form-data;' in event_info["headers"]['Content-Type']:
body = base64.b64decode(body)
environ['wsgi.input'] = StringIO(body)
environ['CONTENT_LENGTH'] = str(len(body))
I think that line 72 of wsgi.py need's to be:
if method == "POST" or method == "PUT":
If your virtualenv's name is the same name as a package in your project's directory or virtualenv it will be added to the excludes list.
I propose that Zappa keeps this just document it better.
Related to #61
It'd be nice if we could do something like
python manage.py rollback dev
to return the previously deployed function, or
python manage.py rollback dev
to rollback 5 versions, etc.
Zappa currently uses two versions of boto. (This is the result of me reusing code from an old project).
It's just the code related S3, so it should be easy enough to replace with boto3 S3 stuff.
Zappa does not actually implement a WSGI app wrapper, it does translate a JSON dict into a WSGI environ, perhaps the function in wsgi.py should be named create_wsgi_environ instead.
The zappa_django project contains a lambda handler however that almost implements a proper WSGI app wrapper.
That handler should be extracted into zappa and fixed to truly call a WSGI app, this would make it much easier to add support for any framework using WSGI.
The ability to have some equivalent of fetching/tailing acces.log and error.log seems like a must.
CloudWatch should make this easy, but it still needs to be implemented.
Two little suggestions that I'd love to see in Zappa.
An optimization currently exists that uses the .pyc instead of the .py if it's available. The problem is that if the .py is newer but not compiled, the old .pyc will continue to be used. Should a .pyc present itself, ensure that the update time of the .pyc exceeds that of the .py file.
multipart/form-data is broken in Zappa, but given the new APIGW features added this week, maybe we can do something about that.
Best practices!
So, using a Flask hello world...
shaknaisrael.py
from flask import Flask
from jinja2 import Environment, FileSystemLoader
import os
app = Flask(__name__)
site_globals = {'site_title':'Shakna Israel'}
THIS_DIR = os.path.dirname(os.path.abspath(__file__))
j2_env = Environment(loader=FileSystemLoader(THIS_DIR), trim_blocks=True)
@app.route("/")
def hello():
locals = {}
render_data = site_globals.copy()
render_data.update(locals)
return j2_env.get_template('index.html').render(render_data)
if __name__ == "__main__":
app.run()
index.html
<html>
<head>
<title>{{ site_title }}</title>
</head>
<body>
Hello World.
</body>
</html>
zappa_settings.json
{
"production": {
"s3_bucket": "shaknaisrael",
"app_function": "shaknaisrael.app",
"delete_zip": true,
"debug": true
}
}
The intial deploy went absolutely fine, worked great. Then I tried to add the template stuff, so I can build a reasonably sized site.
Upon running zappa update production, then it "appears" to upload fine. Except the page is blank. No body in the HTTP request.
But, in the tail logs:
1459875734099] No module named shaknaisrael
[1459875734099] local variable 'app_module' referenced before assignment: UnboundLocalError
Traceback (most recent call last):
File "/var/task/handler.py", line 161, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 55, in lambda_handler
return cls().handler(event, context)
File "/var/task/handler.py", line 150, in handler
debug = bool(getattr(app_module, settings.DEBUG, True))
UnboundLocalError: local variable 'app_module' referenced before assignment
Is Zappa failing to upload the shaknaisrael.py file?
Here's a scenario which my current attempt at a WSGI middleware can't handle regarding multiple cookies (I think the Django-zappa middleware solution can't handle it either.)
foo and bar. (Django or flask).foo with new value.zappa cookie which contained both, with one that only contained foo.Possible fixes:
I think we should do first option as it respects the HTTP semantics instead of hiding them. Unless there are complications I haven't considered?
Hey guys,
Awesome project! I want to be able to run and test my Zappa setup locally and simulate AWS Lambda request, how can I do it currently?
Ex Use Case: lambda function watching for new S3 put object. I can simulate that from AWS UI, but don't know how to do it within Zappa.
Thanks.
If you forget to enter your packages env, source env/bin/activate, the deploying simply prints 'VIRTUAL_ENV':
$ zappa deploy dev
Packaging project as zip...
'VIRTUAL_ENV'
$
(This was after nearly following the instructions with the basic flask app and settings.)
Would be great if this could show a more descriptive message "not inside local virtual env" (or potentially this is as my global virtual env has some packages that not in lambda-package.
Just display warnings if the app_module doesn't exist to avoid the
UnboundLocalError: local variable 'app_module' referenced before assignment
that happens from putting the wrong values in zappa_settings.
I noticed this library uses a sleep to avoid AWS throttling requests - is there a reason the built in exponential retry handler that ships with botocore didn't work?
Also, I was curious why you have to sign your own request to put the integration method in: https://github.com/Miserlou/Zappa/blob/master/zappa/zappa.py#L509
Goal:
After unpacking cookies in the middleware, discard any that might be expired, before feeding them to the WSGI-app.
Normally the client would do this, but the zappa cookie is encoded so the expiration attributes are not accessible to the client.
Right now we use Werkzeug's cookie parser, but it throws away everything else but the key=value. So exp. date or other stuff is discarded. Use (perhaps) the stdlib for parsing cookies: https://docs.python.org/2/library/cookie.html#Cookie.BaseCookie.load
If anyone begins work on this, please leave a note here.
Bottle is like Flask, but even skinnier.
Once we have a full Flask extension working (which we're getting closer to), making a Bottle plugin should be simple as well.
Investigating, I thought we no longer required B64 encoding, but it seems way may, or there is another problem at work.
It could be useful to have Zappa be a more general Lambda utility - for instance, for automatically using lambda-packages and minifying, etc.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.