Coder Social home page Coder Social logo

django-s3-sqlite's Introduction

django-s3-sqlite

PyPI

This project was inspired and started with Zappa. Thanks to Rich Jones for all of his amazing work.

Installation

Install via pip:

$ pip install django-s3-sqlite

Add to your installed apps:

INSTALLED_APPS += ["django_s3_sqlite"]

Using an S3-Backed Database Engine

django-s3-sqlite allows use of an S3-synced SQLite database as a Django database engine.

This will cause problems for applications with concurrent writes**, but it scales very well for high-read applications that don't have concurrent writes (like a CMS for your blog), and it's orders of magnitude cheaper than AWS RDS or Aurora (pennies per month instead of many dollars per month).

** Concurrent writes will often be lost and not show up in concurrent readers. This is because the database is transferred between S3 storage and the Lambda instance for each request.

Django Settings & Commands

DATABASES = {
    "default": {
        "ENGINE": "django_s3_sqlite",
        "NAME": "sqlite.db",
        "BUCKET": "your-db-bucket",
	"AWS_S3_ACCESS_KEY": "AKIA0000000000000000",  # optional, to lock down your S3 bucket to an IAM user
	"AWS_S3_ACCESS_SECRET": "9tIZfakefakefakefakeT9Q6LD6jB5UyofakeISN",  # optional, to lock down your S3 bucket to an IAM user
    }
}

Newer versions of Django (v2.1+) require a newer version of SQLite (3.8.3+) than is available on AWS Lambda instances (3.7.17).

Because of this, you will need to download the provided _sqlite3.so for your Python version (available in the shared-objects directory of this repository) and put it at the root of your Django project. Note that only the Python 3.6 and 3.8 runtimes are supported at this time. These shared object files contain a compiled binary static build of SQLite 3.30.1 that can be used with the corresponding version of Python. You will also need to add this line to your Zappa JSON settings file in each environment:

"use_precompiled_packages": false,

Since SQLite keeps the database in a single file, you will want to keep it as small and defragmented as possible. It is good to occasionally perform a database vacuum, especially after deleting or updating data. There's a command to vacuum your database:

zappa manage [instance] s3_sqlite_vacuum

Creating a Default Admin User

You'll probably need a default user to manage your application with, so you can now:

$ zappa manage <stage> create_admin_user

Or you can pass some arguments:

$ zappa manage <stage> create_admin_user one two three

This will internally make this call:

User.objects.create_superuser('one', 'two', 'three')

Release Notes & Contributors

Maintainers and Creator

This package is largely maintained by the staff of Wharton Research Data Services. We are thrilled that The Wharton School allows us a certain amount of time to contribute to open-source projects. We add features as they are necessary for our projects, and try to keep up with Issues and Pull Requests as best we can. Due to time constraints (our full time jobs!), Feature Requests without a Pull Request may not be implemented, but we are always open to new ideas and grateful for contributions and our package users.

Build Instructions for _sqlite3.so

If you'd like to use a different version of Python or SQLite than what is provided in this repo, you will need to build the static binary yourself. These instructions show you how to build the file: https://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/

After you've downloaded SQLite, follow the instructions to install it into a virtual environment. You must perform the installation on Amazon Linux or CentOS 7 (which Amazon Linux is based on).

django-s3-sqlite's People

Contributors

almogcohen avatar edgarroman avatar flipperpa avatar ipmb avatar jameswinegar avatar jjorissen52 avatar jnoortheen avatar kazlubr avatar pizzapanther avatar revolutiontech avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

django-s3-sqlite's Issues

Security Concerns

This database comes without any password. In my admin interface. How would one manage access to admin console?

'django_s3_sqlite' isn't an available database backend

  raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: 'django_s3_sqlite' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
    'mysql', 'oracle', 'postgresql', 'sqlite3'

have this error wirh django==2.2.10
any suggestions?

Multiple lambdas accessing same db

Hey,
first of all thanks for the package;
i am trying to use this library for e2e tests instead of using the actual db.
my problem is i have multiple lambdas accessing to the same db and it looks like one can not see the change done by the other lambda.
i think it is because the library copies the db file from s3 to local temporarily and uploads back if it has changed.
One lambda is triggering the other lambda and i think the db updated by the first lambda is not being uploaded back on time.

curious any idea how to address this issue. can we force upload somehow before triggering the other lambda?

The provided token is malformed or otherwise invalid

Hi there,

i'm getting:
botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the PutObject operation: The provided token is malformed or otherwise invalid.
I've been exploring the below (bottom of base.py) but no luck so far. Would you have any insights?
Thanks,
Andy.

 self.s3.Object(
                self.settings_dict["BUCKET"], self.settings_dict["REMOTE_NAME"],
            ).put(Body=file_bytes, ContentMD5=base64.b64encode(binascii.unhexlify(current_md5)).decode("utf-8"))

Could you provide instructions for how you created _sqlite3.so?

Hello, I'm running into an issue (not related at all to your repo), but found a post online referencing a problem you fixed by adding _sqlite3.so at the base of the repo. Can you please provide instructions as to how you created your _sqlite3.so file for python3.8?

Use Case Question

Hi there, I see this is exactly the package I want and it is maintained at Wharton!

But I also see that it's not recommended for serious use. So my question is, what would be a real life use case you've seen or an abstract example. Thanks!

Offer pysqlite3-binary as a solution instead of compiling your own

The Readme points to some instructions for building pysqlite3, but that may not actually be necessary. The pysqlite3 project offers precompiled binaries, I was able to get things working through just pip install pysqlite3-binary.

pysqlite3 does not replace sqlite3 though and by default, it seems Django will still attempt to import the normal package. I had to add these lines to my settings.py to get it to work.

# these three lines swap the stdlib sqlite3 lib with the pysqlite3 package
__import__('pysqlite3')
import sys
sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.