tiangolo / full-stack-fastapi-couchbase

Full stack, modern web application generator. Using FastAPI, Couchbase as database, Docker, automatic HTTPS and more.

🚨 DEPRECATION WARNING 🚨

I'm currently not actively using Couchbase nor this generator for any project.

You might still find some internal pieces of code useful for your own use cases, but I won't be able to fix bugs and add features.

If you are starting a new project from scratch, check the alternatives at the FastAPI docs: Project Generation.

You are still free to use this project if you want to, you might still find some internal pieces of code useful for your own use case. And if you already have a project generated with it that's fine as well (and you probably already updated it to suit your needs).

Full Stack FastAPI Couchbase - project generator

To have in mind

Here are some extra thoughts you might want to consider if you decide to go with Couchbase and/or this project generator.

Pros

Couchbase has a great set of features that is not easily or commonly found in alternatives.

It's a distributed database, so, if you have a cluster of several nodes with the data replicated, you don't have a single point of failure.

It provides very high performance.

It has built-in full-text search integrated (using Bleve). It's possible to make all the records be automatically indexed in the search engine without additional code or extra components.

It can be connected/integrated with the Couchbase Sync Gateway, that can be used to synchronize a subset of documents (records) with mobile devices. And with some effort, it can be connected with PouchBD for frontend synchronization (although not querying and/or searching from the frontend, as would be possible with CouchDB).

Cons

The documentation is very scarce. This is sadly a big problem. Many features are not well documented, or not documented at all.

Many configurations are not documented and have to be inferred from comments in forums, reading source code, or checking the configurations for the C client and trying different query parameters in URLs, that becomes quite error prone.

Couchbase didn't seem to be designed to be integrated into CI systems as in this project (or alternatives), at least until the last time I used it. It expects all the set up to be done once via the web UI, not automated. The official Docker image can't be configured. So, this project does all the configuration by sending the HTTP requests from the backend code to the container replicating the HTTP requests done in the web UI. But those steps are not documented, there's no "official" way to configure and start it without using the web UI, so the integration with CI could be error prone.

The Couchbase Sync Gateway official Docker image is not designed to be configured much either, so, the Dockerfile included in this project adds a good amount of custom logic to support that, but that's not really official.

As it uses N1QL, a flavor of SQL, and there's no easy integration with Python, you have to do all the operations in N1QL strings and integrate them with your own code. Without editor support, completion, nor syntax checks for N1QL.

---

Generate a backend and frontend stack using Python, including interactive API documentation.

Features

• Full Docker integration (Docker based).
• Docker Swarm Mode deployment.
• Docker Compose integration and optimization for local development.
• Production ready Python web server using Uvicorn and Gunicorn.
• Python FastAPI backend:
  • Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic).
  • Intuitive: Great editor support. Completion everywhere. Less time debugging.
  • Easy: Designed to be easy to use and learn. Less time reading docs.
  • Short: Minimize code duplication. Multiple features from each parameter declaration.
  • Robust: Get production-ready code. With automatic interactive documentation.
  • Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI and JSON Schema.
  • Many other features including automatic validation, serialization, interactive documentation, authentication with OAuth2 JWT tokens, etc.
• Secure password hashing by default.
• JWT token authentication.
• CORS (Cross Origin Resource Sharing).
• Celery worker that can import and use code from the rest of the backend selectively (you don't have to install the complete app in each worker).
• NoSQL Couchbase database that supports direct synchronization via Couchbase Sync Gateway for offline-first applications.
• Full Text Search integrated, using Couchbase.
• REST backend tests based on Pytest, integrated with Docker, so you can test the full API interaction, independent on the database. As it runs in Docker, it can build a new data store from scratch each time (so you can use ElasticSearch, MongoDB, or whatever you want, and just test that the API works).
• Easy Python integration with Jupyter Kernels for remote or in-Docker development with extensions like Atom Hydrogen or Visual Studio Code Jupyter.
• Email notifications for account creation and password recovery, compatible with:
  • Mailgun
  • SparkPost
  • SendGrid
  • ...any other provider that can generate standard SMTP credentials.
• Vue frontend:
  • Generated with Vue CLI.
  • JWT Authentication handling.
  • Login view.
  • After login, main dashboard view.
  • Main dashboard with user creation and edition.
  • Self user edition.
  • Vuex.
  • Vue-router.
  • Vuetify for beautiful material design components.
  • TypeScript.
  • Docker server based on Nginx (configured to play nicely with Vue-router).
  • Docker multi-stage building, so you don't need to save or commit compiled code.
  • Frontend tests ran at build time (can be disabled too).
  • Made as modular as possible, so it works out of the box, but you can re-generate with Vue CLI or create it as you need, and re-use what you want.
• Flower for Celery jobs monitoring.
• Load balancing between frontend and backend with Traefik, so you can have both under the same domain, separated by path, but served by different containers.
• Traefik integration, including Let's Encrypt HTTPS certificates automatic generation.
• GitLab CI (continuous integration), including frontend and backend testing.

How to use it

Go to the directory where you want to create your project and run:

pip install cookiecutter
cookiecutter https://github.com/tiangolo/full-stack-fastapi-couchbase

Generate passwords

You will be asked to provide passwords and secret keys for several components. Open another terminal and run:

openssl rand -hex 32
# Outputs something like: 99d3b1f01aa639e4a76f4fc281fc834747a543720ba4c8a8648ba755aef9be7f

Copy the contents and use that as password / secret key. And run that again to generate another secure key.

Input variables

The generator (cookiecutter) will ask you for some data, you might want to have at hand before generating the project.

The input variables, with their default values (some auto generated) are:

• project_name: The name of the project

• project_slug: The development friendly name of the project. By default, based on the project name

• domain_main: The domain in where to deploy the project for production (from the branch production), used by the load balancer, backend, etc. By default, based on the project slug.

• domain_staging: The domain in where to deploy while staging (before production) (from the branch master). By default, based on the main domain.

• docker_swarm_stack_name_main: The name of the stack while deploying to Docker in Swarm mode for production. By default, based on the domain.

• docker_swarm_stack_name_staging: The name of the stack while deploying to Docker in Swarm mode for staging. By default, based on the domain.

• secret_key: Backend server secret key. Use the method above to generate it.

• first_superuser: The first superuser generated, with it you will be able to create more users, etc. By default, based on the domain.

• first_superuser_password: First superuser password. Use the method above to generate it.

• backend_cors_origins: Origins (domains, more or less) that are enabled for CORS (Cross Origin Resource Sharing). This allows a frontend in one domain (e.g. https://dashboard.example.com) to communicate with this backend, that could be living in another domain (e.g. https://api.example.com). It can also be used to allow your local frontend (with a custom hosts domain mapping, as described in the project's README.md) that could be living in http://dev.example.com:8080 to communicate with the backend at https://stag.example.com. Notice the http vs https and the dev. prefix for local development vs the "staging" stag. prefix. By default, it includes origins for production, staging and development, with ports commonly used during local development by several popular frontend frameworks (Vue with :8080, React, Angular).

• smtp_port: Port to use to send emails via SMTP. By default 587.

• smtp_host: Host to use to send emails, it would be given by your email provider, like Mailgun, Sparkpost, etc.

• smtp_user: The user to use in the SMTP connection. The value will be given by your email provider.

• smtp_password: The password to be used in the SMTP connection. The value will be given by the email provider.

• smtp_emails_from_email: The email account to use as the sender in the notification emails, it would be something like [email protected].

• couchbase_user: Couchbase main user to be used by the application (code). By default admin.

• couchbase_password: Password of the main user, for the backend code. Generate it with the method above.

• couchbase_sync_gateway_cors: List of CORS origins that the Sync Gateway should allow to talk to directly. Similar to backend_cors_origins.

• couchbase_sync_gateway_user: User to be created for the Couchbase Sync Gateway. This is what allows synchronization using the CouchDB protocol, with Couchbase Lite in mobile apps and PouchDB in the web and hybrid mobile apps.

• couchbase_sync_gateway_password: Couchbase Sync Gateway password. Generate it with the method above.

• traefik_constraint_tag: The tag to be used by the internal Traefik load balancer (for example, to divide requests between backend and frontend) for production. Used to separate this stack from any other stack you might have. This should identify each stack in each environment (production, staging, etc).

• traefik_constraint_tag_staging: The Traefik tag to be used while on staging.

• traefik_public_network: This assumes you have another separate publicly facing Traefik at the server / cluster level. This is the network that main Traefik lives in.

• traefik_public_constraint_tag: The tag that should be used by stack services that should communicate with the public.

• flower_auth: Basic HTTP authentication for flower, in the formuser:password. By default: "root:changethis".

• sentry_dsn: Key URL (DSN) of Sentry, for live error reporting. If you are not using it yet, you should, is open source. E.g.: https://1234abcd:[email protected]/30.

• docker_image_prefix: Prefix to use for Docker image names. If you are using GitLab Docker registry it would be based on your code repository. E.g.: git.example.com/development-team/my-awesome-project/.

• docker_image_backend: Docker image name for the backend. By default, it will be based on your Docker image prefix, e.g.: git.example.com/development-team/my-awesome-project/backend. And depending on your environment, a different tag will be appended ( prod, stag, branch ). So, the final image names used will be like: git.example.com/development-team/my-awesome-project/backend:prod.

• docker_image_celeryworker: Docker image for the celery worker. By default, based on your Docker image prefix.

• docker_image_frontend: Docker image for the frontend. By default, based on your Docker image prefix.

• docker_image_sync_gateway: Docker image for the Sync Gateway. By default, based on your Docker image prefix.

How to deploy

This stack can be adjusted and used with several deployment options that are compatible with Docker Compose, but it is designed to be used in a cluster controlled with pure Docker in Swarm Mode with a Traefik main load balancer proxy handling automatic HTTPS certificates, using the ideas from DockerSwarm.rocks.

Please refer to DockerSwarm.rocks to see how to deploy such a cluster in 20 minutes.

More details

After using this generator, your new project (the directory created) will contain an extensive README.md with instructions for development, deployment, etc. You can pre-read the project README.md template here too.

Sibling project generators

• Based on PostgreSQL: https://github.com/tiangolo/full-stack-fastapi-postgresql.

Release Notes

Latest Changes

• Fix Windows line endings for shell scripts after project generation with Cookiecutter hooks. PR #28.
• Remove package-lock.json to let everyone lock their own versions (depending on OS, etc).
• Simplify Traefik labels for services. PR #27.
• Fix Flower Docker configuration. PR #24.
• Update testing scripts and typo.
• Add normal user Pytest fixture. PR #23.
• Update Dockerfiles to use Couchbase from Debian image. PR #20 by @Gjacquenot.
• Use new Pydantic types. PR #21 by @tehtbl.

0.4.0

• Fix security on resetting a password. Receive password and token as body, not query. PR #16.

• Forward arguments from script to pytest inside container.

• Update Jupyter Lab installation and util script/environment variable for local development.

0.3.0

• PR #15:
  • Update CRUD utils to use types better.
  • Simplify Pydantic model names, from UserInCreate to UserCreate, etc.
  • Upgrade packages.
  • Add new generic "Items" models, crud utils, endpoints, and tests. To facilitate re-using them to create new functionality. As they are simple and generic (not like Users), it's easier to copy-paste and adapt them to each use case.
  • Update endpoints/path operations to simplify code and use new utilities, prefix and tags in include_router.
  • Update testing utils.
  • Update linting rules, relax vulture to reduce false positives.
  • Add full text search for items.
  • Update project README.md with tips about how to start with backend.

0.2.1

• Fix frontend hijacking /docs in development. Using latest https://github.com/tiangolo/node-frontend with custom Nginx configs in frontend. PR #14.

• Update generated README. Minor typos.

• Update Couchbase installation, to include Couchbase command line tools.

• Set /start-reload.sh as a command override for development by default.

0.2.0

PR #7:

• Simplify and update backend Dockerfiles.
• Refactor and simplify backend code, improve naming, imports, modules and "namespaces".
• Improve and simplify Vuex integration with TypeScript accessors.
• Standardize frontend components layout, buttons order, etc.
• Add local development scripts (to develop this project generator itself).
• Add logs to startup modules to detect errors early.
• Improve FastAPI dependency utilities, to simplify and reduce code (to require a superuser).
• Fix/update logic to update users.

License

This project is licensed under the terms of the MIT license.

full-stack-fastapi-couchbase's Issues

Bad Gateway

Hello! Im want create startup disk, and see this error "Bad gateway". Say, pls, how to create startup disk???

[QUESTION] Where to load a Large pickled ML model?

Hi @tiangolo ! I'm having an amazing time developing my projects with FastAPI. I've pointed it out before: this is by far my favorit project on all of github. :)

I have a question, that might perhaps be trivial, for you to answer.

So I have an API where I am loading i Pickled Machine Learning model. Currently I am doing pickle.load(file_model) in a file located inside of ./backend/app/app/api/api_v1/endpoints, but I have also tried loading it inside of ./backend/app/app/api/api_v1/app.py, and ./backend/app/app/main.py. But every time I open http://localhost/docs, it prompts the API to run pickle.load(file_model) again, which takes a lot of load time.

My question: where should I put the pickle.load(file_model), so it only has to load once when the API is started, not every time the user makes a request?

WORKER TIMEOUT +502 Bad gateway

Hi @tiangolo !

As always I'm impressed by your work. As I've said many times before, FastAPI is my absolute favourite project on all of GitHub.

I've set up an API that seems to work fine. However, when I post a large request I get 502 Bad Gateway as a response. When I check the docker log in backend_1 I get the following error message.

[2019-03-31 18:56:46 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:211)
[2019-03-31 18:56:47 +0000] [372] [INFO] Booting worker with pid: 372

Why is the worker timing out? Also when I time how long it takes for the timeout it always varies. Sometimes its after 20 seconds, sometimes 4 seconds, sometimes 15 seconds etc.

Any plan for creating a SQL version?

I'm loving this full-stack but I don't think CouchDB is right for my purposes. Having SQL could be nice.

Can't login in the forntend

Thanks again for a truly AWESOME project!

I've successfully run:

docker-compose up -d

However, when I go to http://localhost, I can't login with the credentials a choose. I'm assuming I'm supposed to use first_superuser and first_superuser_password . Have I missed a step after running docker-compose , or is it a bug?

Broken docker build

docker-compose up -d                                                                                                                                                          ✔
Building backend
Step 1/9 : FROM tiangolo/uvicorn-gunicorn-fastapi:python3.6
 ---> 71af2e82d548
Step 2/9 : RUN wget -O - http://packages.couchbase.com/ubuntu/couchbase.key | apt-key add -
 ---> Using cache
 ---> 8ac43f6bf52e
Step 3/9 : RUN echo "deb http://packages.couchbase.com/ubuntu stretch stretch/main" > /etc/apt/sources.list.d/couchbase.list
 ---> Using cache
 ---> 302582854baf
Step 4/9 : RUN apt-get update && apt-get install -y libcouchbase-dev libcouchbase2-bin build-essential
 ---> Running in 2d0088f6b501
Get:1 http://packages.couchbase.com/ubuntu stretch InRelease [3036 B]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:4 http://packages.couchbase.com/ubuntu stretch/stretch/main amd64 Packages [1438 B]
Get:5 http://security.debian.org/debian-security buster/updates InRelease [39.1 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7899 kB]
Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [99.2 kB]
Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [5792 B]
Fetched 8218 kB in 2s (4472 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libcouchbase2-bin : Depends: libcouchbase2-libevent (= 2.10.3-1) but it is not going to be installed
                     Depends: libevent-core-2.0-5 (>= 2.0.10-stable) but it is not installable
E: Unable to correct problems, you have held broken packages.
ERROR: Service 'backend' failed to build: The command '/bin/sh -c apt-get update && apt-get install -y libcouchbase-dev libcouchbase2-bin build-essential' returned a non-zero code: 100

Frontend / Backend separate deployment

Hi Tiangolo

Hope everything is going great from your side!

Just a "tactical" question about how would be the best approach to accommodate this full-stack to be deployed separately:

Frontend
Netlify recently incorporated Vue to the frameworks available for deployments. To take advantage of this new feature, once you built full-stack-fastapi-couchbase's frontend, what would have to take into account to allow the exchange of information with the backend?

Backend
AWS/GCP/Heroku support the containerisation of several services. I read the dockerswarm.rocks info but when you don't have root access and not explicit IPs, how could I securely expose each service/container of the swarm to be connected by the frontend running on Netlify?

Cheers and thanks in advance for your attention!

Minimum swarm requirements?

What are the minimum requirements for deploying this project? dockerswarm.rocks says: You can set it up, deploy your applications and do everything on a $5 USD/month server. However, I think I'm running out of memory while building the frontend on a 1GB VPS (works when I resize to 2GB). I'm building in a gitlab runner in the Manager Node.

Here are the runner logs:

...
 ---> 65890b36a2c2
Step 9/13 : RUN npm run build
 ---> Running in ca64bfe078cd

> [email protected] build /app
> vue-cli-service build

-  Building for production...
Starting type checking and linting service...
Using 1 worker with 2048MB memory limit
Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
Killed
npm ERR! code ELIFECYCLE
npm ERR! errno 137
npm ERR! [email protected] build: `vue-cli-service build`
npm ERR! Exit status 137
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-07-11T13_35_19_185Z-debug.log
Service 'frontend' failed to build: The command '/bin/sh -c npm run build' returned a non-zero code: 137

Hide endpoint in interactive documentation

Hi! this is not really a bug or an issue. It's more of a questions.

When I deploy the project I don't want /api/v1/api/v1/users/ and some other endpoints to displayed in the interactive Swagger/Redoc documentation. In other words, I want to hide them from the interactive documentation. Is this possible somehow?

Bad gateway error installing on macOs Mojave with Docker for Mac

Happy New Year Tiangolo!

I've just noticed the Impressive work you've done with FastAPI. Congratulations! Great Job!

I wanted to give a try to this new version, having no problems before with full-stack-flask-couchbase, but this time some services seem to have a problem with the gateway. The frontend on localhost didn't allow me to login, the localhost/api page shows a "Bad gateway" message, and localhost/docs and localhost/redoc pages didn't start properly. On the contrary, Couchbase console, Celery Flower and Traefik pages worked perfectly.

Following the workarounds provided by Docker here https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds, I couldn't find a way to determine the Docker engine IP to redirect dev..com.

Would it be better to work full-stack-fastapi-couchbase with Docket Toolbox instead of Docker for Mac?

Any guidance is much appreciated!

Unable to login using the frontend

When I try to log in using the frontend the backend returns:

[2019-01-10 14:55:29 +0000] [17] [INFO] ('172.24.0.6', 41932) - "POST /api/v1/login/access-token HTTP/1.1" 422

I tested the /api/v1/login/access-token API method using Postman by sending the body as json and also as form encoded and it worked correctly. I also tested using the openapi interface and everything worked.

Any idea what might be wrong?

How can I display in the backend log the full request received by Fastapi?

422 Unprocessable Entity

Hi!

When I try to create a new user, using POST /api/v1/users/ I get the response 422 Unprocessable Entity, and the backend will just log:

[2019-02-28 16:15:06 +0000] [45] [INFO] ('172.20.0.2', 51610) - "POST /api/v1/users/ HTTP/1.1" 422

Anyone else has this problem. I've just used the template generator without any customisation.

Best practice for securing passwords on public repository?

What would be the best practice for securing passwords on deploying from a public repository?

Windows development line-ending error in sync-gateway and backend

When developing from windows I get the error:

Problem with /usr/bin/env: “bash\r”

And other issues related to the line endings, I added dos2unix to the dockerfile builds

Incomplete Instructions

After you go thru the instructions it stops at cookie cutter install vars. What do you do next?

On domain change frontend/.env and others don't use DOMAIN environmental variable

Found this when switching domains it was still pointing at the old api

COUCHBASE_SYNC_GATEWAY_CORS_ORIGINS
in env-backend.env and env-sync-gateway.env

in frontend /.env
VUE_APP_DOMAIN_STAG
VUE_APP_DOMAIN_PROD