arpitha-debug / devops-course-starter

DevOps Apprenticeship: Project Exercise

System Requirements

The project uses poetry for Python to create an isolated environment and manage package dependencies. To prepare your system, ensure you have an official distribution of Python version 3.7+ and install poetry using one of the following commands (as instructed by the poetry documentation):

Poetry installation (Bash)

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

Poetry installation (PowerShell)

(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python

Dependencies

The project uses a virtual environment to isolate package dependencies. To create the virtual environment and install required packages, run the following from your preferred shell:

$ poetry install

You'll also need to clone a new .env file from the .env.template to store local configuration options. This is a one-time operation on first setup:

$ cp .env.template .env  # (first time only)

The .env file is used by flask to set environment variables when running flask run. This enables things like development mode (which also enables features like hot reloading when you make a file change). There's also a SECRET_KEY variable which is used to encrypt the flask session cookie.

Running the App

Once the all dependencies have been installed, start the Flask app in development mode within the poetry environment by running:

$ poetry run flask run

You should see output similar to the following:

 * Serving Flask app "app" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with fsevents reloader
 * Debugger is active!
 * Debugger PIN: 226-556-590

Now visit http://localhost:5000/ in your web browser to view the app.

Module-5 Docker

Docker

poetry version .The project uses Python 3.9.0 via a virtual environment created by poetry. poetry version=1.1.4 .To Build images: docker build --tag todo-app . .To Run Container for the image: docker run todo-app
.Move poetry from opt/poetry to current dir /project# mv /opt/pysetup/* .

.Commands used for building todoapp: docker build --tag todo-app . docker run -it --entrypoint /bin/bash todo-app

You should see output similar to the following: Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable.

.Provide --env-file .env while running the container. docker run --env-file .env todo-app Set the port to run on localhost with p variable docker run --env-file .env -p5000:5000 todo-app

Gunicorn Run

Changes made to wsgi.py for gunicorn. app = app.create_app() Added Entrypoint to run For Gunicorn ENTRYPOINT [ "poetry", "run", "gunicorn", "--bind", "0.0.0.0:5000", "todo_app.wsgi:app" ] Build Docker again: docker build --tag todo-app . Run Docker: docker run --env-file .env -p5000:5000 todo-app

You should see output similar to the following: Starting gunicorn 20.1.0 [2021-06-18 15:46:43 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)

Check on Browser Checked with localhost:5000 on browser http://localhost:5000/

Dockerfile

single-stage docker image. Dockerfile includes the steps to import your application and launch it. Create a Workdir. COPY to move files into your image.

Dependencies

Install poetry. To run the flask app ensure the dependencies in pyproject.toml are installed. Expose the port. EntryPoint using Gunicorn.

Dockerfile "multi-stage" from the same docker file. Multi-stage builds are used to generate different variants of a container. 1-development container 2-testing container
3-production container

Use below ENTRYPOINT to run Flask with poetry for dev environment. ENTRYPOINT [ "poetry", "run", "flask", "run", "--host", "0.0.0.0" ]

Docker File steps Explained:

FROM: are instructions to perform specific action. ARGUMENTS: everything on the right for instructions is an argument. Pull the image of python with version 3.9.0 as python-base.

FROM python:3.9.0-buster as python-base Below command creates a workdir called app. WORKDIR /app Below will Set the environment path. ENV PATH="$POETRY_HOME/bin:$PATH" Below From command creates a builder-base from python base. FROM python-base as builder-base
RUN apt-get update
&& apt-get install --no-install-recommends -y
curl
build-essential Install poetry using curl. ENV POETRY_VERSION=1.1.4 RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python COPY command copies the files from root /poetry.lock; moves files into your image. Copies the poetry.lock to pyprojec.toml. COPY ./poetry.lock ./pyproject.toml ./ Specify the Entrypoint for gunicorn. ENTRYPOINT [ "poetry", "run", "gunicorn", "--bind", "0.0.0.0:5000", "todo_app.wsgi:app" ]

builderbase as development

Will build from builderbase as development; will run poetry install through Run command and copies todoapp for dev environment. FROM builder-base as development RUN poetry install COPY ./todo_app /app/todo_app

Create either a development or production image from the same Dockerfile, using commands: $ docker build --target development --tag todo-app:dev . $ docker build --target production --tag todo-app:prod .

Builds the todo app image. Open the docker hub. Click on image. Todo app image is created.

To Run the container for image created in above step.

Below will create and start the container in dev/prod environment. $ docker run --env-file ./.env -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:dev $ docker run --env-file ./.env -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:prod

Open the docker hub. Click on Containers/Apps. New container will be started.

Test the local development

Docker command to build and run Todo App in development environment.

$ docker build --target development --tag todo-app:dev . $ docker run --env-file ./.env -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:dev

• Serving Flask app "todo_app/app" (lazy loading)
• Environment: development
• Debug mode: on
• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
• Restarting with stat
• Debugger is active!
• Debugger PIN: 145-903-278 Browse to http and change the port to 5100 http://localhost:5100/ ToDoApp is displayed on UI.

Build production environment

FROM builder-base as production RUN poetry install --no-dev
COPY ./todo_app /app/todo_app ENTRYPOINT [ "poetry", "run", "gunicorn", "--bind", "0.0.0.0:5000", "todo_app.wsgi:app" ]

Build and run Todo App in Production environment

$ docker build --target production --tag todo-app:prod . $ docker run --env-file ./.env -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:prod

• Serving Flask app "todo_app/app" (lazy loading)
• Environment: development
• Debug mode: on
• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
• Restarting with stat
• Debugger is active!
• Debugger PIN: 145-903-278 Browse to http and change the port to 5100 on browser to view todoapp http://localhost:5100/ ToDoApp is displayed in prod environment.

To view the container logs $ docker logs

To Compose Docker Launch your containers in a single yaml file (docker-compose.yml). To launch use a single command (docker-compose up). $ docker-compose up

Create a .dockerignore file

Specify files and directories that should never be copied to docker images which include. 1-vagrantfile 2-configuration, secrets( .envand .env.template) 3-unwanted IDE 4-tool configuration directories (.git).

ci.yml for Module7 ignore below docker build --target test --tag todo-app:test . docker run --env-file ./.env -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:test docker run --env-file .env.test todo-app:test /app/todo_app [ "poetry", "run", "pytest" ] docker run --env-file .env.test your-image todo_app/tests/unit == reference docker run --env-file .env.test todo-app:test /app/todo_app IGNORE ABOVE

Continuous Integration through GITHUB Actions

name:Continuous Integration which is iniitiated every time code is pushed; on: [push] jobs will build under name: Build and test which runs-on: ubuntu-latest using environment: TodoApp_Environment

Part-2 Set up GitHub Actions to repository

Added ci.yml workflow file in .gitignore under workflows. This workflow file checkout the code and then print "Hello World". To check the output of your build, open repository on the GitHub website > Actions tab. URL should be of the form https://github.com/<your_username>/<your_repository>/actions.

Part-3

Replace the "echo" command with command to build your projects test Docker image. docker build --target test --tag todo-app:test .

Set up Secrets

Git Hub > Actions > Secrets Before Running End to End Tests. Added docker login username and password in Git hub > Actions > Secrets. Login into Git Hub Account. Go to Settings. From Side Menu>> Select Secrets. Actions Secret page is displayed. Click on New Repository Secret. Enter Name :DOCKERHUB_USERNAME Enter Value :arpithadockerhub Click on >> Add Secret. Click on New Repository Secret. Enter Name :DOCKERHUB_TOKEN Enter Value :enter password for docker account Click on >> Add Secret.

Part 4: Run the tests

Step 1: Run the unit and integration tests

• name: Build Unit Tests run: docker build --target test --tag todo-app:test .
• name: Run Unit Test run: docker run --env-file .env.test todo-app:test /app/todo_app/tests

Step 2: Run the E2E tests For running the docker,we use below command which exposes env file (-env-file .env) which needs to be replaced by adding all env values to Secrets (Git Hub > Actions > Secrets).

docker run --env-file .env .test todo-app:test /app/todo_app/tests docker run -e BOARD_ID=${{ secrets.BOARD_ID }} Replace --env-file .env with -e BOARD_ID=${{ secrets.BOARD_ID }}

• name: Build End to End tests run: docker build --target test --tag todo-app:test .
• name: Run the end to end test

run: docker run -e TRELLO_BOARD_ID=${{ secrets.TRELLO_BOARD_ID }} -e TRELLO_TOKEN=${{ secrets.TRELLO_TOKEN }} -e TRELLO_API_KEY=${{ secrets.TRELLO_API_KEY }} -e ID_LIST_NEWLIST=${{ secrets.ID_LIST_NEWLIST }} -e ID_LIST_TODO=${{ secrets.ID_LIST_TODO }} -e ID_LIST_DONE=${{ secrets.ID_LIST_DONE }} -e ID_LIST_DOING=${{ secrets.ID_LIST_DOING }} -e CARDS_TRELLODONE_CARD=${{ secrets.CARDS_TRELLODONE_CARD }} -e CARDS_TODO_CARD_TWO=${{ secrets.CARDS_TODO_CARD_TWO }} -e CARDS_TODO_CARD_THREE=${{ secrets.CARDS_TODO_CARD_THREE }} -e CARDS_TODO_CARD_ONE=${{ secrets.CARDS_TODO_CARD_ONE }} -e CARDS_DEVOPSTRELLOBOARD=${{ secrets.CARDS_DEVOPSTRELLOBOARD }} -e FLASK_APP=${{ secrets.FLASK_APP }} -e FLASK_ENV=${{ secrets.FLASK_ENV }} -e SECRET_KEY=${{ secrets.SECRET_KEY }} -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:test /app/todo_app/SeleniumTest

Part 5: Update the build trigger

Currently we are only building "on push", which means whenever any branch is updated. on: push Another option is "on pull request" which runs for open pull requests, using a version of the codebase where merge has already completed.

• name: Build End to End tests run: docker build --target test --tag todo-app:test .
• name: Run the end to end test run: docker run -e TRELLO_BOARD_ID=${{ secrets.TRELLO_BOARD_ID }} -e TRELLO_TOKEN=${{ secrets.TRELLO_TOKEN }} -e TRELLO_API_KEY=${{ secrets.TRELLO_API_KEY }} -e ID_LIST_NEWLIST=${{ secrets.ID_LIST_NEWLIST }} -e ID_LIST_TODO=${{ secrets.ID_LIST_TODO }} -e ID_LIST_DONE=${{ secrets.ID_LIST_DONE }} -e ID_LIST_DOING=${{ secrets.ID_LIST_DOING }} -e CARDS_TRELLODONE_CARD=${{ secrets.CARDS_TRELLODONE_CARD }} -e CARDS_TODO_CARD_TWO=${{ secrets.CARDS_TODO_CARD_TWO }} -e CARDS_TODO_CARD_THREE=${{ secrets.CARDS_TODO_CARD_THREE }} -e CARDS_TODO_CARD_ONE=${{ secrets.CARDS_TODO_CARD_ONE }} -e CARDS_DEVOPSTRELLOBOARD=${{ secrets.CARDS_DEVOPSTRELLOBOARD }} -e FLASK_APP=${{ secrets.FLASK_APP }} -e FLASK_ENV=${{ secrets.FLASK_ENV }} -e SECRET_KEY=${{ secrets.SECRET_KEY }} -p 5100:5000 --mount type=bind,source="$(pwd)"/todo_app,target=/app/todo_app todo-app:test /app/todo_app/SeleniumTest

Stage , commit and push the changes to github. Check in Actions tab by opening Github account. https://github.com/Arpitha-debug/DevOps-Course-Starter/runs/4536265282?check_suite_focus=true

Project-9.

Create a MongoDB cluster that our application we use. choose a location near you with a free-tier available. Cloud:Azure, Free-Tier Region:northeurope, Location: Ireland. When creating a database below authentication method: username and password. Add your local IP address when prompted so you can access the cluster from your local machine. Get connected to your cluster."Connect your application".

Step 2: Installing Pymongo. Python support for MongoDB comes in the form of PyMongo. Add this dependency to your project with poetry.

Installing Pymongo. poetry add pymongo pymongo[srv]

In .env add: FLASK_APP=todo_app/mongo

in wsgi add below: from todo_app import mongo app = mongo.create_app()

in mongo.py: from flask import Flask, render_template, request, redirect, url_for, session import requests, os,pymongo, pymongo, MongoClient, certifi, ssl

Add client in create_app: client = pymongo.MongoClient("${{ secrets.MONGO_URI }}", ssl_cert_reqs=ssl.CERT_NONE)
todos_collection = client.db.todos

Add a route for root, additem, deletecard.

MongoDB: mongodb+srv://:@cluster0.odgud.mongodb.net/myFirstDatabase?retryWrites=true&w=majority collection in mango DB = todos added getitems to list the items on Todo App Index.html {% for todo in itemcollection %}

{{todo}}

    {% endfor %}

    Able to add and delete items from Todo app to Mongo

MODULE-10

    Github Oauth
    Sign in github account: https://github.com/settings/developers
    Github sign in> Settings> Developer Settings >Oauth Apps> Register a new Application
    Application name: Todo Application
    Homepage URL: http://localhost:8000
    Authorization callback URL: http://localhost:8000/login/callback

You will setup a Authorization Callback URL when you create your new application on Github. to obtain an Access Key, use prepare_token_request to help generate the correct URL/headers/body response = requests.post(token_url,params= {"client_id": client_id, "client_secret": client_secret, "code":request.args['code'] }, headers={"Accept":"application/json"}) 2- Pull out the access token client.parse_request_body_response(response.text) 3-Add the code to url user_uri, headers, body = client.add_token(api_base_url) 4-Get the response user_response = requests.get(api_base_url, headers = headers) 5.Convert the response to Json. data = user_response.json() 6-Extract the id from json. login_user(load_user(data["id"])) 7-Login using id. #login_user(data.id, remember=False, duration=None, force=False, fresh=True) 8-Redirect to homepage. return redirect(url_for("index"))
If the request is successfull Github redirects to that URL with a 'code' parameter.

Include : OAUTHLIB_INSECURE_TRANSPORT=1 in the environment file. Check all the previous endpoints require authentication and your changes haven't broken anything. For unit/integration tests: Set LOGIN_DISABLED=True in .env.test

For end-to-end test's sat the top of the end-to-end test's setup fixture with. os.environ['LOGIN_DISABLED'] = 'True' Part 2 - Adding authorisation. Step 1 - Add mapping to determine role from user. Hardcode a mapping from a user_id to their role. Step 2 - Add checks to endpoints for role. You can access the user object for the logged in user via current_user. use a decorator to avoid duplicating code for role checks.

Step 3 - Alter frontend based on role fter Step 2 the user should be rejected from doing any action they do not have permission to. Update the frontend to avoid showing options to create or alter existing to-dos for users who have the "reader" role. Part 3: CI/CD pipeline and the live site Check that your pipeline is still running successfully. Register a second application on GitHub for the live site because the redirect URL will be different. Use the new client ID and secret in Heroku, different from the values you are using for local development.

• Authentication is implemented correctly: • The app forces you to authenticate before allowing you to access the To-Dos. • Everyone should be able to view the To-Dos after authenticating, i.e. the app should grant the reader role by default

• Authorisation is implemented correctly: • Only users with the writer role can add and update To-Do items. • This authorisation is enforced both on the front end and server side. • The new version of the app is deployed and working on Heroku. • The app still satisfies the criteria from the previous exercises. Check tests locally before pushing to git with. poetry run pytest
poetry run flask run

Module-11

Step 1: Set up Azure Account Step 2: Install the cli.install-azure-cli Open up new terminal window and enter az login, which will launch a browser window to allow you to log in.

Part 2: Setting Up Your Cloud Step 1: Locate Your Resource Group:locate your project exercise resource group in the Azure portal and keep a note of its name. Step 2: Set up a Cosmos database with Mongo API. With the Portal: • New -> CosmosDb Database • Select "Azure Cosmos DB for MongoDB API" in the API field • for now you should permit access from "All Networks" to enable easier testing of the integration with the app. Step 3: Connect your app to the CosmosDB via Portal: • Open the resource, then go to the "Connection String" page in the sidebar. • Update your application to use this connection string instead of the Atlas one (mongodb.net). • Now build & run the production app locally using this connection string, and try using it to check all functionality works.

Part 3: Host the frontend on Azure Web App. Step 1: Put Container Image on DockerHub registry. Step 2: Create a Web App. Portal: • Create a Resource -> Web App. • Fill in the fields, be sure to select "Docker Container" in the "Publish". field, and choose your newly-created Resource Group. On the next screen, select DockerHub in the "Image Source" field, and enter the details of your image(arpithadockerhub/todo_app:latest). Step 3: Set up environment variables. • Portal: • Settings -> Configuration in the Portal. • Add all the environment variables as "New application setting". • add all relevant Env variable from .env

Step 4: Confirm the live site works . • Browse to http://<webapp_name>.azurewebsites.net/ and confirm no functionality has been lost.

Part 4: Set up Continuous Deployment Find your webhook URL. • This can located under Container Services on the app service's page in the Azure portal. • Test your webhook URL. • Take the webhook provided by the previous step, add in a backslash to escape the $, and run on your bash or cmd : curl -dH -X POST "" • eg: curl -dH -X POST "https://
$<deployment_username>:<deployment_password>@<webapp_name>.scm.azurewebsites.net/docker/hook". • This should return a link to a log-stream relating to the re-pulling of the image and restarting the app.

• Add the command to your CD pipeline. • The webhook URL is sensitive because it includes a password for deploying your app.
Add it to your GitHub repository as a secret.

• Run the curl command as a step in your deployment job in your GitHub. Actions workflow, referencing the secret that holds the webhook URL.

• name: Deploy to Azure Web App services run: bash ./todo_app/webhook.sh

• Removed all references to Heroku.

WARNING: The webhook URL contains a dollar sign which should be escaped (\$) to prevent it being interpreted as an attempt to reference an environment variable. Added webhook.sh:curl -dH --fail -X POST "$WEBHOOK_URL". added in run.sh: #!/bin/bash poetry run gunicorn --bind 0.0.0.0:$PORT "todo_app.app:create_app()". added in Dockerfile: FROM builder-base as production #RUN poetry install --no-dev
RUN poetry config virtualenvs.create false --local && poetry install --no-dev COPY ./todo_app /app/todo_app ENV PORT = 5000 ENTRYPOINT ["bash", "./todo_app/run.sh"]

In ci.yml added two jobs 1.Build_job To build and test (run unit tests). Added environment: TodoApp_Environment, Repository secrets. 2-Deploy: name: Deploy to Azure runs-on: ubuntu-latest environment: TodoApp_Environment, Repository secrets env: WEBHOOK_URL: "${{ secrets.WEBHOOK_URL }}" needs: Build_job if: github.ref == 'refs/heads/master'

  and final step to deploy to Azure
  - name: Deploy to Azure Web App services
    run:   bash ./todo_app/webhook.sh

On CMD run below to get the logstream URL curl -dH -X POST WebhookURL Browse this url to find the log https://am-project-test-appservice.scm.azurewebsites.net/api/logstream?filter=op:6eaab31a-dc47-4703-8469-1c7410f78ae8,volatile:false