Coder Social home page Coder Social logo

ad-manager-alerter's Introduction

Ad Manager Anomaly detection

Please note: this is not an officially supported Google product.

This project is meant to serve as inspiration for how you can do anomaly detection for an Ad Manager account by using the Ad Manager API and BigQuery ML.

Setup

This code repository is prepared to be deployed to run directly on Google Cloud.

Products needed:

  • Google Ad Manager
  • Google Cloud Project with billing enabled (billing is required for cloud functions, check https://cloud.google.com/free for the current free tier limits)

For the setup instructions below we're using Google Cloud CLI to run some commands for us, if you haven't installed and configured that yet - do it now.

To get started, clone this repository and make a copy of the config file.

git clone https://github.com/google/ad-manager-alerter.git
cd ad-manager-alerter
cp src/config.example.py src/config.py

We will edit src/config.py during the rest of the steps below.

Create service account

Create a service account in Google Cloud.

Enter the service account email address as SERVICE_ACCOUNT in src/config.py.

Create a Google Cloud Bucket

Create a new bucket in Google Cloud Storage.

Save the name of the bucket in src/config.py.

Important! Make sure you grant access for the service account to read/write to this bucket.

Create a BigQuery dataset (used for alerting)

If you're not planning to use the alerting features you can skip this step.

Create a new BigQuery data set save the name of the dataset in src/config.py.

Important! Make sure you grant access for the service account to run queries onthis data set. You can do this by adding them as a Principal with the role "Big Query Data Owner".

Grant access to run Ad Manager Reports

For this project to work it needs to first fetch reports from the Ad Manager API. If you already are using the API then step 1 can be ignored.

  1. Enable API access in Ad Manager
  2. Add the service account email to Ad Manager with rights to run reports. (Eg. role "Executive")

Deploy to cloud

Making sure we can send emails

If you're not planning to use the alerting features you can skip this step.

Making sure emails are sent and received correctly can be tricky, to make it easier this example uses SendGrid in order to send emails.

If you don't have a SendGrid account you can create one throught the SendGrid page on Cloud Marketplace.

Once your sendgrid account is created you can follow the steps to setup a send account and get your API KEY.

Enable services to deploy to functions / workflows

First time running the commands below you might be prompted to enable some APIs to be able to deploy.

If you want to you can enable them beforehand with the following command:

gcloud services enable artifactregistry.googleapis.com cloudbuild.googleapis.com cloudfunctions.googleapis.com cloudscheduler.googleapis.com containerregistry.googleapis.com  logging.googleapis.com monitoring.googleapis.com pubsub.googleapis.com run.googleapis.com workflows.googleapis.com

Deploy Cloud Function

gcloud functions deploy ad-manager-alerter \
 --source=src/ --trigger-http --gen2 --runtime=python311 \
 --region=us-central1 --entry-point=ad_manager_alerter \
 --memory=512M --timeout=3600 \
 --service-account=[YOUR_SERVICE_ACCOUNT] \
 --set-env-vars SENDGRID_API_KEY=[SENDGRID_API_KEY]

When this command finishes you should see something like:

...
  uri: https://your_cloud_function-abc123.run.app
state: ACTIVE

Copy and save the "uri", you need this in the next stage.

Deploy as Cloud Workflow

Copy the workflow.example.yaml to workflow.yaml and edit the cloud function URL to match the URL you got from the previous step.

If you want to only run parts of this example library (eg. only export to GCS) modify this file and remove the steps you don't want to run.

gcloud workflows deploy ad-manager-alert-workflow --source workflow.yaml

Verify that it's working by running the workflow, this can be done in the terminal via:

gcloud workflows run ad-manager-alert-workflow

If it runs without issue now could be a good time to set it up to run on a schedule via Cloud Scheduler.

If there are issues here, pause and fix them before continuing.

Deploy to Cloud Scheduler

When you verified it's working you can setup a new Cloud Scheduler to run the workflow at an interval.

Overview of modules

In order to setup the entire workflow to detect anomalies we've created three modules, Ad Manager report downloader, BigQuery import and Model creation and the Anomaly Detector and alerter.

Ad Manager report downloader

This module is responsible for requesting and downloading Ad Manager reports into Google Cloud Storage.

It has two main entry points "download_report" and "download_all_saved_reports".

  • download_report is used for the report that will be used for alerting purposes.
  • download_all_saved_reports downloads all reports that have been shared with the service account used to run the script.

The authentication method chosen for this project is to generate access tokens directly from the Cloud Function as described here: Authenticating applications directly with access tokens.

BigQuery import and model creation

This module is responsible for importing the CSV from the report downloader into a BigQuery table, and build the forecasting model from that table given the settings in config.py.

Anomaly Detector and alerter

This module is reponsible for detecting anomalies and send email alerts with charts when they occur.

ad-manager-alerter's People

Contributors

demirj avatar gernberg avatar paulo-lopes-estevao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ad-manager-alerter's Issues

Improvement: Refactor the 'def ad_manager_alerter' function to make it more accessible and scalable.

def ad_manager_alerter(request):

def ad_manager_alerter(request):
  method = request.args["method"]
  if "download_report" == method:
    return report_downloader.download_report()
  elif "prepare_bq" == method:
    return ml_builder.prepare_bq()

...

I noticed that we can improve the 'def ad_manager_alerter' function by reducing multiple if-else statements and using a dictionary.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.