Coder Social home page Coder Social logo

gcp-log-manager's Introduction

GCP Log Manager

GCP Log Manager is a tool designed to collect app logs from devices and provide analytics about daily user statistics. This program simplifies the process of managing logs using various Google Cloud Platform (GCP) services.

System Overview

  1. The GCP Log Manager collects app logs from devices and provides analytics about daily user statistics.
  2. The workflow involves event logs sent to the /api/postLog endpoint, which are published to Google Cloud Pub/Sub.
  3. A Dataflow job then populates a Google BigQuery table, and the /api/getAnalytics API call queries BigQuery to retrieve daily user statistics.

This solution streamlines log management, facilitates data analysis, and empowers users to make informed decisions based on log data.

Installation and Setup

.env Setup

Place create .env files on both server and scripts directory by following the .env.sample format with GCP account specific values.

GCP Service Account Credentials

Place your GCP service account credentials JSON file inside the project directory (./gcp-log-manager). Name the JSON file gcp-service-account.json.

Running the Application

Dockerized Version

To run the GCP Log Manager as a Docker container, follow these steps:

  1. Make the start script executable:
chmod +x ./start.sh
  1. Run the start script to build the Docker image, start the application container, and manage the Dataflow job:
./start.sh
  1. Prompt exit to close the application and remove the container/image

Non-Dockerized Version

To run the GCP Log Manager without Docker, follow these steps:

  1. Make the start script and related scripts executable:
chmod +x ./scripts/startDataflow.sh ./scripts/cancelDataflow.sh
  1. Install the required dependencies by running the following command:
npm install
  1. Start the application using Node.js:
cd ./server; node ./index.js
  1. You can stop the application with Command + C (CTRL + C) command.

API Endpoints

  1.  `POST localhost:8080/api/postLog`
    

This endpoint is used to send logs to the GCP Log Manager. The expected input format is as follows:

{
 "type": "$type",
 "session_id": "$session_id",
 "event_name": "$event_name",
 "event_time": $event_time,
 "page": "$page",
 "country": "$country",
 "region": "$region",
 "city": "$city",
 "user_id": "$user_id"
}

This endpoint returns a success message as a response.

  1. GET localhost:8080/api/getAnalytics

This endpoint provides aggregated daily user statistics. It returns data in a specific format as follows:

{
   "total_users": 916,
   "daily_stats": [
         {
            "date": "2022-01-16",
            "average_session_duration": 304,
            "active_user_count": 267,
            "new_user_count": 10
        },
        ...
      ]
}

Using Helm for Deployment

  1. Build and push the docker image of this project to an image repository.
  2. Change the values of the "image" variables on ./helm/deployement.yaml and ./helm/templates/values.yaml
  3. Adjust deployment and service settings.
  4. Deploy the service with:
helm install gcp-log-manager .

Dependencies

The GCP Log Manager uses the following packages:

  • @google-cloud/bigquery: "^7.2.0"
  • @google-cloud/dataflow: "^3.0.0"
  • @google-cloud/pubsub: "^4.0.1"
  • body-parser: "^1.20.2"
  • dotenv: "^16.3.1"
  • express: "^4.18.2"
  • shelljs: "^0.8.5"

gcp-log-manager's People

Contributors

moghun avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.