allensuvorov / tasker

Modified Proxy Service. Two micro-services. One - receives tasks, saves to DB, returns statuses. The other - polls new tasks, concurrently runs them, updates statuses.

Tasker - job scheduler

Project Description

The client sends a task to the service to perform an http request to a 3rd-party services. The task is described in json format, the generated task id is returned in response and its execution starts in the background.

Project Architecture

Two microservices. First microservice receives tasks, saves them to DB, returns statuses. Second microservice polls DB to get new tasks, concurrently runs them - makes HTTP requests to 3rd-party services and updates statuses in the DB.

• Project flowchart
• Project spec

Configuration

Set the postgres Database connection string / Data Source Name (DSN) to the env variable "DATABASE_DSN".

To test/run the app

To run and test the agent (job scheduler):

• NOTE: agent runs independently. No need to start the server.
• First, fill up the DB with test data by running this test:
  • path: internal/server/storage/db_test.go
  • test name: "TestTaskStorage_Create30Tasks_RealDB"
• Run the agent: cmd/agent/agent.go

To run and test the server:

• To start the server run: cmd/main/main.go
• For a POST request run this test:
  • path "internal/server/storage/db_test.go"
  • test name: "TestTaskStorage_CreateTask_RealDB"
• For a GET request run this test:
  • path: internal/server/storage/db_test.go,
  • test name: "TestTaskStorage_GetTaskStatus_RealDB"

tasker's Issues

Add tests for the main logic.

There are test for DB only now.

Response Body not read or closed

https://pkg.go.dev/net/http#Response.Body

"
// The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.
"

Problem: polling is not a perfect solution. Find another solution.

"Implemented task polling with timeout. With this approach, we can go to the database without receiving new data, or vice versa, forcing the user to wait an extra 500ms until it's time to go for tasks. At this point, we expected to see work with channels, when, after writing to the database, the request handler is immediately launched."

Which would it be, to keep services decoupled?

• message queque
• gRPC
• other?

Tests compliance checklist

• Tests need to be prod DB safe.
• Tests need to be autonomous.

Setup DSN via env/config

Set up DSN via local env

Portfolio ready - check list

• resolve all issues
• tests
• describe the functionality and what's implemented (polling etc.)
• short video - 3-5 minutes

Remove these global variables taskCh & resultCh

Remove these global variables taskCh & resultCh

• var taskCh = make(chan entity.TaskEntity)
• var resultCh = make(chan entity.ResultEntity, 5)

Try not to use global vars, they make it harder to write tests.

Mode Domain up to Internal folder

Domain contains items used by both, service and agent. Currently domain folder is in service folder. It should be up in the Internal folder.

Tests need to be prod DB safe.