shortlink-org / shortlink

Shortlink service (Microservice example) ⭐️ Star the repo if you like it!

shortlink

ShortLink is an open-source educational project that provides a pretty user interface and respects GDPR.

The goal of the project is to demonstrate the practical application of microservices architecture.

---

---

High Level Architecture 🚀

The project covers the entire process - from identifying Bounded Contexts to implementing microservices using cutting-edge technologies and best practices.

Please star ⭐ the repo if you want us to continue developing and improving ShortLink! 😀

Boundaries

Tip

Our project follows Domain-Driven Design (DDD) principles, organizing code into distinct domains for clarity and easier updates.

Bounded Context	Description	Type subdomain	Docs
API Gateway	Gateway for all services	Supporting	docs
Auth Boundary	Auth services	Generic	docs
Billing Boundary	Payment services	Generic	docs
Chat Boundary	Chat services	Supporting	docs
Delivery Boundary	Delivery services	Supporting	docs
DS Boundary	Data Science services	Supporting	docs
Extension Boundary	Extension services	Supporting	docs
Link Boundary	Link services	Core	docs
Marketing Boundary	Marketing services	Supporting	docs
Notification Boundary	Notification services	Generic	docs
Platform Boundary	Platform services	Supporting	docs
Search Boundary	Search services	Supporting	docs
Shop Boundary	Shop services	Supporting	docs
ShortDB Boundary	ShortDB services	Supporting	docs
UI Boundary	UI services	Supporting	docs

Contributing

• Getting Started

Architecture decision records (ADR)

Important

An architecture decision record (ADR) is a document that captures an important architecture decision made along with its context and consequences.

• Docs ADR

Decisions:

• main decisions
• ops decisions

Also, each boundary context and service has its own ADR. You can find them in the relevant sections.

License

Warning

This project includes dependencies licensed under the GNU Lesser General Public License (LGPL). Users must comply with LGPL terms when using or modifying these dependencies. For detailed information on each LGPL library used in this project, please refer to the respective license documentation included with each library. For comprehensive license compliance information, including dependencies and their licenses, you can read more details in our FOSSA report.

shortlink's Issues

api: add support SOAP

Link:

• https://github.com/tiaguinho/gosoap
• https://levelup.gitconnected.com/golang-soap-based-services-ccc4b3e3ee2e
• https://fale.io/blog/2018/12/03/calling-a-soap-service-in-go/

store: add method `search`

Extend List method in store

As an example: GetList(query)

todo:

• Support in API
• Support in-store

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: .github/renovate.json5
Error type: Invalid JSON5 (parsing failed)
Message: JSON5.parse error: JSON5: invalid character '\"' at 12:3

add method: links

ToDo:

Store:

• Add method: get links for all DB's
• Add test

API:

• gRPC-gateway
• http
• GraphQL

Fix:

• Fix generate proto
• Fix test coverage
• Fix generate GraphQL
• API for DGraph

try tarantool

• Руководство по использованию Tarantool Cartridge в Kubernetes

Add coverage

Add Notify interface

• Add interface

type Notify interface {
  Run() error
  Send(string) error
}

Add implementation:

• All API
• All Store

Next Step:

• API subscribe to store (need to get a response of write to store)

Link

• https://refactoring.guru/ru/design-patterns/observer
• [Observer/PubSub] https://medium.com/german-gorelkin/go-observer-bc95f465961b
• https://chatbotslife.com/making-a-very-simple-telegram-bot-with-golang-f7cb6601a320
• Event dispathcer

try rocksdb

ci: add circleCI

https://github.com/skanehira/pst/blob/master/.circleci/config.yml

api: add cloudevents

ToDo

• Init

Link

• https://github.com/cloudevents/sdk-go

Add API: grpc, grpc-web

To-Do

• grpc-gateway: redirect
• grpc-gateway: delete
• format response <-> http-chi
• generate swagger-docs

Link

• https://github.com/grpc-ecosystem/grpc-gateway/blob/master/third_party/googleapis/google/api/annotations.proto
• https://blog.maddevs.io/go-rest-%D0%B8%D0%BB%D0%B8-grpc-f5d52d7ffff6
• https://habr.com/ru/post/337716/

add goose for migration

ToDo

• Migration PostgreSQL
• Migration MySQL
• Migration SQLite

Link

• https://github.com/pressly/goose

Add Logger interface

ToDo

• Interface
• Zap implementation
• Logrus implementation
• Context for close method
• Test

Link

• https://forum.golangbridge.org/t/whats-so-bad-about-the-stdlibs-log-package/1435/16
• https://www.mountedthoughts.com/golang-logger-interface/
• https://blog.gopheracademy.com/advent-2016/context-logging/
• https://github.com/sandipb/zap-examples/tree/master/src/customencoder
• https://dev-journal.in/2019/05/27/adding-uber-go-zap-logger-to-golang-project/
• https://dave.cheney.net/tag/logging
• http://notes.burke.libbey.me/context-and-logging/

Add opentracing

• Init
• Create a package
• services
  • logger (MQ)
  • api
  • metadata
  • bot (MQ)
• Add to store
• Add to log package

Link

• https://medium.com/@aablinov/%D1%82%D1%83%D1%88%D0%B8%D0%BC-%D0%BF%D0%BE%D0%B6%D0%B0%D1%80-%D1%82%D1%80%D0%B0%D1%81%D1%81%D0%B8%D1%80%D0%BE%D0%B2%D0%BA%D0%B0-%D1%81-opentracing-%D0%B8-jaeger-69f0ae553b86
• https://github.com/batazor/go-logger/blob/master/pkg/jaeger/jaeger.go
• https://github.com/yurishkuro/opentracing-tutorial/tree/master/go
• https://www.jaegertracing.io/docs/1.8/client-libraries/

Add HELM chart

To-Do

• Minimal Helm Chart

И только что вышел Helm 3 :tada::tada::tada:

Анонс: http://amp.gs/slud
Что изменилось по сравнению с 2 версией http://amp.gs/slui
Как мигрировать http://amp.gs/sluw
Плагин для миграции http://amp.gs/sluF
Результаты security аудита http://amp.gs/slur

monitoring: Prometheus

PromQL

Links:

• https://grafana.com/blog/2020/02/04/introduction-to-promql-the-prometheus-query-language
• https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085
• https://promcon.io/2019-munich/talks/promql-for-mere-mortals
• https://timber.io/blog/promql-for-humans
• Achieving multi-tenancy in monitoring with Prometheus & the mighty Thanos Receiver
• How we eliminated service outages from ‘certificate expired’ by setting up alerts with Grafana and Prometheus

Docs:

• https://exporterhub.io/

Add github CI

Stage:

• test
• build

try use `error.Is` and `error.As`

kubernetes: linter

DigitalOcean выпустили clusterlint — утилиту для проверки ваших Kubernetes кластеров на следование best practices.

• https://github.com/digitalocean/clusterlint

try traefik v2.0

Link

• https://docs.traefik.io/

To-Do

• Install
• Tracing
• Metrics
• Logs

TLS

• https://blog.containo.us/traefik-2-tls-101-23b4fbee81f1
• https://dev.to/nflamel/how-to-have-https-on-development-with-docker-traefik-v2-and-mkcert-2jh3

devops: gitlab ci-pipeline

ToDo

• Linter && Test (Parallers)
• Build
• GitLab registry
• Check LICENSE
• Notify to slack
• Use GitLab CI Template (https://github.com/batazor/gitlab-ci-jobs)
• Cache dependencies
• GitLab Release
• Badger

Forkflow

• Automatically stop environments after a specified period
• review apps
• HotFix (Need rapid deploy to master)
• Research GitLab auto-deploy
• Cannary deploy
• DAST

artifacts

• Automatically bring in artifacts from cross-project jobs

intergration

• sentry
• Knative

k8s

• intergration
• logs
• metrics

Environment

• Yandex
  • Improt/Export alert/dashboard?
• AWS
• Google
• Minikube??? (From CI)

Link:

• https://medium.com/@TianchenW/run-gitlab-pipeline-dry-f9fd65c28a23
• https://about.gitlab.com/blog/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/

Use gogole-like hash

https://hashids.org/go/

Refs

• Creating a Readable URL Shortener
• Who's Using Hashids

add github template for Issue, Milestoune

try ignite

Link

• https://ignite.apache.org/
• https://github.com/amsokol/ignite-go-client

Add API: graphQL

Todo

• work version

Add config

// Schema represents a GraphQL schema with an optional resolver.
type Schema struct {
	schema *schema.Schema
	res    *resolvable.Schema

	maxDepth              int
	maxParallelism        int
	tracer                trace.Tracer
	validationTracer      trace.ValidationTracer
	logger                log.Logger
	useStringDescriptions bool
	disableIntrospection  bool
}

After

• Generate resolver
• Generate methods?
• Learn spec GraphQL:
  • Add filter
  • Add fragment
  • Union field

improve makefile

Link

• https://makefile.site/
• https://gist.github.com/subfuzion/0bd969d08fe0d8b5cc4b23c795854a13
• https://github.com/uber-go/tools/blob/master/Makefile

Add dockerfile

use `zap` for logging

try for optimization parse JSON

https://github.com/json-iterator/go
https://github.com/mailru/easyjson

monitoring: try loki

https://grafana.com/oss/loki/

Release v1.6.0 - https://github.com/grafana/loki/releases/tag/v1.6.0

• https://medium.com/@yannig.perre_32769/loki-centralization-of-logs-using-the-prometheus-way-3e2d3c7f6227

TODO

• monitoring docker-container (docker-compose)
• dashboard for application
• parse json
  • Nginx as example
• Prometheus metrics
• OpenTracing

Add ansible role for deploy to vagrant

ToDo

• vagrant
• Install Docker/Docker-compose
• Run this service
• Test (use molecule)

Links:

• https://habr.com/ru/post/437216/
• https://robertdebock.nl/
• https://gitlab.com/nayls-project/devops-pets/raspberry-ci
• How to Use Ansible to Install and Set Up Apache on Ubuntu 18.04

add dgraph

add consul kv

Todo

• integration
  • viper #731
  • traefik
• consulCatalog
• store
  • metadata
• API
  • add/del service
  • consul template

try tidb

add Neo4j

Add store - Neo4j

• Add implementation for the interface DB
  https://github.com/batazor/shortlink/blob/84fa2a365f09396f92556febde2e711c1682a780/internal/pkg/db/type.go#L9
• Add option of switcher https://github.com/batazor/shortlink/blob/84fa2a365f09396f92556febde2e711c1682a780/internal/pkg/db/db.go#L27
• Add implementation for the interface of Repository (optional)https://github.com/batazor/shortlink/blob/84fa2a365f09396f92556febde2e711c1682a780/internal/services/link/infrastructure/store/crud/type.go#L13
• Add tests (simple example)

Add sqlite

add leveldb

monitoring: kibana

TO-DO

Step 1

• Init
• View metric

Step 2

• View log https://grafana.com/blog/2019/10/16/new-in-grafana-6.4-the-logs-panel/

store: add field - `create_at`

ToDo

• Add field to link
• Update stores
• Update API

structure code: entities

Link

• https://medium.com/@hatajoe/clean-architecture-in-go-4030f11ec1b1

server: echo

ToDo

• https://echo.labstack.com/

link: add parse link

To-Do:

• Parse link
• Save description
• Save image

Link

• https://habr.com/ru/post/269055/
• https://github.com/badoux/goscraper

Example

version API/Store

ToDo

Add version package:

• API
• Store

Package

• https://github.com/hashicorp/go-version

Add graph service layer

securing gRPC connections with Go and TLS

Links

• https://itnext.io/practical-guide-to-securing-grpc-connections-with-go-and-tls-part-1-f63058e9d6d1
• https://blog.gopheracademy.com/advent-2019/go-grps-and-tls/

grpc: improve

ToDo

• add a message -> start service
• add logging
• add opentracing
• custom message about of error https://github.com/graph-gophers/graphql-go#custom-errors
• ui https://github.com/gusaul/grpcox

Handle Timeouts in Golang

Links

• https://medium.com/swlh/the-simplest-way-to-handle-timeouts-in-golang-11e371dc6188
• https://4gophers.ru/articles/rukovodstvo-po-nethttp-taimautam-v-go
• https://ieftimov.com/post/make-resilient-golang-net-http-servers-using-timeouts-deadlines-context-cancellation/

Add graph infrastructure

vault [tutorial]

• As DB ;-)

Intargation with GitLab

• https://docs.gitlab.com/ee/ci/secrets/
• https://docs.gitlab.com/ee/ci/examples/authenticating-with-hashicorp-vault/

Integration with k8s

• Effective Secrets with Vault and Kubernetes

tutorial

• Безопасное использование секретов: шаблон для использования HashiCorp Vault

Try TiKV

ToDo

• Connect
• CRUD

Link

https://tikv.org/