planvx / aweme Goto Github PK

View Code? Open in Web Editor NEW

2.0 0.0 1.0 267 KB

a lite short-form video backend server

License: Apache License 2.0

Go 98.79% Dockerfile 0.72% Shell 0.50%

aweme's Introduction

aweme

一个轻量级短视频应用程序后端

项目结构

.
├── .github
├── cmd
│   └── aweme/main.go
├── configs
│   └── config.yml
├── docker-compose.yml
├── docs
├── internal
├── readme.md
├── renovate.json
└── scripts
    ├── codegen.sh
    ├── schema.sql
    ├── .env
    └── Dockerfile

.github/workflows: 该目录包含了 GitHub Actions 的配置文件。
cmd: 该目录包含了项目的入口包，即 main() 函数所在。
configs: 该目录包含了项目的配置文件，例如：数据库配置、日志配置等。
docker-compose.yml: 该文件用于在 Docker 环境下运行项目时所需的配置。
docs: 该目录包含项目的 API 文档, 可以使用 codegen.sh 脚本生成。
internal: 该目录包含项目的内部代码逻辑，例如：API 定义、数据访问层、逻辑层等。
scripts: 该目录包含项目的脚本文件，例如：代码生成脚本、数据库脚本等。
scripts/.env 文件包含了作为环境变量的配置信息，请按需填写，例如：数据库连接信息、密钥等。

kubernetes 部署

以下操作在默认的 namespace 下进行，如果需要部署到其他 namespace，请在命令中指定。

创建配置

kubectl create cm aweme-config --from-file=aweme-app-config.yaml=configs/config.yml --from-file=aweme-app-schema.sql=scripts/schema.sql

创建 secret，其中包括了数据库连接信息、密钥等
```
kubectl -f deploy/secret.yaml
```

部署 MySQL 和 redis 及相关 headless service

kubectl -f deploy/mysql.yaml
kubectl -f deploy/redis.yaml

部署应用及相关 service
```
kubectl -f deploy/aweme.yaml
```
（可选）配置 HPA 水平自动伸缩
```
kubectl -f deploy/aweme-hpa.yaml
```

运行

项目运行可参考以下步骤：

在项目目录下，运行 scripts/codegen.sh 脚本来生成项目所需的代码，包括测试代码和 API 文档，（可选）在运行该脚本时，指定 MOCKERY 环境变量，以告诉脚本使用 Mockery 工具生成相应的 Mock 代码用于逻辑层测试。
根据实际情况修改 scripts 下的 .env 文件，并在其中配置环境变量，例如：数据库连接信息、密钥等。
运行 docker-compose up 命令来启动项目。该命令会根据 docker-compose.yml 文件中的配置来构建和启动项目的容器。

aweme's People

Contributors

Stargazers

Forkers

sixwaaaay

aweme's Issues

bug: query fields and skip default transaction

not use * to avoid table metadata query
not use default transaction
replace First with Take

Graceful shutdown logic: Write code to enable the application to gracefully shut down services and release resources when a shutdown signal is received.

http server
redis connection
mysql connection
object storage connection

offical native opentelemetry support needed

api span
data query span
redis query span
tracing backend

uptrace distro is coupled with uptrace

bug: error with partial write success when multiple writes fail midway

Now, we are ignoring the Redis write errors that occur after successfully writing to MySQL. However, there is another problematic write error that occurs when multiple consecutive writes fail midway, with some succeeding and others failing.

JWT Middleware Configuration: Write middleware code to integrate JWT verification into the HTTP request processing flow.

defines the config for JWT middleware.

general handler logic for wrap functions

A universal Handler logic, and provides generic interfaces to integrate the function of the business logic layer into the process

v0.1 requirements

types for api

Inconsistent behavior in Health Check between nerdctl compose and docker-compose

The bug causes Health Check to behave differently in nerdctl and Compose compared to Docker Compose. As a result, the user experience is inconsistent and may lead to unexpected behavior. Therefore, we need to modify the config of docker-compose.yml

Fix the circular imports caused by the position of the type definition: Identify circular reference problems in the code, adjust the position of the type definition to resolve the issue.

types in api should be imported in logic, and the route will import as well, so circular imports occurs

sql query scripts to generate video queries

filter videos with user id and limit count
filter videos with a milliseconds timestamp

User registration logic: Implement user registration functionality, save user information encoded with bcrypt, and generate a JWT token.

password should be encoded with bcrypt
sign a jwt

bug: db connection management

db connection close isn't controled

wrapper for usermodel and video model

wrap generated queries to UserModel, VideoModel

Use the Uber fx framework to build the application framework: Use the fx framework to build the application framework to facilitate the management of application components.

lifecycle management
application entry

Add JWT signing logic: Write code to implement JWT signing functionality, ensuring data security.

login / register
whitelist default

bug: echo#pre middleware add trailing slash caused ERR_TOO_MANY_REDIRECTS

actually only requests to the api should be added trailing slash.

bugs:

/swaggger/index.html

publish list api parameters missing

user_id parameter is a query string

bug: swagger docs should be excluded when build

add build tags to exclude docs when release

refactor to event driven architecture

dapr powerd by kafka

Data access layer interface definition: Define the data access layer interface, including operations such as create, read, update, and delete.

interface definetion for logic test with mock tools

Definition of various module layers: Define the various hierarchical structures of the application, including the handler layer, logic layer, and data access layer.

layer to build the application

Application configuration definition: Define the configuration parameters required by the application, including database connection information, log levels, etc.

jwt expireTime,secret
redis hosts, password
mysql
object stroage

"like" feature logic

like action
query listed video list

id generator is needed to add

ref: https://github.com/sony/sonyflake
its generated uint64 id can be covert to int64 safely (the 1st bit is unused now)

deploying an application with Kubernetes

ConfigMap to store application configuration
Secret to store sensitive information
StatefulSet MySQL and Redis
Deployment for applications

definition of hanlders and routers

orm code generation

ref: [Friendly & Safer GORM powered by Code Generation.](https://gorm.io/gen/)

Routing register logic: Group the routes to facilitate management and maintenance.

public apis
half-public apis
private apis

publish list query

tests
videomodel interface changes
api definition changes

bug: missing method to query by user id in video model interface

FindByUserID(ctx context.Context, userID int64, limit, offset int) ([]*Video, error)

Data access layer model definition: Define the data access layer model, including the structure and fields of database tables.

macth the sql script definetion

refactor: migrate to raw gorm

remove gorm/gen
refactor the code of data access layer

redis connection doesn't release

redis connection doesn't release when shutdown applicaiton

User login logic: Implement user login functionality, use bcrypt verify username and password, and generate a JWT token.

bcrypt
token generation

api handler definition and swagger document

refactor data access layer interface into command and query classes

we need to split the data access layer interface into smaller methods that perform specific tasks. This will make the code more readable, maintainable and testable.

The data access layer interface should have smaller and more focused methods
The logic layer code should work as expected with the new methods

changes needed in middlewares

publish video logic

upload to s3 storage
save data to database

add basic ci

automating the integration of code changes

continuous go test
continuous go build

add Prometheus metrics handler

ref https://github.com/labstack/echo-contrib/prometheus

#75

S3 storage integrate

connection config
upload endpoint for publish video

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

docker-compose

docker-compose.yml

dockerfile

scripts/Dockerfile

github-actions

.github/workflows/ci.yaml

actions/checkout v3

actions/setup-go v4

codecov/codecov-action v3

actions/checkout v3

actions/setup-go v4

docker/login-action v2

docker/build-push-action v4

mysql 8.0

.github/workflows/release.yaml

actions/checkout v3

actions/setup-go v4

goreleaser/goreleaser-action v4

actions/checkout v3

actions/setup-go v4

docker/login-action v2

docker/build-push-action v4

gomod

go.mod

go 1.19

github.com/DATA-DOG/go-sqlmock v1.5.0

github.com/alicebob/miniredis/v2 v2.30.1

github.com/brpaz/echozap v1.1.3

github.com/go-playground/validator/v10 v10.11.2

github.com/golang-jwt/jwt/v4 v4.5.0

github.com/labstack/echo-jwt/v4 v4.1.0

github.com/labstack/echo/v4 v4.10.2

github.com/minio/minio-go/v7 v7.0.50

github.com/redis/go-redis/extra/redisotel/v9 v9.0.2

github.com/redis/go-redis/v9 v9.0.2

github.com/sony/sonyflake v1.1.0

github.com/stretchr/testify v1.8.2

github.com/swaggo/echo-swagger v1.3.5

github.com/swaggo/swag v1.8.10

github.com/uptrace/opentelemetry-go-extra/otelgorm v0.1.21

go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho v0.40.0

go.opentelemetry.io/contrib/instrumentation/runtime v0.39.0

go.opentelemetry.io/otel v1.14.0

go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v0.36.0

go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.13.0

go.opentelemetry.io/otel/metric v0.36.0

go.opentelemetry.io/otel/sdk v1.14.0

go.opentelemetry.io/otel/sdk/metric v0.36.0

go.uber.org/fx v1.19.2

go.uber.org/zap v1.24.0

golang.org/x/crypto v0.6.0

gopkg.in/yaml.v3 v3.0.1

gorm.io/driver/mysql v1.4.7

gorm.io/gorm v1.24.5

gorm.io/plugin/dbresolver v1.4.1

moul.io/zapgorm2 v1.3.0

Check this box to trigger a request for Renovate to run again on this repository

hpa for application

autoscaler/v2

cpu
memory

JWT Whitelist: Configure a whitelist so that some API requests do not require JWT verification, improving performance.

initial whitelist
configurable whitelist
http 401 handler

release workflow is needed

trigger:
git tag x.y.z

tool:
goreleaser
target platforms:

windows
linux
target arch:
amd64

windows with zip format
linux with tar.gz format