uniwise / parrot

POEditor pull through cache

Parrot the POEditor pull-through cache

---

This cache makes it possible to pull app translations directly from POEditor instead of compiling them into your app. This means no more re-builds every time translations are updated!

Features

• Battle tested: The software is in active use on the WISEflow platform with high request rates daily.
• All the formats: Parrot can provide all formats supported by POEditor.
• Cache choices: Parrot comes with a Filesystem and Redis cache to facilitate single app deployments and highly distributed deployments.
• OpenAPI: The Parrot API has been documented in OpenAPI specification which can be found in the doc/ directory.
• Easy deployment: A docker image and helm chart is provided.

Configuration

The server can be configured with a yaml configuration file specified with the --config option. Every configuration can be overridden with an environment variable, where dots are replaced by underscores, such that API_TOKEN=xxx will set the api.token value

key	description	type	default
server.port	port for the main http server	int	80
server.gracePeriod	grace period for the http server to shutdown	duration	10s
log.level	log level	string	info
log.format	format of the log. Can be "text" or "json"	string	json
cache.type	type of cache to use for translations	string	filesystem
cache.ttl	time to live for cache items	duration	1h
cache.renewalThreshold	threshold at which the server will preemptively fetch a new translation	duration	30m
cache.filesystem.dir	directory of the filesystem cache	string	default user cache directory
cache.redis.mode	mode of the redis connection to back the redis cache. "single" or "sentinel"	string	single
cache.redis.address	address of the redis server, in case the single mode is used	string
cache.redis.username	username to authenticate against redis	string
cache.redis.password	password for redis authentication	string
cache.redis.maxRetries	max retries for redis client to connect to redis. Set to -1 for infinity	int	-1
cache.redis.db	redis db index	int	1
cache.redis.sentinel.master	master name for sentinel setup	string
cache.redis.sentinel.addresses	list of sentinel addresses	[]string
cache.redis.sentinel.password	password for authenticating against sentinel instances	string
prometheus.enabled	enable prometheus metrics	boolean	true
prometheus.path	expose prometheus metrics under path	string	/metrics
prometheus.port	port to expose the prometheus metrics under	int	9090
api.token	secret token to authenticating against poeditor	string

API specification

The REST API of Parrot is documented in the OpenAPI format. The specification file can be found here docs/api.yml and a Swagger UI is available here uniwise.github.io/parrot.

License

Parrot is available under the Apache 2 license.

This project uses open source components which have additional licensing terms.

parrot's Issues

Support multiple formats

Support multiple formats in the REST endpoint:

/v1/project/{project}/language/{language}/{format}

Support custom request id header name

We've settled on using x-trace-id as the header name for request ids. Echo does not allow easy configuration of this, so we probably have to write our own middleware

Improve healthcheck

Looks like the service might be throwing 504s for the first couple of seconds of running with a redis cache. Observed one, when rolling the production deployment. The log also hints that the cache is first ready after ~5 seconds of starting the service:

{"level":"info","msg":"Server listening at :80","time":"2021-03-25T09:58:36Z"}
{"level":"info","msg":"Prometheus metrics exposed at :9090/metrics","time":"2021-03-25T09:58:36Z"}
{"level":"info","msg":"sentinel: discovered new sentinel=\"747233771ef03c2613b4ffe2b00157407ca2063d\" for master=\"parrot\"","subsystem":"cache","time":"2021-03-25T09:58:42Z"}
{"level":"info","msg":"sentinel: discovered new sentinel=\"822a5febf7ca75fd42401b3bd12d80512e2701b1\" for master=\"parrot\"","subsystem":"cache","time":"2021-03-25T09:58:42Z"}
{"level":"info","msg":"sentinel: discovered new sentinel=\"b1e9b8d977ca27938692e926b2cce5640e87991e\" for master=\"parrot\"","subsystem":"cache","time":"2021-03-25T09:58:42Z"}
{"level":"info","msg":"sentinel: new master=\"parrot\" addr=\"10.100.77.76:6379\"","subsystem":"cache","time":"2021-03-25T09:58:42Z"}

I propose we include a service healthcheck in the /health endpoint to make sure the cache is ready for reads and writes before routing traffic to it

Support ETag in requests

Add ETag to rest requests to allow for better browser caching.

The ETag could be the MD5 sum of the file.

Cache purging should be possible for maintenance

We should implement a cache purge endpoint. Something like:

POST /project/:project/language/:language/purge

Add noCache header

Assessment frontend build pipeline fetches the terms each time. If we hit a bad cache time (i.e with new changes in POEditor), we would be safe if we could skip cache.

Redis cache hset and hget

The redis cache should use hset and hget to easier clear translations by project id.

List available languages for project