akvo / akvo-flow-maps Goto Github PK

We are using https://github.com/MastodonC/franzy which doesn't compile with the latest Kafka client version

Maybe some indices are required

As per akvo/akvo-lumen#1098

For walking-skeleton, behind a feature flag

Even when the DB or Kafka go bad

If the process crashes while creating a new tenant db, that db will be never properly setup.

On starting up the consumer process, it should check the status of the tenant db and finish creating anyone that is half done.

Make sure that the consumer is consuming and that it doesn't lag behind

Just hardcode the credentials for the dev environment.

• Create a new DB when the first message for a tenant is received
• Create a new user/pass and store it in the master db
• Auth proxy to read the credentials from the master db

For walking-skeleton, behind a feature flag

See akvo/akvo-maps#42

Including a good enough Kafka cluster

Only Flow should be able to create new maps.

Secure the create map endpoint by doing some Authz on the Windshaft proxy, maybe by using Keycloak service accounts.

Windshaft tiles endpoint should still be available to external clients. Probably we will need some Nginx proxy to disallow any POST request to Windshaft

Use property files, env vars and/or Kubernetes secrets

Maybe in a new project or repository.

We are using https://github.com/konukhov/kfk-avro-bridge to transform avro messages to Clojure maps.

Either work with avro Generic Records directly or make a proper release of the kfk-avro-bridge.

Probably, some small changes will be required on the kfk-avro-bridge as it is mangling the field names with camel-case, snake-case and kebab-case.

I think the default log retention for the Kafka consumer topic offset topic is one day, which means that if our consumer goes down for more than 24 hours, it will start from the beginning.

Change the retention to one week, so we have more time to detect and fix any possible issue

Right now any exception will crash the consumer thread.

Either make the process more resilient to excepted exceptions (like DB timeouts) or make sure that the process crashes and except K8s to restart the process.