klippx / phobosjs-mock

phobosjs-mock

Some simple ideas in code how we could be using phobos in practice.

The core ideas are

• No multiprocessing, keep it simple.
  • No need to farm out resources on each node, assume micro instances
  • This alone should greatly reduce complexity of the Phobos code and make it easier to understand/contribute
  • Instead, scale horizontally
  • One service instance <=> one consumer <=> one configuration
• Avoid meta programming and implicit contracts between users' services and phobos
  • The user is in control
  • The code written in services should be simple and readable, even if you don't know kafka (or even phobos!) you should still be able to understand
• Move as much config as possible to .phobosjs
• Require as little config as possible inside the users' services, if any.
• Consider possibility of support such as
  • Providing test tools for writing the consumer applications
  • Possibility to decode avro encoded kafka messages
  • Possibility of wrapping the message consumption and dealing with results (errors/success)
  • phobos-checkpoint-ui
  • phobos-sensor
    • => phobos-prometheus (provide collectors and exporters for internal consumer/producer metrics)
    • => phobos-grafana (provide some rudimentary graphing of the data exported above)

phobosjs-mock's Issues

Some ideas

phobosjs-mock/standaloneConsumer.js

Line 22 in 9e55fad

app.on('messageReceived', (message) => {

It should use async functions for the messageReceived so it allows users to fetch data from external APIs or query databases, etc

app.on('messageReceived', async (message) => {
  const decodedMessage = avro.fromBuffer(message)
  const statement = createStatement(decodedMessage.data())
})

Maybe we should drop the event syntax altogether and go directly to functions, like:

app.onMessage(async (message, metadata) => {
  // metadata => { retryCount, retryTime, etc }
  const decodedMessage = avro.fromBuffer(message)
  const statement = createStatement(decodedMessage.data())
})

Maybe some hooks for errors so the app developer can track/measure the errors

app.onError(async error => {})

phobosjs-mock/standaloneConsumer-beforeConsume.js

Line 30 in 9e55fad

beforeConsume: (payload) => {

I would use keySerializer and valueSerializer instead of before* and have them as async functions.

const app = createConsumer({
  topic: 'statement_available',
  keySerializer: async key => key.toString(),
  valueSerializer: async value => await avro.fromBuffer(value)
})

Last comments. This library could use a .phobojs javascript file instead of JSON. The js file can have comments and support interpolation, etc.