airtai / faststream Goto Github PK

• Remove FastAPI dependencies from FastKafka class
• Add option for generating html
• Rework _on_startup
• Rework _on_shutdown
• Add context manager dunders
• Implement FastKafka run CLI
• Implement CLI function for serving docs
• Cleanup Uvicorn helpers and unnecessary functions
• Rework tests to use FastKafka run command and remove passing FastAPI instance

there is a CLI command for exporting documentation that requires npx and stuff (that can also be installed using CLI command)

• create workflow action for creating and deploying FastKafka docs to github (for random projects using FastKafka)

• Add exit_code return value to run_script_and_cancel helper function
• Assert exit_code is equal to 0 when run_script_and_cancel runs
• Add tests for run_script_and_cancel helper function
• Check log levels for Guide_02_First_steps

TaskGroup must be closed in the same task it was opened, otherwise exception is raised
This breaks FastKafka when async context manager methods are used separately

• Replace task group with asyncio.create_task calls to enable using aenter and aexit methods in FastKafk

• Remove passing of custom producers in @produces decorator
• Add cleanup of stale producers in populate_producers and close_producers methods

Implement the tester app that produces the messages and consumes, same as mockup and can be used for testing the user applications

The issue happens when the code does not flush the output, such as in regular python print().

• Write a warning in the 02_First_Steps for flushing

• Implement partition key mechanism for @produces decorator
  Implemented behaviour:
  1. A method decorated with @produces can return defined message as-is: the message is wrapped in a Event object with key=None and passed to producer = message sent without defined key, partition chosen at random
  2. A method decorated with @produces can return defined message wrapped in an Event object with key argument value in bytes = message sent to kafka with defined key, partition chosen using the defined key

• Implement benchmark for @produces decorator
• Implement benchmark for @consumes decorator
• Implement benchmark for @process decorator
• Document the benchmarks

• requirement(batch): batch support is a real need! and i see it on the issue list.... so hope we do not need to wait too long

https://discord.com/channels/1085457301214855171/1090956337938182266/1098592795557630063

If the Zookeeper or Kafka process crashes while creating it's most likely that the requested port is occupied.
To solve this, if the process crashes, try with a randomly generated port.

• Zookeeper random port allocation
• Kafka random port allocation

• Remove unnecessary process_msgs and integrate it back to _aiokafka_consumer_loop
• Rework process_msgs tests to use mock patching
• Remove creating and sending async callback to process stream every time when msg is consumed
• Write docs
• Fix failing CI tests

https://bytewax.io

• Prepare project repository and structure
• Mockup services for ESA
• Test mockups

• Implent mocking of consume and produce functions
• Implement mirroring of consumers and producers

It is working locally, but failing in CI. Please see:

https://github.com/airtai/fast-kafka-api/actions/runs/3777280244/jobs/6421027319

Most likely you need to define some variables such as KAFKA_HOSTNAME

This is necessary to enable the re-entrance of the Tester and FastKafka app

see https://github.com/airtai/fast-kafka-api/blob/main/docker/dependencies.yml

Changed to pass to LocalKafkaBroker

• add topics function to KafkaApp for listing topics
• Implement creation of topics in LocalKafkaBroker
• Wait for topics to be created in Broker

• Remove stale uvicorn imports from notebooks
• Transfer nest-asyncio requirement to dev requirements
• Replace Ilock with posix_ipc
• Implement mechanism for port allocation in LocalKafkaBroker
• Cleanup logging in LocalKafkaBoker

• replace docker compose with a simple docker run (standard run_jupyter.sh should do)
• replace all tests to use LocalKafkaBroker
• update documentation