zowe / event-management-wg Goto Github PK
View Code? Open in Web Editor NEWResearch and experimentation by the event management workgroup
Research and experimentation by the event management workgroup
Run a series of tests for data format compatibility, given the following:
Live service operations:
Wire fmt:
Document the producer and consumer experiences working with data at different schema levels, encountered error conditions, workarounds, etc.
Make recommendations on our use of AVRO and how strongly we should recommend it as part of best practices / conformance criteria.
See #6 for implementation details. Create the same content, but using Node.JS instead of Java.
Create a producer which can track Job events - Submitted, Completed, Phase Changed, etc.
Consider how the z/OS event capture could be re-used as part of a later library creation, #4 .
Question:
Granularly:
Follow-ups:
Create a Kafka producer capable of creating events on a configurable set of topics and time slices.
This producer may be re-used in future performance tests to populate Kafka with data.
POC a simple producer library which contains APIs or interfaces for capturing z/OS events and publishing them to a Kafka bus. Refactor the existing Jobs producer to use this library.
Consider extensibility - if this library can't capture every type of event, is there a way we can subscribe for events or allow extenders to write their own z/OS event capture interface?
Using mock events of different size and formats (500 bytes, 1KB, 5KB, 10KB, 100KB), use Avro and Cloud Events plugins to write events to Kafka and consume them.
Use topics like the following:
WIRE.FMT.PLAIN
WIRE.FMT.CE
WIRE.FMT.AVRO
WIRE.FMT.AVRO_CE
Avro schemas should be placed in:
WIRE.FMT.AVRO.SCHEMA
WIRE.FMT.AVRO_CE.SCHEMA
If possible, measure the performance difference between the different formats, both in terms of throughput and disk usage. Document the user experience when choosing the different formats.
If possible, integrate the Avro and CloudEvent functionality into the producer and consumer libraries, update those issues, or open a new issue to do so.
See #20 . In addition to that SASL callback, we could provide a SAF verifier. (Would only work in Kafka running on Z).
Is it possible to have APIML frontend connections to the Kafka Broker?
If so, what are the impacts on:
Work through an installation of Kafka on Z, without use of a z/OS package manager.
There will be struggles with the Kafka scripts, which use features in bash
.
sh
? Start/stop/etc.bash
could help, but we don't want pre-req on software which requires licensing.Create a simple Kafka client in Java which can consume configurable set of topics and log/publish some basic consumption statistics ( msg count per topic, global, etc.).
Evaluate the behavior of the system under different data exhaustion scenarios. Discuss the results with the WG.
Broker up scenarios:
Broker down scenarios:
Build a simple webhook client which consumes messages from Kafka topics, and allows users to configure webhook endpoints to forward the content to. Simple user interface or no user interface at all - just a configuration file.
Test mainframe security integrations with Kafka. What configuration options do we have?
This should include:
We should assume:
Follows #15 .
The same POC should be expanded to consider authorization limitations. The solution should ensure that the user submitting the job is able to retrieve their job information without seeing other jobs running on the system. i.e., the user should not gain unauthorized visibility into the system.
Suggestion: use dynamic topic to send user end-of-job notification. Here's an off-the-cuff flow, the agent managing the authN/authZ check and copying the message to the new topic is undefined.
User1 - > ZOWE.SYS1.JOB.NOTIFICATION
{ jobId: '...', jobNumber: '...' }
-> Kafka authN/authZ check -> responds { topic: 'ZOWE.SYS1.USER1.JOB.EVENTS' }
.
Kafka -> ZOWE.SYS1.USER1.JOB.EVENTS
-> consume from ZOWE.SYS1.JOBS
filter where (jobId=,jobNr=), push to new topic.
User -> retrieves event from ZOWE.SYS1.USER.JOB.EVENTS
.
The SASL authentication option in Kafka has a PLAIN operating mode where a username and password is passed to the server-side component and routed to a user-defined callback so long as user's JAR is on the classpath (sasl.server.callback.handler.class
).
We could use this feature do deliver an authentication workflow that leverages APIML's /auth/
endpoint.
Create a simple client with the same specs as #7 , but using Node.JS.
From WG Notes:
How are we monitoring Kafka?
Document what metrics and integrations are available from Kafka. JMX, OpenTelemetry, etc. How would customers aggregate, visualize, exhaust these metrics?
Install a Kafka instance on our z/Linux + K8S infrastructure.
The Kubernetes infrastructure may be out of date and require an upgrade. Networking challenges unknown...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.