AsyncAPI Applications Relations Finder about cupid

[derberg]

@arjungarg07 they all feel kinda the same tbh, I mean the same principle behind as in case or mermaid and plantUML. As you wrote, I think more important are tests now, and a way to test all those outputs not only from unit tests point of view but also integration tests.

And then something more beautiful........ React Flow ❤️

[magicmatatjahu]

@arjungarg07 Yes, elements should be the exact output from function.

[derberg]

@arjungarg07 Alvaro's blog post is out https://www.asyncapi.com/blog/building-async-flight-notification-service and the project where he has AsyncAPI files is here https://github.com/amadeus4dev/amadeus-async-flight-status

[derberg]

I'm not sure what you mean here? Did you mean that someone can use this library as a plugin I guess, right?

I think it would be cool to make it possible to provide plugins for this library. So by default, it outputs some custom format, but one can also install an additional plugin and use it with the library to get mermaid (as an example) format. Of course in first version we do not have to have pluggability, just a thought. We can also have a switch in first version where you can just pass option and say that instead of default you want mermaid.

awesome-d3

definitely, mermaid is on my head only because you get UI out of the box, but other formats would be great too

It would also be just another helper function, which can be used by developers to visualize the change in the architecture.
Like: add(, <subsrcibe/publish>, ), can be used to detect how a change in one microservice is going to impact others.

ok, so you import 4 files, get a visualization of architecture and then can add new element to the "visualization" to see how it will look like if some new service will publish to some existing channel, good, makes sense. Just keep in mind that if we make it possible to add something to the structure, we should be able to later export added element into AsyncAPI - just making it clear so you do not extend the scope of the task to much, so you have enough time to finish it. We can make add as a nice to have feature

I think all is clear and you can start drafting a proposal

[pcornelissen]

I'm also looking for some UI to discover relationships in a "mesh" of services. Great to see that you made something so quickly. I'm looking for quite some time for something to document event driven systems in a better way than linked confluence pages ;-)

In my mind there is a web UI+API where you can register asyncAPI files and then you can browse all these in a format similar to the docs generated in https://playground.asyncapi.io/?load=https://raw.githubusercontent.com/asyncapi/asyncapi/master/examples/2.0.0/simple.yml but with links between services that share events/messages.

I'm not very fond of JS, I'm more a Java guy, so my first instinct is to create a nice Spring boot app to do the heavy lifting and a small vue.js UI to display everything. But that takes time and effort and when something else is developed somewhere, I could use the time more productively ;)

[derberg]

@arjungarg07 repo ready and as you can see, the initial issue is also transfered

[pcornelissen]

@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that asyncapi/spec#531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.

I'd like to throw into the ring that structurewise it doesn't really matter is a service is running in 1, 2 or 200 instances, it shouldn't appear with a cardinality of 200 in the flow. The main point of the message approach is that you don't care on the sender side how many workers/consumers there are. It may be an important fact for the consumer to be for example single instanced, or it is stateful, but that has IMHO nothing to do with the message flow documentation.

[magicmatatjahu]

@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :)

[arjungarg07]

same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server

Yeah! but I would like to play it safe and not make any assumptions😅

@arjungarg07 Please don't focus on first weeks on output shape (also using shape of ReactFlow isn't a good idea, we should be agnostic here). We can change it later, because currently most important part is making, as you wrote, some MVP :)

Sure thing.

[arjungarg07]

but honestly it's a hard to understand in first look what is connecting to what

Yup! I mean the whole thing is to visualize the relations and the first look at this image feels somewhat complex :(
I would try to add the config theme in the script to visualize it much easily than now

What do you want to do next?

will start coding the script for generating mermaid diagrams syntax asap 🚀

[arjungarg07]

Do you mean metadata like payload's schema of message etc?

Yeah, message and schemas too.

About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to true.

Sure. I will then update the metrics 👍🏽 In the next weeks maybe we could imply these metrics as class models and extend the APis from there itself.

[magicmatatjahu]

Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that 🤔

Yes, you're right about that, but partially :) For mermaid you can only use payload, headers and bindings metadata from channel, for another syntax you can only use payload etc. Of course, It will be "huge" data for default syntax, but we can read/reuse/change it for another syntaxes. You should think about it as a bridge to another syntax.

[arjungarg07]

Yeah, so this is the simplest flow for ReactFlow.

import React from 'react';
import ReactFlow from 'react-flow-renderer';

const elements = [
  { id: '1', type: 'input', data: { label: 'Node 1' }, position: { x: 250, y: 5 } },
  // you can also pass a React Node as a label
  { id: '2', data: { label: <div>Node 2</div> }, position: { x: 100, y: 100 } },
  { id: 'e1-2', source: '1', target: '2', animated: true },
];

export default () => <ReactFlow elements={elements} />;

So our library function for react flow should just generate the elements array which could be directly passed by the user, right?

[magicmatatjahu]

@arjungarg07 Important thing: If you will use inside output the VDOM nodes, like in your example <div>Node 2</div> you should write them with React.createElement() function, not as JSX, because if you use the JSX, you need then transpile your package by babel.

[magicmatatjahu]

@arjungarg07 Can we close the issue?

[derberg]

we can definitely close it, purpose of this issue was to discuss initial idea and the scope.

[arjungarg07]

I am trying to get myself familiar with parser-js for now, since I believe that would be the main tool we would be using and building this idea on top of. Would try to resolve some good first issues and get familiar with the codebase.

[arjungarg07]

@derberg Also can you please point out to some already implemented async-api collection so that I can test this idea on those files manually for PoC.

[derberg]

@arjungarg07 we do not have anything out of the box. These either would have to be created from scratch or you can try out https://solace.com/try-it-now/ that gives an event portal where you can design a system in a kind of design tool. You first create objects, connect them to explain how they are related and then you can export AsyncAPI files from there. Decide on your own what is better, play with some tool that already supports AsyncAPI so you learn how tooling can support the spec, or you learn the spec by just creating few files for some fake system, and try to "connect" them (I can of course help).

you can also wait few days and there is Alvaro that works on a blog post on a real system based on AsyncAPI and he has AsyncAPI documents there, he just needs to do some fixes to that. I can keep you posted, especially that you will anyway be busy with the onboarding tasks from the parser

[arjungarg07]

Yeah, thanks for sharing the design tool. I would try to build AsyncAPI files as per the use cases described in the idea and play with it to draw some relations manually first.
Sure, I will do some onboarding tasks from the parser till Alvaro has those documents ready. Thanks.

[arjungarg07]

@derberg I build a prototype in which I tried to identify relations between the AsyncAPI docs on the Rideshare microservice (provided by solace) and model the output as shown in the image. This is just a basic structure and can be changed according to the requirements.
Link to the source code: https://github.com/arjungarg07/relation-finder-prototype

Rideshare Graph View

Basic output structure

Please give your feedback. Thanks😄

[arjungarg07]

@derberg I did some research about what more functionalities we can provide by the library:

• Filters to highlight relations that match a criteria given by the user.
• option to provide output format as grouped events which are common to two services
• showcasing all the events referencing the same Schema
• option to add/remove service subscribe/publish on channel which would help in visualizing the architecture if any of the puzzles is removed or added.

Please share your opinion on this and also throw some light on what more functionality we can provide by this library. Thanks😃

[derberg]

Loving it so far mate, I thought about this idea for very long but never had time to code it 😞 awesome to see what you're doing here and ideas that pop up in your head, really 🙏🏼

Do not worry about the structure much because as you noted, this can be modified once the actuall work on the library starts. And the structure will change for sure, will be more complex, with more data. I'm more interested at the moment what you think about making library extendable in a way that someone can provide a plugin for the library, so the output is not our custom Map but class diagram syntax from mermaid.

Lemme just list all comments 1by1

• Hint for future, when you will work on this topic and will need to traverse schemas, keep in mind we have great support for it with cupport for circular references
• I have doubts if these Rideshare files is the only sample project to use here. These files do not have servers, which at the end means that there is no validation if given channel from App A and the same channel from App B are connecting to the same server or not. So what I mean is that we can discover relation that is not necessarily there. Now, this doesn't mean you did something wrong, because servers are not mandatory in AsyncAPI, we should definitely support a flow without servers. Just the main flow we need to support is the one with servers matching across files (so I guess the example from Alvaro). So once servers are there, do matching, if not, don't
• Because of above, this is why the UI from Solace kinda shows like relations are "directly" connecting microservices through messages (I think these are messages) which is fine without server/broker, but in case we have server matching, it should show that App A has a link with App B through some message, but relation goes through a given server

Filters to highlight relations that match a criteria given by the user

I thing this would be a great thing. As I understand this would be besically a set of helpers that someone could use in frontend. So instead of writing custom code to traverse through found relations, frontend dev can just use a helper to get relations only of a given application, or only relations where given message is in play. That would be awesome, much easier then to do some "click to narrow view" magic 😄

option to add/remove service subscribe/publish on channel which would help in visualizing the architecture if any of the puzzles is removed or added.

you mean in the UI already? we are talking about library here. Of course it is good if some features are driven by how we see it in the UI, just need to know what you mean exactly and what would be the purpose of add for example? what I add could later be saved to a file or just returned as string?

option to provide output format as grouped events which are common to two services

I'm not sure what you mean here? oh wait, events are messages for you I guess, right? so you might mean here what I wrote above 😄 if yes, then yes, but it is not option on output but simply another function available imho

showcasing all the events referencing the same Schema

you mean not focusing on channels but on messages? find where they are used and show what schemas they have? Like it, it will for sure help to check if there are some inconsistencies maybe

[arjungarg07]

Loving it so far mate, I thought about this idea for very long but never had time to code it 😞 awesome to see what you're doing here and ideas that pop up in your head, really 🙏🏼

@derberg Thank you for the kind words, all thanks to you😄

I'm more interested at the moment what you think about making library extendable in a way that someone can provide a plugin for the library,

I'm not sure what you mean here? Did you mean that someone can use this library as a plugin I guess, right?

so the output is not our custom Map but class diagram syntax from mermaid.

Yeah, we can provide the output in a way that can match the syntax from mermaid, also we can search some more related tools which use the same syntax as mermaid from this collection: awesome-d3

• Just the main flow we need to support is the one with servers matching across files (so I guess the example from Alvaro). So once servers are there, do matching, if not, don't

Got it, I will look into it.

you mean in the UI already? we are talking about library here. Of course it is good if some features are driven by how we see it in the UI, just need to know what you mean exactly and what would be the purpose of add for example? what I add could later be saved to a file or just returned as string?

It would also be just another helper function, which can be used by developers to visualize the change in the architecture.
Like: add(<channel name>, <subsrcibe/publish>, <message name>), can be used to detect how a change in one microservice is going to impact others.

you mean not focusing on channels but on messages? find where they are used and show what schemas they have? Like it, it will for sure help to check if there are some inconsistencies maybe

Yeah!

@derberg Shall we solidify all these functionalities? also please tell me if we can provide any more features by this library as per your opinion so that I could jot them down in my official proposal with the workflow of how I will be implementing them😄

[arjungarg07]

I think it would be cool to make it possible to provide plugins for this library. So by default, it outputs some custom format, but one can also install an additional plugin and use it with the library to get mermaid (as an example) format. Of course in first version we do not have to have pluggability, just a thought. We can also have a switch in first version where you can just pass option and say that instead of default you want mermaid.

Got it👍🏽

ok, so you import 4 files, get a visualization of architecture and then can add new element to the "visualization" to see how it will look like if some new service will publish to some existing channel, good, makes sense. Just keep in mind that if we make it possible to add something to the structure, we should be able to later export added element into AsyncAPI - just making it clear so you do not extend the scope of the task to much, so you have enough time to finish it. We can make add as a nice to have feature

Sure, will keep in mind, it could be added if the project finishes before time.

I think all is clear and you can start drafting a proposal

Sure. Thanks!

[arjungarg07]

@derberg Thanks, will soon be opening issues for further discussions 😃

[arjungarg07]

Some discussed points:

• Would keep Map as a default output and would be configuring it to obtain output following mermaid syntax.
• Providing documentation for the developers as a walkthrough to integrate more custom formats in this library.

[arjungarg07]

• it should show that App A has a link with App B through some message, but the relation goes through a given server

@derberg Since the Apps relation goes through a given server/broker, in the case of multiple servers described in a single AsyncAPI file, which one do we have to consider?

[magicmatatjahu]

@arjungarg07 Hello! At the moment we cannot define in spec which server exposes some channels, but currently we have PR for that asyncapi/spec#531 so for your question: you should consider all servers, which will be logical, it means: the application A can be hosted by 2 servers then you are considering 2 servers through which application B can communicate.

[derberg]

cool tool to consider https://reactflow.dev/
I admit I have not much knowledge about it, but assume that whatever is used to render flows, probably accepts some specific format, and this is the format that could be an output here

[arjungarg07]

@magicmatatjahu Thanks for clearing it out.

So we would distinguish relations between files on 3 levels.
1st level: server URL
2nd level: server protocol
3rd level: channels to which services subscribed or publishing

We could club 1st and 2nd levels too

Please let me know your thoughts on this 🤔

cool tool to consider https://reactflow.dev/

@derberg yeah! i checked it out yesterday and it's super cool, would love to experiment with it but first focussing on MVP 😄

[derberg]

but first focussing on MVP

👏🏼

We could club 1st and 2nd levels too

same URL with different protocols would be super unusual so you can definitely assume this is always referring to the same server

[magicmatatjahu]

@pcornelissen Thanks for your insights :) I almost agree with you, but there is some situation when pinned channel only for particular server can change the rest of flow. We can use the relationship (as a user/client) to check which flow will be "created" when for example I as a user/client will send message to create a new order in e-shop. Sending the order to this same channel, but in two different server can change flow. This situation is rare, but still possible. Of course, we should in first implementation go with simple solution which will cover almost cases, and then go with more complex and rare.

If we don't understand each other, it may be because I look at the current problem (app relationship) from the perspective of the user/dev who wants to send a message to a given channel and know what operations will be performed after sending the message/event, such as creating an order, adding loyalty points etc. on the example of an e-shop.

[arjungarg07]

@derberg @magicmatatjahu @jonaslagoni I ran the script for on Alvaro's flight service with a slight change of adding one more server in one of the spec.
Generating default output as:

Map(2) {
  'mqtt://localhost:1883,mqtt' => Map(2) {
    'flight/update' => {
      sub: [ 'Flight Monitor Service' ],
      pub: [ 'Flight Notifier Service' ]
    },
    'flight/queue' => {
      sub: [ 'Flight Subscriber Service' ],
      pub: [ 'Flight Notifier Service', 'Flight Monitor Service' ]
    }
  },
  'mqtt://163.124.11,mqtt' => Map(2) {
    'flight/update' => { sub: [ 'Flight Monitor Service' ], pub: [] },
    'flight/queue' => { sub: [], pub: [ 'Flight Monitor Service' ] }
  }
}

So the mermaid diagram of this output should be like this:

Any feedbacks are much appreciated.

[magicmatatjahu]

@arjungarg07 I love it! Really :) As I see everything is good, 'publish' operation go to channel, from channel you link to service/app's 'subscribe' operation. At the moment is good, but honestly it's a hard to understand in first look what is connecting to what, I mean, you should look on diagram and see that you can publish something to (eg) FlightNotifierService when you will publish to flight/update channel and etc. It's not a problem of your code, but probably with our "description" of publish/subscribe in spec, but overall, great job! :) What do you want to do next?

[arjungarg07]

Do we have to add metadata information in the default output itself or have a separate API for it?

[magicmatatjahu]

@arjungarg07 Do you mean metadata like payload's schema of message etc? You operate on parsed spec, currently you save sub/pub only by name to array for appropriate channel name, to save metadata you can change the array to Map, so you have:

    'flight/update' => {
      sub: [ 'Flight Monitor Service' ],
      pub: [ 'Flight Notifier Service' ]
    },

but you can change it to Map<string, ChannelMetadata>

 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

Of course you can save as Channel metadata whole object of appropriate channel of given app as reference :)

... in the default output itself or have a separate API for it?

About separate API we can think in the next weeks, at the moment we can add metadata by default to default output/syntax or add option to enable adding metadata, which will be by default set to true.

[arjungarg07]

@derberg @jonaslagoni @magicmatatjahu Consider we need to get the metadata information like message and schema for each channel on this spec.

channels:
  flight/update:
    description: |
      Provides updates from a subscribed flight
    subscribe:
      summary: Inform about the status of a subscribed flight
      message:
        $ref: '#/components/messages/flightStatus'
 flight/queue:
    description: |
      Queues a flight in order to retrieve status
    publish:
      summary: Subscribe about the status of a given flight
      message:
        $ref: '#/components/messages/flightQueue'

I can't find any API available in parser-js through which I can get the served message information on a particular channel, like

const messageInfo = doc.channel('flight/update').message();

Same for schemas. Although we do have APIs doc.allMessages() and doc.allMessages() but it doesn't give info about on which channel it is being served.
Please correct me if I am missing something.

[magicmatatjahu]

@arjungarg07 Because message is in the operation item of channel - publish or subscribe. You must use in your case doc.channel('flight/update').subscribe().message();

What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?

EDIT: We have also corresponding hasSubscribe() and hasPublish() functions to check if given operation is available in channel. There is a full API of parser-js https://github.com/asyncapi/parser-js/blob/master/API.md

[arjungarg07]

Because message is in the operation item of channel

Ohhk, this I missed :) Thanks!

What about saving all channel's metadata (with description etc) as metadata - not only message - in relation?

TBH, I didn't discuss it with Luksaz about what all to save in metadata coz i thought of asking it along the development phase. I would suggest to consider all channel's metadata as you said.

EDIT: Also, do we need to have payload as it is for metadata or just some specific parts of it?

EDIT: We have also corresponding hasSubscribe() and hasPublish() functions to check if given operation is available in channel. There is a full API of parser-js https://github.com/asyncapi/parser-js/blob/master/API.md

Yeah, I have been checking from here itself :)

[magicmatatjahu]

EDIT: Also, what do we need to have payload as it is for metadata or just some specific parts of it?

Payload is a shape of (usually) json which should be sent to channel or will be subscribe from channel, so payload is a
complete whole that cannot be divided into smaller parts :) If you think about different problem, please describe it in different way, because probably we don't understand each other :)

[arjungarg07]

@magicmatatjahu Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references, when we will extract the metadata information from let's say message payload do we have to consider some specific fields from it or just include the whole payload itself like this:

message: {
      payload: {
        type: 'object',
        '$id': 'PlayerItemPickupPayload',
        additionalProperties: false,
        properties: {
          pickupTimestamp: {
            type: 'string',
            format: 'date-time',
            description: 'The timestamp the item was picked up',
            'x-parser-schema-id': '<anonymous-schema-4>'
          }
        }
      },
      'x-parser-original-schema-format': 'application/vnd.aai.asyncapi;version=2.0.0',
      'x-parser-original-payload': {
        type: 'object',
        '$id': 'PlayerItemPickupPayload',
        additionalProperties: false,
        properties: {
          pickupTimestamp: {
            type: 'string',
            format: 'date-time',
            description: 'The timestamp the item was picked up'
          }
        }
      },
      schemaFormat: 'application/vnd.aai.asyncapi;version=2.0.0',
      'x-parser-message-parsed': true,
      'x-parser-message-name': '<anonymous-message-1>'
    }
  }

Considering the whole payload would be a lot.

Also, I was thinking of wtiting tests after implementing the metadata extraction funcitonailty. What do you think? 🤔

[magicmatatjahu]

@arjungarg07 I see your problem but you also put example on which I can describe my thinking :)

You can treat only payload as metadata, but we must also know that dev can put inside message object specific information for given protocol, this same as in given server, so for me this is also very important metadata which we should show to user. Also channel can have specific protocol information in bindings object https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#channelBindingsObject You can have protocol info in message, operation (publish/subscribe), channel and server object. If we only consider message and it's value, someone can say that headers are also very important as metadata. I would like to say that whole data in a channel is very important metadata, so I think that treating whole channel as metadata would be the best option for our case, so I will remind you in which way I would like to do that:

 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

and then you don't focus which part of message is important, because you treat whole channel as metadata.

Also:

Yeah, what I mean to say is that since we have great support for traversing schemas with support for circular references

No, we don't, we only handle simple circular refs :) I created issue for that asyncapi/parser-js#293 so please have it in mind when you will use traverse function :)

What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future.

[arjungarg07]

No, we don't, we only handle simple circular refs :) I created issue for that asyncapi/parser-js#293 so please have it in mind when you will use traverse function :)

Yeah, I just tried to iterate over it and it caused recursive stackoverflow😅

What do you think about treating channel as metadata. It's easiest solution for our problem and will give us as many possibilities for the future

I agree with the explanation that the metadata's importance could change for every dev and treating the channel as metadata itself would be the best possible solution 👍🏽

 'flight/update' => {
   sub => {
     'Flight Monitor Service' => Channel metadata,
     'Flight Notifier Service' => Channel metadata,
   }
 }

Here flight/update is the channel and since channel's metadata would be same for every service(Flight Monitor Service,Flight Notifier Service, etc...), won't be mapping the same metadata with every service, instead we should have it like this:

 <channelName> => {
    sub: [ ],
    pub: [ ],
    metadata: {}
  }

EDIT: I get it, you are right! channel's metadata could change for every service, sorry :)

@magicmatatjahu Also, I think treating channel as metadata would have a huge ton of info to put in mermaid class diagram. How we would deal with that 🤔

[arjungarg07]

@derberg @magicmatatjahu Check out these tools: kroki, chartmage, UMLGraph, yuml

I am working on getting syntax ready for dotUML but yeah first include the tests of the already implemented functionality 😅

[arjungarg07]

@derberg @magicmatatjahu I was playing a bit with React Flow 😅 and it turns out that they don't have any playground online where we can just paste the syntax and render the diagram. Then I thought of generating an HTML page coz I don't want to set up a whole react project just for a simple diagram, but for that(HTML file) I am not able to find and cdn/browser package for react-flow-renderer so that I can include it in that HTML file and code it further. I think if I could get this cdn package which I could include in HTML code, we could use puppeteer to automate opening the HTML page and taking a screenshot of the page, and save it in the system.

Any workarounds/suggestions for this?

[derberg]

function for react flow should just generate the relations, the code responsible to show them, not entire website. This is a library, I integrate it however I want in my system, generate relations and enable in my HTML. So basically the goal is to focus on react flow code that shows relations between nodes, the HTML to present it can just be an example, part of test or a demo. Does it make sense?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴
It will be closed in 60 days if no further activity occurs. To unstale this issue, add a comment with detailed explanation.
Thank you for your contributions ❤️