IGC Repo Proxy - SemanticAsignment triggers multiple events about egeria-connector-ibm-information-server

[ruxandraRosu]

@cmgrote It looks like it's updating also the relationalColumnType entity. The operation was only a semantic assignment. I've attached some event samples.
event1.txt
event2.txt
event3.txt
event4.txt
event5.txt
event6.txt

[ruxandraRosu]

Yes, it's more of an optimisation issue, it doesn't break anything from what I've seen

[cmgrote]

Hi @ruxandraRosu -- would be good if we can rule out that these are coming through from an environment where metadata pre-existed before the IGC Repository Connector was pointed at it (and possibly before event notification was enabled in the environment). In a scenario where metadata pre-existed, recall that the IGC Repository Connector's Event Mapper will automatically "catch-up" to ensure that all relationships (and entities at the ends of those relationships) for the asset you update in IGC also have events triggered...

Assuming that we first setup event notification in the environment, and connect the IGC Repository Connector to the same environment while it is empty (before any metadata has been populated), I would expect that making a SemanticAssignment relationship alone would result in the following:

• 2 UPDATED_ENTITY events (one for the GlossaryTerm, one for the Referenceable assigned to it)
• 1 NEW_RELATIONSHIP event

If you can share details on the messages themselves it would be interesting to see if there might be something else happening: eg. another UPDATED_ENTITY and UPDATED_RELATIONSHIPs because of some other relationship also being triggered from some other property that was changed at the same time as the SemanticAssignment was setup. The 2+1 I mention above I would only expect if you modify eg. a database column in IGC and only change its assigned_to_terms property (and nothing else). If other properties are also changed, and one of those properties represents a relationship, that will trigger other relationships as well...

All of that said, it's probably worth waiting until #544 is merged before re-testing, as that makes a number of other changes which may impact here as well.

[cmgrote]

Thanks! That example is a good one, as currently changes aren't really distinctly identified between things like RelationalColumn and RelationalColumnType I'm likely outputting both when a property on IGC's database_column (which effectively represents both OMRS entities) changes. I'll try to investigate how to better distinguish these to limit the messages. Is it safe to assume this is more of an optimisation than a breaking issue currently, though?

[cmgrote]

Browsing through the latest set of sample payloads (after the type simplification from TypeEmbeddedAttribute and NestedSchemaAttribute) this duplication appears to be resolved -- at least for these SemanticAssignment examples.

Will close for now, but suggest we open a new issue if we identify any duplication elsewhere.