Comments (7)
Looking through the logic of the existing connector, I'm a bit wary of creating an entirely new connector as I suspect that much of the code will overlap (the queries to retrieve the jobs, track the last sync timestamp, configuration options for how much to retrieve, which jobs to include, constructing unique names for the various objects, choosing the attributes of each object to query and retrieve, how these are mapped across to the Egeria OMAS payloads, etc, etc).
I suspect the simplest way to ensure that this overlap can be maintained just once will be to add some configuration options to the existing connector to decide in which "mode" to run it (granular or high-level). This should still allow multiple instances of the connector to be run with different configurations, just as if they were two separate connectors... But it would mean that we don't have the complexity of trying to maintain two separate code areas that have a significant amount of overlap (inevitably causing maintenance headaches and regressions).
We could also create a third module which contains the overlapping pieces, but this is likely to be the most amount of work in the near-term as it will mean creating a new connector, creating the new module containing common code, and then modifying the existing connector to use this new module as well.
So I'd suggest we simply add a "mode" configuration option to the existing connector as a compromise? Keeps maintenance relatively simple, and means we don't have significant up-front work as well. (We can probably limit the complexity of conditional logic by wrapping the "mode"s up into different top-level methods in the connector class, and then just call that top-level method depending on the mode in which it has been configured.)
from egeria-connector-ibm-information-server.
Agree, the proper way will take way more effort and time. Then +1 for adding configuration parameter 'mode', and as always by default if not set we have the current processing mode.
from egeria-connector-ibm-information-server.
Per the auto-link above, I've added the initial logic to hopefully allow this mode to be configured. For flow1
of our minimal sample, this produces:
{
"qualifiedName": "_(host)=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1",
"displayName": "flow1",
"description": "",
"owner": "Administrator IIS",
"name": "flow1",
"lineageMappings":
[
{
"sourceAttribute": "_(host)=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1",
"targetAttribute": "(host)=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME"
},
{
"sourceAttribute": "(host)=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names",
"targetAttribute": "_(host)=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1"
}
],
"collection":
{
"qualifiedName": "(host)=INFOSVR::(transformation_project)=minimal",
"name": "minimal"
},
"updateSemantic": "REPLACE"
}
It can be configured as part of the overall connector configuration using the mode
setting of JOB_LEVEL
(if left out of the configuration, this defaults to the GRANULAR
setting which should give the original behaviour still):
{
"class": "DataEngineProxyConfig",
"accessServiceRootURL": "{{baseURL}}",
"accessServiceServerName": "omas",
"eventsClientEnabled": true,
"dataEngineConnection": {
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.egeria.connectors.ibm.datastage.dataengineconnector.DataStageConnectorProvider"
},
"endpoint": {
"class": "Endpoint",
"address": "{{igc_host}}:{{igc_port}}",
"protocol": "https"
},
"userId": "{{igc_user}}",
"clearPassword": "{{igc_password}}",
"configurationProperties": {
"mode": "JOB_LEVEL",
"limitToProjects": [ "minimal" ]
}
},
"pollIntervalInSeconds": 60
}
I've tried to test on my end but am getting various errors back that seem to be related to events processing in the OMAS (whether using the original behaviour or this new high-level lineage), so I'm presumably not using the latest configuration or something (not sure). Would be great if you can test further and see if it needs further revision?
from egeria-connector-ibm-information-server.
Forgot to ask, but what is the plan for these aspects in this new "job-level" mode?
- Sequences: do we still represent these as a higher-level abstraction on top of the jobs?
- Ports:
- do we create new PortImplementations that define the inputs / outputs for the jobs (which were previously PortAliases pointing down to the underlying PortImplementation details of individual stages)?
- do we still create PortAliases for the sequence level, that would presumably now refer to the new PortImplementations (above)?
- or do we entirely leave Ports out of this mode of operation (since they really need SchemaTypes anyway, which we won't have)? And if we go this route, do we just use sequences as part of a process hierarchy, but leave out any PortAliases?
from egeria-connector-ibm-information-server.
I am in favour of the most minimal option that is not including Sequences ( we did not do anything on this level right now in DE, right @popa-raluca ? ) nor Ports. I think we should leaving them out because they will be not used.
My reasoning behind this: In the new "job-level" mode we do not want to get the Implementation level details for the process (what are the stages of job and how they interconnect on schema level) making portImplementations and related schemas obsolete.
Similary for PortAliases, I do not thik we need them without having high level process-to-process mapping like job-to -sqeuence or job to job (never seen job-job in data stage btw).
I suggest to start outputing the minimal set like in the request sample above. (and we can always add sequence level details if we understand that there is use-case for this later on)
from egeria-connector-ibm-information-server.
Right now DE creates the sequence level processes. They get propagated through AL and stored in OLS, but they are not used in the querying part. I don't think sequences are needed for the new "job-level" mode right now.
from egeria-connector-ibm-information-server.
I've tried to test on my end but am getting various errors back that seem to be related to events processing in the OMAS (whether using the original behaviour or this new high-level lineage), so I'm presumably not using the latest configuration or something (not sure). Would be great if you can test further and see if it needs further revision?
On it.. we are going to start testing using some of the samples we have in our test environments using latest Egeria core. The output above looks expected and I am looking forward to see how it will go with real data. Keep you posted.
from egeria-connector-ibm-information-server.
Related Issues (20)
- Lack of context for data_files without data_file_folders HOT 8
- NPE when trying to get EntityDetail for rid that is not present in IGC
- Implement a NOOP event mapper HOT 1
- findEntities with multiple conditions does not return entities HOT 1
- Add differentiation between primary and secondary categories HOT 1
- Property 'assigned_to_terms' no longer supported for searching using 'main_object' generic type HOT 1
- Implement new classification for primary category HOT 1
- IGC repository connector - validation for instance properties mapped with wrong primitive type HOT 3
- Results for a relationship search by type (with multiple mappings) extend beyond pageSize
- Improve handling of complex search criteria for findEntities method HOT 2
- Searching for NEQ minCardinality on SchemaAttributes does not work
- repository-services getEntityDetail is ignoring MetadataCollectionId part of GUID HOT 2
- Move data_file_field mappings from TabularColumn to TabularFileColumn
- Data Stage connector batch processing sync state per instance
- Possible inconsistent behaviour HOT 2
- Data Stage connector capability to process virtual assets
- Mapping new typedef properties (Failed test ConnectorTest.verifySupportedTypes) HOT 2
- Can igc-adapter create anchor classifications for data asset types?
- SonarCloud Quality Gate Code duplication rule too restrictive HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from egeria-connector-ibm-information-server.