IBM Information Server connectors for Egeria: repository proxy connector for IGC, data engine proxy connector for DataStage.
Home Page: https://odpi.github.io/egeria-connector-ibm-information-server
License: Apache License 2.0
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Remove the supported status validation from the verifyTypeDef methods and into the addEntity, etc methods -- so long as ACTIVE and DELETED are supported for read-only scenario, the others we can detect at creation (write) time whether we support them or not.
Need to understand what methods need to be implemented and their intended behaviour.
Per consolidation of SchemaAttribute and SchemaType properties that will remove the need to manage eg. RelationalColumn, RelationalColumnType split (see: odpi/egeria#1317)
At the moment we are only forwarding the IGC RID as the GUID, which isn't sufficient as the RID is only a hashed internal key and therefore not globally unique across instances (of IGC).
We need to make the RIDs globally unique to be proper GUIDs:
On testing the event mapper bugs in #17 a number of different times it seems that the payloads that are generated are not consistent. In particular, sometimes certain objects are identified as having changed (typically around data_file_records and their contained data_file_fields) and other times not. When it did appear, it looked like it was to do with the items array of relationship properties being ordered differently: or at last being detected as being ordered differently.
Needs deeper investigation: have added thorough debug logging to output the diffs that are calculated and the full source and target objects that were diff'd. Next step is to reproduce the problematic diffs (with debug logging on) to see whether the problem appears to be from:
items array ordering from IGC from one query to another (within a relationship property, so sorting cannot be applied)JsonDiff results despite the compared objects actually having the same array orderingThe issue is that the diffs being generated include either replace or remove ops, causing DELETE_PURGED_RELATIONSHIP_EVENTs to be triggered -- and in many cases the last event raised for a given relationship is this DELETE_PURGED_RELATIONSHIP_EVENT which would mean the relationship is entirely removed from the cohort (never replaced with a NEW_RELATIONSHIP_EVENT).
The qualifiedName does not end up being unique for types that use a generated prefix from IGC, for example:
database_table -> RelationalTable, RelationalTableTypedatabase_column -> RelationalColumn, RelationalColumnTypeetc.
Anyone have a particular priority they're after?
DELETE_PURGE for all of those as welldatabase_table should do so for database_columns) and send a DELETE_PURGE for all of those as wellCan we setup a project for this repo under the ODPi org in Sonarcloud, to get quality checks against this repo? (Let me know if we should assign to someone else -- but I don't have admin privileges to do it myself 🙂)
Currently EntitySummary objects are always initialised as EntityDetail objects, to make use of the skeleton entity helper method. Not sure if this should really be continued, or if we should create an EntitySummary-specific skeleton creation helper method?
Implement a discovery service that exposes / integrates Information Analyzer to the Open Discovery Framework of Egeria.
Implement a capability to default the zone(s) for all assets within a connected IGC repository eg. based on a configuration parameter; with a way to store any overrides (at per-asset level) that may be determined elsewhere in the cohort.
Investigate adding a mapping for the OMRS Glossary type, for example:
GlossaryTermCategoryWill possibly require some re-working of how we identify the same item in IGC (category) being mapped to multiple OMRS types.
However, this would also be useful for operations like declassifyEntity to be able to remove terms from other category-based concepts (like SubjectArea) while still being able to retain the terms within the same Glossary (by moving their parent_category assignment up to the top-level category that represents the glossary).
As a tactical step towards odpi/egeria#1622, investigate implementing a parallel write to the DE OMAS InTopic:
(Parallel in the same way as DE OMAS: all granular steps in parallel, only then all jobs in parallel, etc -- to ensure that order is still there for the granular objects to exist before attempting to create PortAliass that refer to the PortImplementations and so on)
A semantic assignment triggers multiple OMRS events: UPDATED_ENTITY(3), NEW_RELATIONSHIP(1), UPDATED_RELATIONSHIP(2).
Are all events necessary? This could add some overhead on omrs/omas's level.
Currently information from IGC is typically retrieved by a double-search: a first search (by ID, with no properties) to see what type of asset is being requested, then using this type information to determine what properties to include in the follow-on search, and then a second search (by ID, including those properties).
(This is due to the fact that doing searches limited to the properties of interest is generally more performant than retrieving an asset and all of its properties when many of those properties will not actually be used.)
Ideally we could reduce this to a single up-front search; however, to do so we would need to know the IGC asset type of the RID being requested. There unfortunately isn't a way to determine this based on the RID itself...
The simplest solution would probably be to further extend the GUIDs that IGC returns to embed the IGC data type as well -- which would allow us to then use this to do the lookup in a single search, by avoiding the need for the initial property-less by-ID search to determine the data type (from which to then choose the properties to retrieve).
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project igc-rest-client-library: Compilation failure: Compilation failure:
[ERROR] /Users/jonesn/IdeaProjects/egeria/open-metadata-implementation/adapters/open-connectors/repository-services-connectors/open-metadata-collection-store-connectors/ibm-
igc-repository-connector/igc-rest-client-library/src/main/java/org/odpi/openmetadata/adapters/repositoryservices/igc/clientlibrary/model/generated/v11702/DsparameterSet.java:[6,23] error: package javax.annotation does not exist
[ERROR] /Users/jonesn/IdeaProjects/egeria/open-metadata-implementation/adapters/open-connectors/repository-services-connectors/open-metadata-collection-store-connectors/ibm-igc-repository-connector/igc-rest-client-library/src/main/java/org/odpi/openmetadata/adapters/repositoryservices/igc/clientlibrary/model/generated/v11702/DsparameterSet.java:[21,1] error: cannot find symbol
[ERROR] symbol: class Generated
This is due to removal in java 9 - so issues with the import
need to figure out the clean, safe way to fix & work in both 8 and 11.
some tips at https://stackoverflow.com/questions/48238014/how-can-be-solved-java-lang-noclassdeffounderror-javax-annotation-generated
Currently the general mappings use Java Reflection to dynamically access properties within the Jackson beans. This is necessary to keep the mapping logic generic, while still using Jackson as a (de)serialisation framework, but Reflection is slow... A little research indicates that as of Java 8 there is a new approach available: https://docs.oracle.com/javase/8/docs/api/java/lang/invoke/LambdaMetafactory.html
... which seems to use the underlying direct access of Java itself and is intended to therefore be nearly as fast as directly calling a getter -- while still allowing for dynamic invocation akin to Reflection.
Maybe possible to migrate the existing Reflection-based dynamic retrieval to this approach?
References:
A number of minor versions of Information Server seem to have issues with one of the queries used to retrieve information about DataStage job details, specifically the ability to retrieve all ds_stage_columns by searching based on either their link.job_or_container property or even the link property directly (using an in semantic`).
Versions that appear not to work:
Versions that appear to work:
May require either deeper investigation of the non-working releases, or consider alternative mechanism for retrieving the stage column details that works more reliably across all releases.
Relationship searches for relationship-level properties with literally-mapped values will still return results when a NONE semantic is used with a literal value that matches the literal mapping. When this is the case, the search should instead ensure that no results are returned (since all results will have that literal value).
Ensure that all of the necessary jar files are created for the combined connector package:
A number of subtle search bugs were discovered in v1.2:
qualifiedName are only possible using an exact match semantic, but should be possible as well with contains, startsWith and endsWith (even if there is a performance penalty associated with these)qualifiedName were being used to restrict the search for optimization purposes in all cases, even when there is a match criteria of NONE (resulting in fewer results in some cases)Identified further edge cases in searching that are problematic:
_name is in fact the concatenation of the courtesy_title and full_name fields (cannot use the context, which will always be empty for these types, or _name alone to search).The nested LineageMapping objects should only contain mappings between the PortImplementations owned by that process -- and the process-to-process mappings should come in a subsequent (and separate) payload.
When calling the above method it seems to be capable of returning a result with more entries than the designated pageSize.
A concrete example is when searching for relationships of type "SemanticAssignment".
In the IGC repository being used in the CTS/IGC connector test, there appear to be 18 instances of relationship type SemanticAssignment. In the CTS relationship search test case a find... is issued with the following parameters:
typeGUID is set to null - meaning relationships instances of any type can be returned
matchProperties contains "confidence" set to a value of 100.
pageSize is set to 10
The result returned contains 31 instances:
3. : ForeignKey
18 : SemanticAssignment
10 : DataClassAssignment
Note that this is a "wild" test - in that it does not specify a type filter. The result may therefore be an aggregate of separate underlying type-specific queries, which could lead to a result greater than pageSize. However, the result contains more than the pageSize of instances of type Semantic~Assignment, so it appears that even a sub-query can find and return more than pageSize instances.
Side-note: I could make the CTS resilient to getting back more instances than it is expecting but as things stand it is proving to be quite a good test.
In another test, I noticed that relationships of type DataClassAssignment can also exceed the pageSize limit, when searched for with the appropriate typeGUID and an empty MatchProperties parameter. In this test pageSize is also set to 10, but 53 instances were returned.
I'm happy to reproduce and provide screen captures etc if it helps.
For each virtual asset, create the "full context" including:
Need to decide how best to do this: eg. via Data Platform OMAS, Data Engine OMAS, or elsewhere?
Hi ODPI Egeria team,
I deployed Egeria on a dedicated Kubernetes cluster on the IBM Cloud (Yellow zone). Between the cluster and the IGC (Blue Zone) is a firewall, which was configured to allow communication. Everything seemed to work fine until the IGC was restarted. From that point on, it was neither possible to exchange information (Glossary or assets), nor has the connection ever recovered. Purging and reinstalling the helm chart restores the working state, but naturally leads to losing all assets, that have been transferred up to this point. Here is a piece of the IGC pod log:
2019-06-12 15:13:20.623 ERROR 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Request failed -- check IGC environment connectivity and authentication details.
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error
at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:124) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
2019-06-12 15:13:20.624 ERROR 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Unable to complete request -- check IGC environment connectivity and authentication details.
2019-06-12 15:13:20.624 ERROR 1 --- [ Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.
java.lang.NullPointerException: Unable to complete request -- check IGC environment connectivity and authentication details.
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:477) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
2019-06-12 15:13:20.625 WARN 1 --- [ Thread-4] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Synchronous auto-commit of offsets {InfosphereEvents-0=OffsetAndMetadata{offset=4398, leaderEpoch=null, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2019-06-12 15:13:20.625 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Revoking previously assigned partitions [InfosphereEvents-0]
2019-06-12 15:13:20.625 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] (Re-)joining group
2019-06-12 15:13:20.625 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Group coordinator xxxxxxx.xxx.xxx:59092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
2019-06-12 15:13:21.221 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Discovered group coordinator xxxxxxx.xxx.xxx:59092 (id: 2147483646 rack: null)
2019-06-12 15:13:21.222 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] (Re-)joining group
2019-06-12 15:13:21.817 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Successfully joined group with generation 1
2019-06-12 15:13:21.817 INFO 1 --- [ Thread-4] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Setting newly assigned partitions [InfosphereEvents-0]
2019-06-12 15:13:26.308 WARN 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Request failed -- session may have expired, retrying...
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
2019-06-12 15:13:47.991 WARN 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Request failed -- session may have expired, retrying...
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.openNewSessionWithRequest(IGCRestClient.java:215) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:444) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
2019-06-12 15:13:47.991 ERROR 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Opening a new session already attempted without success -- giving up on POST to https://xxxxxxx.xxx.xxx:444/ibm/iis/igc-rest/v1/search with {"types":["database"],"properties":["name","short_description","long_description","host","labels","stewards","assigned_to_terms","implements_rules","governed_by_rules","database_schemas","alias_(business_name)","location","dbms","dbms_server_instance","dbms_vendor","dbms_version","database_type","imported_from","data_connections","data_connection_mappings","mapped_to_mdm_models","read_by_(static)","written_by_(static)","read_by_(design)","written_by_(design)","read_by_(operational)","written_by_(operational)","read_by_(user_defined)","written_by_(user_defined)","impacted_by","impacts_on","bi_reports","bi_models","suggested_term_assignments","data_policies","blueprint_elements","in_collections","created_by","created_on","modified_by","modified_on"],"pageSize":1000,"where":{"conditions":[{"property":"_id","operator":"=","value":"b1c497ce.6e83759b.9g85t7kh2.9ge978u.81ro06.344cvqapg9q7d4ofhmog8"}],"operator":"and"}}
2019-06-12 15:13:47.991 ERROR 1 --- [ Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient : Unable to complete request -- check IGC environment connectivity and authentication details.
2019-06-12 15:13:47.992 ERROR 1 --- [ Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.
java.lang.NullPointerException: Unable to complete request -- check IGC environment connectivity and authentication details.
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:477) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
(@cmgrote edited for log formatting)
as changed in the core repo as well, to confirm that we have no unused dependencies sitting around and no required dependencies that are not explicitly included where needed
Some values in IGC are list-wrapped in some versions of the software and not others, or within some level of the inheritance hierarchy but not others. (For example: length of a database_column vs. the same of its super-type data_item.)
We need to ensure that when a primitive value is expected, this list-wrapping of IGC is properly unwrapped to give a singular primitive value for OMRS.
Followon from odpi/egeria#2348 and PR odpi/egeria#2353 -- This may need to be transferred to the IGC connector repo (@cmgrote), but may also be a generic issue we need to tackle in our connectors
A user recently was trying to make use of the IGC connector and saw the following
Wed Jan 08 11:09:28 GMT 2020 myserver Exception OMRS-AUDIT-0034 The connector to the local repository failed with a org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectionCheckedException exception and the following error message: OCF-CONNECTION-500-010 Invalid Connector class org.odpi.egeria.connectors.ibm.igc.repositoryconnector.IGCOMRSRepositoryConnector
From the client the following was seen when activating the server (instance)
{"class":"SuccessMessageResponse","relatedHTTPCode":500,"exceptionClassName":"org.odpi.openmetadata.adminservices.ffdc.exception.OMAGConfigurationErrorException","exceptionErrorMessage":"OMAG-ADMIN-500-001 Method activateWithSuppliedConfig for OMAG server myserver returned an unexpected exception of org.odpi.openmetadata.repositoryservices.ffdc.exception.OMRSConfigErrorException with message OMRS-CONNECTOR-400-004 The connection null passed to the EnterpriseOMRSRepositoryConnector is invalid.","exceptionSystemAction":"The system is unable to configure the OMAG server.","exceptionUserAction":"This is likely to be either an operational or logic error. Look for other errors. Validate the request. If you are stuck, raise an issue."}
The initial interpretation was that the class was not found. However all became clear when logging was enabled (-Dlogging.level.root=ERROR)
nigel@MEGA:/mnt/c/users/nigel/tmp$ export LOGGING_LEVEL_ROOT=ERROR
nigel@MEGA:/mnt/c/users/nigel/tmp$ java -jar server-chassis-spring.jar
ODPi Egeria
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / ____/ / ___/ ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ /__ ____ _ _
/ / / // /|_/ // /| | / / __ \__ \ / _ \ / __/| | / // _ \ / __/ / /_/ // // | / _\ / /_ / | / _// || |
/ /_/ // / / // ___ |/ /_/ / ___/ // __// / | |/ // __// / / __ // // / \ / /_ / _// / // / / / / /
\____//_/ /_//_/ |_|\____/ /____/ \___//_/ |___/ \___//_/ /_/ /_/ \__/\//___//_/ \__//_/ /_/ /_/
:: Powered by Spring Boot (v2.2.2.RELEASE) ::
No OMAG servers listed in startup configuration
Wed Jan 08 11:09:24 GMT 2020 OMAG server platform ready for more configuration
Wed Jan 08 11:09:28 GMT 2020 myserver Information OMRS-AUDIT-0001 The Open Metadata Repository Services (OMRS) is initializing
Wed Jan 08 11:09:28 GMT 2020 myserver Information OMRS-AUDIT-0003 The local repository is initializing with metadata collection named myserver with an id of a98b82c1-f638-44fc-bb74-764ebcdac87f
Wed Jan 08 11:09:28 GMT 2020 myserver Information OMRS-AUDIT-0029 The local repository outbound event manager is initializing
2020-01-08 11:09:28.498 ERROR 4522 --- [nio-8080-exec-5] o.o.e.c.i.i.clientlibrary.IGCRestClient : Request failed -- check IGC environment connectivity and authentication details.
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://infosvr:9446/ibm/iis/igc-rest/v1/search": infosvr; nested exception is java.net.UnknownHostException: infosvr
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:751) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:605) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:496) [egeria-connector-ibm-information-server-package-1.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:528) [egeria-connector-ibm-information-server-package-1.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:791) [egeria-connector-ibm-information-server-package-1.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient.start(IGCRestClient.java:186) [egeria-connector-ibm-information-server-package-1.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.odpi.egeria.connectors.ibm.igc.repositoryconnector.IGCOMRSRepositoryConnector.initialize(IGCOMRSRepositoryConnector.java:111) [egeria-connector-ibm-information-server-package-1.3-SNAPSHOT-jar-with-dependencies.jar:na]
at org.odpi.openmetadata.frameworks.connectors.ConnectorProviderBase.getConnector(ConnectorProviderBase.java:225) [open-connector-framework-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.frameworks.connectors.ConnectorBroker.getConnector(ConnectorBroker.java:399) [open-connector-framework-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.repositoryservices.localrepository.repositoryconnector.LocalOMRSConnectorProvider.getConnector(LocalOMRSConnectorProvider.java:159) [repository-services-implementation-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.repositoryservices.localrepository.repositoryconnector.LocalOMRSConnectorProvider.getConnector(LocalOMRSConnectorProvider.java:112) [repository-services-implementation-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.repositoryservices.admin.OMRSOperationalServices.getLocalOMRSConnector(OMRSOperationalServices.java:1128) [repository-services-implementation-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.repositoryservices.admin.OMRSOperationalServices.initializeLocalRepository(OMRSOperationalServices.java:636) [repository-services-implementation-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.repositoryservices.admin.OMRSOperationalServices.initialize(OMRSOperationalServices.java:379) [repository-services-implementation-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adminservices.OMAGServerOperationalServices.activateWithSuppliedConfig(OMAGServerOperationalServices.java:328) [admin-services-server-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adminservices.OMAGServerOperationalServices.activateWithStoredConfig(OMAGServerOperationalServices.java:149) [admin-services-server-1.3-SNAPSHOT.jar!/:na]
at org.odpi.openmetadata.adminservices.spring.OperationalServicesResource.activateWithStoredConfig(OperationalServicesResource.java:41) [admin-services-spring-1.3-SNAPSHOT.jar!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_232]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_232]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_232]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar!/:9.0.29] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar!/:9.0.29] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar!/:9.0.29] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar!/:9.0.29] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar!/:9.0.29] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.29.jar!/:9.0.29]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
Caused by: java.net.UnknownHostException: infosvr
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_232]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_232]
at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_232]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) ~[na:1.8.0_232]
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) ~[na:1.8.0_232]
at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[na:1.8.0_232]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_232]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_232]
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[na:1.8.0_232]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[na:1.8.0_232]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[na:1.8.0_232]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[na:1.8.0_232]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[na:1.8.0_232]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[na:1.8.0_232]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167) ~[na:1.8.0_232]
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:742) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
... 67 common frames omitted
2020-01-08 11:09:28.518 ERROR 4522 --- [nio-8080-exec-5] o.o.e.c.i.i.clientlibrary.IGCRestClient : Unable to complete request -- check IGC environment connectivity and authentication details.
2020-01-08 11:09:28.518 ERROR 4522 --- [nio-8080-exec-5] o.o.e.c.i.i.clientlibrary.IGCRestClient : Unable to start IGCRestClient: no authorization provided.
Wed Jan 08 11:09:28 GMT 2020 myserver Exception OMRS-AUDIT-0034 The connector to the local repository failed with a org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectionCheckedException exception and the following error message: OCF-CONNECTION-500-010 Invalid Connector class org.odpi.egeria.connectors.ibm.igc.repositoryconnector.IGCOMRSRepositoryConnector
The error in the logs was essential to progress debugging this situation - here it was that the connector code was launched, but could not initialize due to a lack of connectivity with the server
(as an aside, IGC specific, but connection refused could be a transient, not permanent error... ie in the situation where systems are coming up....)
currently i am running java -Dloader.path='/home/centos/egeria/egeria-connector-ibm-information-server-package-1.3-20200106.120036-38-jar-with-dependencies.jar' -jar egeria_server_1.3.jar
in cento 7 unix environment and but i having issue while connecting IGC below is the error
Wed Jan 08 08:22:06 CET 2020 myserver Information OMRS-AUDIT-0001 The Open Metadata Repository Services (OMRS) is initializing
Wed Jan 08 08:22:06 CET 2020 myserver Information OMRS-AUDIT-0003 The local repository is initializing with metadata collection named myserver with an id of c0a75f07-cc8f-4af3-a706-1b98c1e2508c
Wed Jan 08 08:22:06 CET 2020 myserver Information OMRS-AUDIT-0029 The local repository outbound event manager is initializing
Wed Jan 08 08:22:06 CET 2020 myserver Exception OMRS-AUDIT-0034 The connector to the local repository failed with a org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectionCheckedException exception and the following error message: OCF-CONNECTION-500-010 Invalid Connector class org.odpi.egeria.connectors.ibm.igc.repositoryconnector.IGCOMRSRepositoryConnector
can some one help to resolve this
At least the following:
Note that creating or saving jobs in the DataStage Designer client does not trigger any events into the InfosphereEvents topic... May need to consider an alternative mechanism for polling for changes to jobs (modified_on timestamp?) to trigger events in the EventMapper.
The above method returns an unexpected number of results during searches for the following types:
FileFolder
TabularSchemaType
In the former case the match type is Exact. In the latter it is Suffix.
The searchCriteria in both cases is a RepositoryHelper generated regex expression such as:
(For FileFolder) : \Q(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files\E
(For TabularSchemaType) : .*\Q(data_file_folder)=CocoPharma::(data_file)=Patient-Patient.csv::(data_file_record)=Patient-Patient\E
Happy to reproduce and provide debug details
Cleanup compiler warnings, mostly around casting.
We had previously decided to store the IGC RID in the qualifiedName of certain DataStage entities (like jobs and stages), with the idea that this would give us a way to look them back up by ID.
However, we have since modified the GUID structure from IGC so that the metadata collection ID of the IGC repository is embedded in the GUIDs -- therefore, the RID alone is insufficient to do this reverse-lookup. Furthermore, because the DataStage connector uses its own configuration (does not re-use any IGC connector), it will not be aware of what the metadata collection ID of the IGC repository is when setting these qualifiedNames. As a result, these RIDs are probably less useful than what we're using as a qualifiedName everywhere else...
Should we therefore revert back to using the same qualifiedName as used elsewhere rather than a RID?
During entity search tests the following error was seen on the following 3 types (only):
Glossary
Team
DataClass
The error is the same in each case: the attribute being searched is always "description".
Other attributes have been successfully searched (including "qualifiedName", "displayName") and returned the expected entity instances. It seems to be just when searching on attribute "description" that things go wrong.
The error is reported as a RepositoryErrorException.
This is always during TypeFiltering tests on the preliminary query in which no filter is specified.
The actual error is wrapped and reported as:
"errorMessage": "CTS test repository-entity-property-search caught exception RepositoryErrorException from method findEntitiesByProperty whilst trying to find entities with no type filter . Exception message was : An unexpected org.odpi.egeria.connectors.ibm.igc.clientlibrary.errors.IGCConnectivityException exception was caught by findEntitiesByProperty; error message was Opening a new session already attempted without success -- giving up.. Method was invoked with parameters: limitResultsByClassification : null, sequencingOrder : null, typeFilter : null, asOfTime : null, matchCriteria : All, limitResultsByStatus : null, matchProperties : InstanceProperties{propertyNames=java.util.HashMap$KeyIterator@2ae4d647, propertyCount=1, instanceProperties={description=PrimitivePropertyValue{primitiveValue=\QThis glossary contains Glossary Terms and Categories that are related to the Coco Pharmaceuticals data\E, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}}}, sequencingProperty : null, fromEntityElement : 0, pageSize : 10",
I'm happy to reproduce and capture debug etc.
Investigate use of an external component (like Redis) for the storage of the OMRS stub information, to both improve overall performance of the event mapper as well as (probably) increase its scalability.
During CTS testing using the relationship search test case, the above method is seen to return instances that are not a match for the search parameters.
The concrete example is:
Type: DataClassAssignment
There are reported to be 53 instances of this type in the repository.
Only 7 instances have the property "partialMatch" set - in all 7 it is set to "true"
These 7 instances also have values for "confidence", "valueFrequency" and "status"
Performing findRelationhsipsByProperty with a MatchProperties parameter containing partialMatch : true and matchCriteria set to ALL, the method returned 26 results. Only 7 of the returned instances contained a value for partialMatch.
I am happy to reproduce and provide screen dumps if it helps.
Implement required method and event mapper functionality to handle refresh requests and place such events into the OMRS topic when requested.
Currently the classification categories generate GlossaryCategory instance events -- while the top-level Classifications category is treated properly, any subcategories (like Confidentiality) still generate GlossaryCategory instance events.
These therefore still seem to be treated as GlossaryCategory objects somewhere in the event mapper, which they should not be.
ie. need to use the IGC prefixes for things like TabularColumnType, etc on the qualifiedNames for them to be appropriately found.
Currently the various reads_from and writes_to properties that are useful for lineage are duplicated across the various lineage-participating object types (10's of them). Instead of this duplication, promote the properties into InformationAsset -- where that way they can also be more reliably re-used across the object inheritance structure by eg. the Data Engine proxy code.
Hi ODPI Egeria team,
I deployed Egeria on a dedicated Kubernetes cluster on the IBM Cloud (Yellow zone). Between the cluster and the IGC (Blue Zone) is a firewall, which was configured to allow communication. This time the communication breaks down after trying to publish an asset type, that can't be mapped. Again: From that point on, it was neither possible to exchange information (Glossary or assets), nor has the connection ever recovered. As in my earlier issue "Egeria-IGC-connectivity issues - Part I", purging and reinstalling the helm chart restores the working state, but naturally leads to losing all assets, that have been transferred up to this point. Here is a piece of the IGC pod log:
2019-06-11 18:48:54.469 ERROR 1 --- [ Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : EntityDetail could not be retrieved for RID: b1c497ce.54bd3a08.9g868p8ap.1n94o64.biqmmt.d2vu7883hkqtnc5gknm6a
2019-06-11 18:48:54.469 WARN 1 --- [ Thread-4] o.o.o.a.r.i.r.stores.EntityMappingStore : Unable to find mapping for IGC type: view
2019-06-11 18:48:54.469 ERROR 1 --- [ Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.
Scenarios:
ProcessTemplateProcess -- this includes the case where a lineage mapping ("alias" / same-as) has been created for the inputs / outputs after deployment (but still before run-time).Process, with relationship to a Process(Template) above; run-time stats (elapsed time, record counts, etc) would only go to Open Lineage services (not via Egeria)We should rename this repository to encompass IBM Information Server (the full suite) rather than just IGC. It already covers some metadata that comes from Information Analyzer, and with the addition of the Data Engine Proxy for DataStage it's covering more and more of the suite as a whole rather than just IGC.
Some bugs noted in the current IGC event mapper logic that need to be fixed:
UPDATE when the relationship in IGC changes, irrespective of the endpoints being the same or different -- should probably be a DELETE_PURGE followed by a NEW when the GUID of the relationship will change (ie. one of the endpoints of the relationship changes). UPDATE should presumably only be used when the properties of the relationship change but the endpoints remain the same (ie. the GUID remains unchanged).DELETE_PURGE event (or at least it is deleted until some other relationship on the changed category occurs?)DELETE_PURGE event for the original, nor a NEW_RELATIONSHIP for the new oneUPDATED_RELATIONSHIP events are being sent before NEW_RELATIONSHIP events, when one end of the relationship is a generated type (eg. __|RDBST|__)DELETE_PURGE for all of those as welldatabase_table should do so for database_columns) and send a DELETE_PURGE for all of those as wellDELETE_PURGE event seems to be generating many PURGED_ENTITY events as well (see below) -- not sure if this is correct (or if not, where it has gone wrong?)Per discussion under odpi/egeria#1732 there is likely re-work that can be done to the IGC proxy connector in order to better align with the OCF.
Consider caching any information used in classification decisions up-front as part of server startup, and refresh as-needed based on notifications.
For example: the details of Confidentiality (modelled as a category with a few contained terms) need to be retrieved repeatedly, as separate REST calls, for each entity related to one of its contained terms -- it in order to confirm the parent category of those contained terms is Confidentiality (and is not some just some "normal" term).
If cached up-front, this could significantly reduce the repeated re-retrieval of the same basic relationship information from a number of different entity starting points.
At least when configuring a connection using the repository proxy service, the AuditLog of the (remote) repository connector is not set before the repository connector itself is initialized. The result is that we have no access to record anything into the audit log as part of the (remote) repository connector initialization.
(Conversely, for the (remote) event mapper connection the audit log does appear to be set before the initialization of the event mapper itself.)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.