Coder Social home page Coder Social logo

odpi / egeria-docs Goto Github PK

View Code? Open in Web Editor NEW
22.0 10.0 31.0 362.17 MB

Documentation repository for the Egeria project.

Home Page: https://egeria-project.org

License: Other

CSS 49.16% HTML 50.13% JavaScript 0.72%
egeria documentation metadata open-source open-metadata metadata-management metadata-extraction metadata-api metadata-editor metadata-parser

egeria-docs's Introduction

License

README

This repository exists to host the documentation for the Egeria project across its various code repositories. The documentation itself should be accessed through GitHub pages at: https://egeria-project.org.

Contributing

This project welcomes contributors from any organization or background, provided they are willing to follow the simple processes outlined below, as well as adhere to the Code of Conduct. Review the community guide to find out more.

To review how to specifically contribute to the documentation, review our documentation guide.


License: CC BY 4.0, Copyright Contributors to the Egeria project.

egeria-docs's People

Contributors

adinh808 avatar adinhtdsibm avatar bogdan-sava avatar cjjaramillo avatar cmgrote avatar davidradl avatar dependabot[bot] avatar dwolfson avatar fedtried avatar ginaisaia avatar gnahz-eh avatar habiblawal1 avatar holyjak avatar jacobhudson avatar jmertic avatar jonahluckett avatar juergenhemelt avatar lcpopa avatar lpalashevski avatar mandy-chessell avatar marius-patrascu avatar mihaiiliescu avatar moudemans avatar omahs avatar planetf1 avatar popa-raluca avatar raunak-s avatar snk95 avatar t4ylan avatar wortelsoep avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

egeria-docs's Issues

Rework Key Concepts

  • "How are XXX modeled " - describing the difference between entity summary, entity detail and instance audit header. This should go in the metamodel section under references - it is only relevant to a developer using OMRS.
  • "A common tripping point for conformance" - this is too nuanced a concept for the overview page - it should be in the documentation for writing a repository connector.
  • #36
    • OMAG Server Platform
    • OMAG Servers
    • Concept description of each server (from Reference/Egeria Concepts) and to each of these pages add a link to the relevant adminstration page.
    • Metadata concepts
    • Other concepts under "Reference/Egeria Concepts" and "Reference/General Concepts" such as Anchors, GUIDs, Governance Zones etc.
  • The metadata instances section is totally OMRS centric. It is not true for the OMASs which is what most consumers will use. The OMASs use the term "metadata element" and there are no entities and relationships in their vocabulary. I am wondering if it does not belong under the OMRS or type system documentation. It can then be used by repository connector implementors

Egeria developer / Application developer - clarify docs

The egeria developer guide can be found fairly easy by navigating from the home page.

Currently we have 5 types of interfaces listed. However two link to the docs on writing a new connector, whilst the other 3 all link to the list of OMASs.

It would be helpful to

  • help an application developer find the java client libraries they need - both in source code and if writing an app using maven (pointing to a sample might also be helpful here)
  • help an app developer find our REST API docs - recognizing that a) we need to clarify if these are public/supported or not b) explain how to compose request bodies c) somehow link through to the openapi spec, but recognizing that is only deployed currently on a server once built
  • remove references to the kafka payloads, or document
  • point developers working on egeria back to the developer resources detail for more internal/standards based information -- a different perspective

See also odpi/egeria#3458
cc: @CDaRip2U

Broken links on https://odpi.github.io/egeria-docs/guides/documentation/guide/

The following links generate a 404 error:

Add initial documentation for deploying/using the Egeria UIs

Themain egeria docs need to be extended to cover running the Egeria UIs

  • Running the react UI natively
  • Running the react UI in a container (api server & static content)
  • Running the polymer UI natively
  • Running the polymer UI in a container (api server & static content)

Currently there is limited documentation

Capitalisation discussion

Overall the structure is looking excellent. Here are some specific comments ...

  • Under "Overview" -> "Our Solution"
    • "OMAG server" title should be "OMAG Server" - this is a reoccuring problem - OMAG Server Platform is consistent capitalized but "OMAG Server" is not - nor are the different types of OMAG Server. They are often referenced in lower case eg "Metadata access point" rather than "Metadata Access Point".
    • The description of the servers is not accurate... --> see #31
  • Under "Overview/Key Concepts" --> see #32
  • Should the road map and status (https://egeria.odpi.org/open-metadata-publication/website/roadmap/) be under "Overview" with links to releases ? --> see #33
  • Under "Using": --> see #34
  • Under "References/Services" --> see #35

Update postman docs to suggest importing direct from github

Having just setup a clean postman workspace today, the latest version (I was using 9.2.0) now allows import of collections direct from GitHub

This can be added as a suggested route for importing our collections.

  • Create a new workspace (suggested, to avoid clutter)
  • Import collection
  • select github
  • authenticate
  • select odpi / egeria / egeria-release-3.4 (or latest)
  • import all

Need to refine the docs to provide this as a preferred path, but also point out they can be done individually
Also review which collections should be imported.
Raising as a reminder to add this into Dojo docs

Rework References/Services sub-section

  • Helpful to group OMAS, OMIS, OMES, OMVS under a sub heading of "Registered Services" and the rest under "Fixed Services" to show which are optional in the platform and which are always present. I am assuming these pages will become these services "Home page" that the Egeria diagnostics -eg audit log messages - point to. These links are in the code in the component description enums.
  • Something weird with the Governance Zone page https://odpi.github.io/egeria-docs/concepts/governance-zone/ in that it does not have any menus

Mistake in Tutorial Docs

Hi there, on the following page: https://odpi.github.io/egeria-docs/getting-started/dojo/egeria-dojo-day-1-3-1-2-configuring-the-platform/ just under the 2nd Postman screenshot is the following text:

"If the baseURL variable is set to a different value to the server platform then Postman can not connect. In the screen capture below, you can see the baseURL is set to the default of https://localhost:9443 when it should be https://localhost:9443 because the platform is running in docker."

Both locations stated here are identical so one must be wrong. I also note that the URL referenced in the screenshot underneath shows the port number 8080 for the baseURL.

Looks like something is amiss here?

image

Add docs clarifying how to specify TLS configuration for docker images

We document how to configure TLS when running egeria at https://odpi.github.io/egeria-docs/guides/admin/configuring-the-omag-server-platform/?h=tls#brief-background-on-tls

However this only covers directly launching the chassis, and not when running in docker.

Our docker image for Egeria is based on Redhat UBI openjdk11 image.
Various environment variables can be set.

We currently set
ENV JAVA_OPTS_APPEND="-XX:MaxMetaspaceSize=1g"

to avoid some memory issues found with CTS, but more generally this environment variable allows other JVM options to be passed. This would include the ssl configuration referred to in the original link

The docs here should be extended to cover the docker image (there is currently none), and specifically how TLS can be configured when using the image

Further, this should be extended to our other docker images we build for connectors and UI (cc: @davidradl @sarbull @cmgrote @wbittles ) - some of these being based on other images

Also to note - as part of the work on the k8s operator, it too will be making TLS certs available via secrets, and in addition to documentation for the operator, it would be useful to document this technique for the benefit of those deploying with their own yaml/charts etc in k8s

Allow additional connectors to be loaded into container image at runtime

In our 'egeria-base' helm chart there is information on adding additional libraries (such as connectors) for deployment at https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/charts/egeria-base

This has been tested previously and allows additional libraries to effectively be added to LOADER_PATH in the deployment

However when testing odpi/egeria-database-connectors#73 this fails

For example

However this then fails with:

$ helm install base egeria-base                                                                                                                                                                                                                                     [12:57:55]
Error: create: failed to create: Request entity too large: limit is 3145728
$ ls -la egeria-base/libs                                                                                                                                                                                                                                           [12:58:13]
total 2056
drwxr-xr-x   4 jonesn  staff      128 15 Jun 12:57 .
drwxr-xr-x  10 jonesn  staff      320  9 Jun 11:08 ..
-rw-r--r--   1 jonesn  staff    43350 15 Jun 12:56 egeria-connector-postgres-2.10-SNAPSHOT.jar
-rw-r--r--@  1 jonesn  staff  1005332 15 Jun 12:57 postgresql-42.2.21.jar
jonesn:charts/ (master)

These libraries are being encoded in base64, and as such will take more space than the binaries, quite likely exceeding the number given, which is the max size of a kubernetes API request...

This approach therefore needs rethinking. One viable alternative is to build a customized docker image with the required connector (+egeria) - indeed we already do this for the current SAS Viya connector at https://github.com/odpi/egeria-sas-viya-connector
Need to consider other options, but in either case it seems the current documented approach will not work well for k8s as this 3MB limit is very small (but was enough to pass my testing ....!)

Automatic generation/assistance making release notes

Our current release notes are fairly brief, and focus on the main new features or significant changes in a release.

There is a variety of information that would be useful to include such as

  • dependency changes (mostly dependabot originated)
  • Enhancements
  • list of commits
  • API changes
  • type changes

In some cases we may wish to include this, in other cases point to a query

However to generate these automatically (at least as a draft to cut/paste) we need a more rigid process - documentation and enforcement through actions ie to

  • be able to easily list the PRs merged into a particular release
  • be able to consistently link back from a PR and/or commits to the relevant issues
  • more consistent and enforced tagging of issues and PRs (ie bugfix vs security vs enhancement)
  • Adding ability to pull in a short abstract from a PR/issue relevant to release notes
  • filtering - tech preview vs release vs development
  • .....

Opening as a future item to look at ....

Document how to debug REST request/response

When debugging, it can be useful to log both requests/response made to the server (within the client, 3rd party app, or chassis) as well as the server chassis's inbound requests & response sent.

From a client perspective there are options such as using RestTemplate logging ( spring property logging.level.org.springframework.web.client.RestTemplate=DEBUG )

For the server, we run tomcat, and access logs can be enabled as per https://howtodoinjava.com/spring-boot2/logging/embedded-server-logging-config . There are also other options such as enabling the spring logging actuator, or a logging filter that may be clearer.

As a first step we should document suggested current logging mechanisms, and if insufficient look to enhance that capability in the code

(Originated from a user's query)

Outline of what content should be in what GitHub repository (ultimately)

To capture an overall strategic outline of what repositories we should have, and what content should be in each. Should form a "roadmap" that we gradually drive towards with separating out the core repository content + provide initial documentation for what all the repositories are, why the exist, and where to go for what.

migrate data governance repository content

We have moved the webinar content from the data governance repository. We still point to workshops and presentations into the data governance site from the current repository. I suggest moving this content from the data governance site here. Also we agreed in a community meeting that the governance principles fro the data governance git repo should be moved into this repository, as they then would be searchable with the rest if the content.

Rework introduction page

The description of the servers is not accurate - probably comes from an out-of-date file. The platform implements the
services and the servers are collections of activated services that host connectors to the different technologies that egeria is exchanging metadata with. Then take people to the key concepts to explain more detail rather than sending them off to look at the deep implementation details in terms of connectors, services and frameworks.

Dark mode of website not rendering nicely on main page

When swapping to dark mode for the main page of the MkDocs rendered site, the colours of the content (and the "hero" image) are not inverted to be readable in dark mode. All other pages seem to work fine in dark mode.

Add link to further docs on Redhat UBI-8 openjdk-11 container

When using the docker container for egeria server-chassis (and ui-chassis), sometimes additional parameters need to be passed.

Since our container is based on Redhat UBI8 openjdk-11 image, this capability is already part of the image.

Some older (v3) docs are at Our container image is based on Redhat UBI8 openjdk-11 . There are some docs at
https://access.redhat.com/documentation/en-us/red_hat_jboss_middleware_for_openshift/3/html/red_hat_java_s2i_for_openshift/index

This covers the use of additional parms such as JAVA_ARGS which can add additional parameters to the invocation of the selected executable. for example in the case of the ui-chassis this could include cors settings.

Will add to docs...

Add documentation on GitHub repositories

It would be useful to capture key information about the GitHub repositories we use for Egeria.
This is primarily for developers.
The information could contain

  • Name of repo
  • Link to repo
  • Owner(s)/Admin [github ids]
  • Description/purpose - what this repo is all about
  • Dependencies on other egeria repos (compile, test, runtime)
  • Other repos dependent on this (could omit as duplicate of above and more to maintain)
  • What is provided by repo - build artifacts/attachment, raw jar/lib from build, maven packages (where), docker images, charts etc [need to dig into this categorization further], github releases. include links
  • Where issues should be raised (link) - could be main egeria, or actual repo (more likely)
  • Languages / frameworks in use
  • ci/cd methodology/schedule
  • release strategy/policy
  • security stance/scans

An alternative option is to keep the link in the docs light, with only a repo name/link, and then point to a well defined .md file within each repository. This should be of a mandated format, but is then owned by the linked project. we loose searchability, and it is dependent on each repo making the change. It also may make it tricker to identify inconsistencies

We could model this in egeria.....

Rework Using section

  • The developer guide needs to move under "Using" since it is for developers using Egeria - rather than Egeria contributors
  • The link in "Build a connector" should point to the 'What is a connector" page in the developer guide.
  • The connector catalog should be here too rather than in its own section since choosing connectors is a key part of planning and administering. There are missing links for the runtime connectors.
  • "Managing external identifiers" needs to be grouped with the descriptions of modelling types, deduplication, lineage etc. Grouped with concepts such as Anchors, Lineage etc on the Overview/Key Concepts page
  • The CTS should be here under Using rather than in the Reference section - or part of the developer guide?

Key Concepts navigation / structuring

Under the Key Concepts page, the 4 types of governance server should be listed since all of the other types of server are shown. Could also show them in the inheritance hierarchy structure. I would also suggest that nested under Key concepts, there should be:

  • OMAG Server Platform
  • OMAG Servers
  • Concept description of each server (from Reference/Egeria Concepts) and to each of these pages add a link to the relevant adminstration page.
  • Metadata concepts
  • Other concepts under "Reference/Egeria Concepts" and "Reference/General Concepts" such as Anchors, GUIDs, Governance Zones etc.

Improve clarity of k8s instructions for Windows

The instructions for setting up microk8s on windows at https://odpi.github.io/egeria-docs/guides/operations/kubernetes/k8s/ refer the user to the official windows instructions.

Whilst that's initially appropriate -- to download and run the GUI installer, it then needs to be more explicit.

Following install/reboot the same instructions are used as for macOS (so this can be common) ie

Microsoft Windows [Version 10.0.19044.1348]

nigel@MEGA C:\Users\nigel
$ microk8s start
Started.

nigel@MEGA C:\Users\nigel
$ microk8s status --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    ha-cluster           # Configure high availability on the current node
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    ingress              # Ingress controller for external access
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    kata                 # Kata Containers is a secure runtime with lightweight VMS
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    openebs              # OpenEBS is the open-source storage solution for Kubernetes
    openfaas             # openfaas serverless framework
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    storage              # Storage class; allocates storage from host directory
    traefik              # traefik Ingress controller for external access

nigel@MEGA C:\Users\nigel
$ microk8s enable dns storage helm3
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Enabling Helm 3
Fetching helm version v3.5.0.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.7M  100 11.7M    0     0  4935k      0  0:00:02  0:00:02 --:--:-- 4935k
Helm 3 is enabled

nigel@MEGA C:\Users\nigel
$ microk8s kubectl get all --all-namespaces
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-kube-controllers-578cfb99d5-zd9px   1/1     Running   0          5m17s
kube-system   pod/coredns-7f9c69c78c-2pqxx                   1/1     Running   0          29s
kube-system   pod/calico-node-cxpfc                          1/1     Running   0          5m17s

NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.152.183.1    <none>        443/TCP                  5m24s
kube-system   service/kube-dns     ClusterIP   10.152.183.10   <none>        53/UDP,53/TCP,9153/TCP   29s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   5m24s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           5m24s
kube-system   deployment.apps/coredns                   0/1     1            0           29s
kube-system   deployment.apps/hostpath-provisioner      0/1     0            0           18s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-578cfb99d5   1         1         1       5m17s
kube-system   replicaset.apps/coredns-7f9c69c78c                   1         1         0       29s

nigel@MEGA C:\Users\nigel
$

A few other observations:

  • It's also worth adding a caveat to docker desktop & make it the last item in the list (including a possible note that the egeria team cannot help)
  • typo 'Furether information'
  • Moving the microk8s example to a separate page, and adding in rancher desktop on the summary may add clarity - distinguishing between the general principles & choice of runtimes vs one we picked to be very explicit about to help new starters in k8s
  • Under 'accessing applications' we could add an explicit example of accessing microk8s ports locally (just as we need in the dojo) - be it via port forwarwarding, nodeport etc. Where possible this should link into the microk8s docs

ON Helm....

A caveat is noted at https://odpi.github.io/egeria-docs/guides/operations/kubernetes/helm/ that the user needs to use 'microk8s.kubectl' or 'microk8s.helm'. This is non-standard. For the dojo we will explicitly use 'microk8s helm3 ....' but some users may still be confused by these joint instructions. An adominition, perhaps offering the variant may help as otherwise the user may pickup an old version or wrong version of helm, and for those new to k8s this could be confusing

And testing our charts - the instructions working ok on x86 windows 10


nigel@MEGA C:\Users\nigel
$ microk8s helm3 repo add egeria https://odpi.github.io/egeria-charts
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
"egeria" has been added to your repositories

nigel@MEGA C:\Users\nigel
$ microk8s helm3 search repo egeria
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
egeria/egeria-base      3.3.0           3.3             Egeria simple deployment to Kubernetes
egeria/egeria-cts       3.3.0           3.3             Egeria Conformance Test Suite deployment to Kub...
egeria/egeria-pts       3.3.1           3.3             Egeria Performance Test Suite deployment to Kub...
egeria/odpi-egeria-lab  3.3.0           3.3             Egeria lab environment

nigel@MEGA C:\Users\nigel
$ microk8s helm3 install lab egeria/odpi-egeria-lab
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2551/credentials/client.config
NAME: lab
LAST DEPLOYED: Thu Nov 11 12:16:33 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
ODPi Egeria lab tutorial
---

The Egeria tutorials have now been deployed to Kubernetes.
It may take a minute or so for everything to start up.

Open your web browser and go to addressofmycluster:30888 to
get started

You may need to contact your cluster admin, or read your cloud provider helptext to understand
the correct 'addressofmycluster' to use.

If you experience problems, check memory consumption on your nodes. A minimum of
a 3 node cluster, 2GB per node; or a desktop environment with 6GB dedicated is recommended.

Please provide any feeback via a github issue at https://github.com/odpi/egeria or
join us on slack via https://http://slack.lfai.foundation

- The ODPi Egeria team

nigel@MEGA C:\Users\nigel
$

Migrate Types documentation

There's a reasonable chunk of work to do regarding migrating the Types documentation. The tasks would involve:

  • Saving the individual diagrams in each page as a separate SVG file (per documentation guide's instructions)
  • Updating the various pages to point to these SVG files instead of the PNG files
  • Ensuring that every type (entity, relationship or classification) being explained on each page has its own heading (this allows us to direct-link to the explanation of an individual type via anchors):
    • the top-level types for that page should be second-level headings (##)
    • any subtypes of those types can be third-level headings (###)

Naturally follow the documentation guide as part of the updates (remove the license footers from each page, retain the license headers, ensure appropriate use of capitalization vs italics and bold, etc).

The base model (area 0) has been migrated already according to these principles as a starting point -- only 6 more areas to go! ๐Ÿ˜ฌ

(For reference, here's the "example of 'good'" in the repo itself: https://github.com/odpi/egeria-docs/tree/main/site/docs/types/0, with this being the output: https://odpi.github.io/egeria-docs/types/0/0010-base-model/)

installing egeria tutorial docs wrong - results in tls problem

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Getting error when running OMAG server
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java -Dserver.port=9443 -jar server-chassis-spring-3.4-SNAPSHOT.jar
Project Egeria - Open Metadata and Governance
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / / / / ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ / ____ _ _
/ / / // /|
/ // /| | / / __ _
\ / _ \ / __/| | / // _ \ / __/ / /
/ // // | / \ / / / | / // || |
/ /
/ // / / // ___ |/ /
/ / / // _// / | |/ // // / / __ // // / \ / / / // / // / / / / /
_
//
/ /
//
/ |
|_/ // ___/// |/ _/// // // _////// _/// // /_/

:: Powered by Spring Boot (v2.5.6) ::

2021-11-19 15:28:37.507 INFO 1380 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9443 (https)
2021-11-19 15:28:43.048 ERROR 1380 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.12.jar!/:5.3.12]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.12.jar!/:5.3.12]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.odpi.openmetadata.serverchassis.springboot.OMAGServerPlatform.main(OMAGServerPlatform.java:95) ~[classes!/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467) ~[server-chassis-spring-3.4-SNAPSHOT.jar:na]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:229) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:43) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.12.jar!/:5.3.12]
... 22 common frames omitted
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:238) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:282) ~[spring-boot-2.5.6.jar!/:2.5.6]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:213) ~[spring-boot-2.5.6.jar!/:2.5.6]
... 24 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1075) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:234) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 26 common frames omitted
Caused by: java.lang.IllegalArgumentException: /home/dimikhan/egeria-install/egeria-omag-3.4-SNAPSHOT/server/truststore.p12 (No such file or directory)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:231) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1208) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1294) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:614) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1072) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 28 common frames omitted
Caused by: java.io.FileNotFoundException: /home/dimikhan/egeria-install/egeria-omag-3.4-SNAPSHOT/server/truststore.p12 (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method) ~[na:na]
at java.base/java.io.FileInputStream.open(FileInputStream.java:219) ~[na:na]
at java.base/java.io.FileInputStream.(FileInputStream.java:157) ~[na:na]
at java.base/java.io.FileInputStream.(FileInputStream.java:112) ~[na:na]
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86) ~[na:na]
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184) ~[na:na]
at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:197) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLHostConfig.getTruststore(SSLHostConfig.java:730) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:422) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:245) ~[tomcat-embed-core-9.0.54.jar!/:na]
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97) ~[tomcat-embed-core-9.0.54.jar!/:na]
... 34 common frames omitted

Expected Behavior

As per - https://odpi.github.io/egeria-docs/education/tutorials/omag-server-tutorial/task-starting-the-omag-server-platform/ the OMAG server will be started.

Steps To Reproduce

Following page - https://odpi.github.io/egeria-docs/education/tutorials/installing-egeria-tutorial/
Using Windows 10 with WSL2 - Linux version Ubuntu 20.04 LTS.

Could able execute git clone https://github.com/odpi/egeria.git and run mvn clean install. But when checking egeria-install/egeria-omag-3.4-SNAPSHOT/server path I dont see the same directory structure as mentioned in https://odpi.github.io/egeria-docs/education/tutorials/installing-egeria-tutorial/ page. Below is the directory structure.
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ ls lib server-chassis-spring-3.4-SNAPSHOT.jar

However, finally tried to start the server as below -
'dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java -Dserver.port=9443 -jar server-chassis-spring-3.4-SNAPSHOT.jar'

Environment

- Egeria: 3.4
- OS: Windows 10 WSL2 Ubuntu 10.04 TLS
- Java: 
dimikhan@DESKTOP-2VO8QEQ:~/egeria-install/egeria-omag-3.4-SNAPSHOT/server$ java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
- Browser (for UI issues): N/A
- Additional connectors and integration: N/A

Any Further Information?

No response

Clarify behaviour and ensure documentation for startup behaviour (kafka)

If Kafka is down, the default kafka connector will try for around 10 times, to ensure the cluster is up. It does this by attempting to get a non-null list of available brokers via an AdminClient.

There are docs on this behaviour at: https://github.com/odpi/egeria/tree/master/open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector

These docs need to be incorporated in the new user docs on egeria-docs
The default also appears to be 60s, not 5s as the doc above states (at least in the coco environment, and I cannot see an override) ie

on Sep 13 12:46:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 2
Mon Sep 13 12:47:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 3
Mon Sep 13 12:48:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 4

Some servers are also members of multiple cohorts. In this case the retry loop is repeated per cohort. For example cocoMDS2 tries cocoCohort then devCohort. After cocoCohort fails, we report an exception saying initialization failed. However the code continues serially with the next cohort (this is all whilst the /instance server start operation is being executed.. and which is in danger of timing out at the http request level)

Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Exception OMRS-AUDIT-0006 Configuration error detected while connecting to cohort devCohort, exception org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException was caught with error message: OCF-KAFKA-TOPIC-CONNECTOR-400-002  Egeria was unable to initialize a connection to a Kafka cluster.  The message in the exception was: java.util.concurrent.ExecutionException
Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Exception OMRS-AUDIT-0006 Supplementary information: log record id cb3bac6d-6eff-4661-b750-c70394fa0d76 org.odpi.openmetadata.frameworks.connectors.ffdc.ConnectorCheckedException returned message of OCF-KAFKA-TOPIC-CONNECTOR-400-002  Egeria was unable to initialize a connection to a Kafka cluster.  The message in the exception was: java.util.concurrent.ExecutionException and stacktrace of
OCFCheckedExceptionBase{reportedHTTPCode=400, reportingClassName='org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector', reportingActionDescription='KafkaMonitor.waitForThisBroker', reportedErrorMessage='OCF-KAFKA-TOPIC-CONNECTOR-400-002  Egeria was unable to initialize a connection to a Kafka cluster.  The message in the exception was: java.util.concurrent.ExecutionException', reportedErrorMessageId='OCF-KAFKA-TOPIC-CONNECTOR-400-002 ', reportedErrorMessageParameters=[java.util.concurrent.ExecutionException, egeria.omag.openmetadata.repositoryservices.cohort.devCohort.OMRSTopic.registration, org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s)], reportedSystemAction='The system is unable initialize.', reportedUserAction='Ensure that Kafka is available', reportedCaughtException=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s), reportedCaughtExceptionClassName='java.util.concurrent.ExecutionException', relatedProperties=null}
	at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector.start(KafkaOpenMetadataTopicConnector.java:290)
	at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.start(OMRSTopicConnector.java:258)
	at org.odpi.openmetadata.repositoryservices.metadatahighway.OMRSCohortManager.setSecurityVerifier(OMRSCohortManager.java:347)
	at org.odpi.openmetadata.repositoryservices.metadatahighway.OMRSMetadataHighwayManager.setSecurityVerifier(OMRSMetadataHighwayManager.java:128)
	at org.odpi.openmetadata.repositoryservices.admin.OMRSOperationalServices.setSecurityVerifier(OMRSOperationalServices.java:850)
	at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector$KafkaStatusChecker.waitForBrokers(KafkaOpenMetadataTopicConnector.java:502)
	at org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicConnector.start(KafkaOpenMetadataTopicConnector.java:272)
	... 61 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1631533556915, tries=1, nextAllowedTryMs=1631533557016) timed out at 1631533556916 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

Mon Sep 13 12:45:56 BST 2021 cocoMDS2 Cohort OMRS-AUDIT-0060 Registering with open metadata repository cohort devCohort using metadata collection id 9614d9b7-d223-4a8c-926d-23ce286c477c
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Cohort OMRS-AUDIT-0062 Requesting registration information from other members of the open metadata repository cohort devCohort
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0031 The iotCohort cohort inbound event manager is starting with 2 type definition event consumer(s) and 2 instance event consumer(s)
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0026 Initializing listener for cohort iotCohort
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0019 An OMRS Topic Connector has registered with an event bus connector for topic egeria.omag.openmetadata.repositoryservices.cohort.iotCohort.OMRSTopic.registration
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0001 Connecting to Apache Kafka Topic egeria.omag.openmetadata.repositoryservices.cohort.iotCohort.OMRSTopic.registration with a server identifier of 89af21fe-e3fa-4dfe-a307-bfe782616006
Mon Sep 13 12:45:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 1
[I 2021-09-13 12:46:05.873 ServerApp] Saving file at /common/environment-check.ipynb
Mon Sep 13 12:46:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 2

The retry loop in logging an audit message only reports:

Mon Sep 13 12:34:56 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 10

It may be useful to enhance this with

  • what part of server initialization - cohort, which cohort - or topic?
  • How many retries in total and/or interval

As this will help someone monitoring the system and understanding what is going wrong.

cocoMDS2 continues to do the same with iotCohort

After cohort initialization is complete we continue with OMASs ie:

Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Connected Asset Services
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Connected Asset Services has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-METADATA-MANAGEMENT-0001 The Open Connector Framework (OCF) Metadata Management Service is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-METADATA-MANAGEMENT-0003 The Open Connector Framework (OCF) Metadata Management Service has initialized a new instance for server cocoMDS2
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAG-ADMIN-0010 The Open Metadata Access Services (OMASs) are starting
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Asset Catalog OMAS
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Asset Catalog OMAS has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CATALOG-0002 The Asset Catalog Open Metadata Access Service (OMAS) is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CATALOG-0001 The Asset Catalog Open Metadata Access Service (OMAS) has initialized a new instance for server cocoMDS2
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0040 An enterprise OMRS connector has been created for the Asset Consumer OMAS
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMRS-AUDIT-0041 The enterprise OMRS connector for the Asset Consumer OMAS has started
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OMAS-ASSET-CONSUMER-0001 The Asset Consumer Open Metadata Access Service (OMAS) is initializing a new server instance
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0001 Connecting to Apache Kafka Topic egeria.omag.server.cocoMDS2.omas.assetconsumer.outTopic with a server identifier of 89af21fe-e3fa-4dfe-a307-bfe782616006
Mon Sep 13 12:55:57 BST 2021 cocoMDS2 Startup OCF-KAFKA-TOPIC-CONNECTOR-0015 The local server is attempting to connect to Kafka, attempt 1

After the failure of asset catalog shutdown then starts:

Mon Sep 13 13:05:58 BST 2021 cocoMDS2 Shutdown OMAG-ADMIN-0005 The cocoMDS2 server has begun the shutdown process

So currently the behaviour is

  • Try and start all cohorts - do not immediately fail
  • Try and start all OMASs - immediately fail

The intent needs clarification and documentation.

  • Do we stop after the first timeout is hit - I think this is most likely correct. Kafka not there - we exit after configured retries.
  • Do we continue to go through each cohort? ie to allow cohort init to continue in case kafka comes up later - this could be risky, though it's a possibility. For this approach moving the retry loop right into the kafka topic connector consumer/producer threads with a quieter logging approach would simplify the overall logic (see also below re: topics)
  • We continue with the existing design - but need to clearly document

It's worth noting that broker availability is one possible check, but there are other failures. For example a topic may not exist - this may be short lived as it will get created by producers, but is still something that could continue indefinately for some topics. Further, if auto topic creation is disabled it will never exist without admin action. This is already handled within the producer/consumer directly and events are already queued to be sent.

Update macos k8s docs with reference to apple silicon / m1

Not all k8s implementations available for macOS work on the latest m1/ARM based macbooks

This can cause confusion for anyone trying to use our helm charts .

Though fundamentally egeria is agnostic to k8s implementation and platform, we are also trying to engage new users & help them learn about egeria, and a working k8s implementation is a crucial part of that.

We need to consider documenting an appropriate route, and add caveats so that users don't waste their time.

Clarify use of serverURLroot

At https://odpi.github.io/egeria-docs/guides/admin/servers/configuring-a-metadata-access-point/#set-the-server-url-root we explain serverURL root

I have had a few people discuss this on slack & get confused when running in a container or complex/vpn environment

We need to clarify that this address is used for other servers in the cohort to call this server's REST endpoints - or at least that it is the default for that. Effectively it is how this cohort member advertizes its services to other members

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.