dedica-team / nivio

Management of application landscapes for teams

Nivio

Nivio is an application landscape management for teams (developers, operators, managers). It follows a bottom-up no-op approach, i.e. there is no interface for manual data maintenance. It is designed to gather the application landscape information from configurable items, preferably code repositories.

Getting Started

Online Demo

→ Try out the Pet Clinic demo

Running locally in Docker

You need docker to run the demo:

docker run -e DEMO=1 -p 8080:8080 dedica/nivio

Point your browser to http://localhost:8080/#/landscape/petclinic to explore the features. An OpenAPI documentation is available at http://localhost:8080/swagger-ui.html.

Further Reading

• We recommend reading the User Manual for configuration options.
• Refer to the developer docs if you want to work on the source code.

Copyright & Trademark Acknowledgment Statements

Nivio uses the Material Design Icons released under the MIT / Apache 2.0 licenses.

The nivio logo has been designed by Alfred Rehbach

nivio's Issues

Service description templates

Service description templates could be used to prefill service descriptions.

• avoids duplicate and repetitive data entry in descriptions
• some formats like docker compose file are hard to extend with information such as service owners

Expected behavior:
Service descriptions are prefilled with template values if a template is referenced. Values derived from the service description overwrite template values.

Possible solution:
Configure several templates and reference them in source references (similar to "autoGroup" or "format"). Process templates first, copy values to service descriptions.

Side note: AutoGroup might be replaced by templates.

Get rid of several type casts by improving the type definition

You might be able to get rid of several type casts by improving the type definition here.

E.g. something like:

    private static <T> List<T> findAll(FullyQualifiedIdentifier fqi, Collection<T extends LandscapeItem> items)

Originally posted by @Matthimatiker in #39

This also applies to several other classes, especially the interface implementations ("description" vs objects used in memory).

Development Setup instructions missing path argument for docker build command

In the Development Setup instructions the command docker build -t nivio:latest needs an additional path argument .:

$ docker build -t nivio:latest
"docker build" requires exactly 1 argument.
See 'docker build --help'.

Usage:  docker build [OPTIONS] PATH | URL | -

Build an image from a Dockerfile

vs.

$ docker build -t nivio:latest .
Sending build context to Docker daemon  89.03MB
Step 1/4 : FROM openjdk:11-jre
 ---> d0f756e22061
Step 2/4 : VOLUME /tmp
 ---> Using cache
 ---> 35db91450eb6
Step 3/4 : ADD target/nivio-0.1.0.jar app.jar
 ---> Using cache
 ---> a99db397926f
Step 4/4 : ENTRYPOINT ["java","-jar","/app.jar"]
 ---> Using cache
 ---> 7f5eaa888e8c
Successfully built 7f5eaa888e8c
Successfully tagged nivio:latest

Dashboard

Create a dashboard GUI with

1. overview and all landscapes with states, warnings, log error (whatever is available)
2. free floating zoom/pan navigable landscape map (already exists, needs to be extended later)

Use React and Material-UI (https://material-ui.com/)

Rancher API: Investigate why items can end in "common" group

Rancher contains only named stacks (?) which are transformed to groups. So there should not be any running (check!) service witout group.

report.html output requests additional files from localhost using absolute paths

I have just noticed that the server started by nivio is accessible from the entire local network. So far so good. However, the information in report.html is not displayed correctly, because it seems that absolute paths are used for additional files. CSS and images are requested by localhost, but are of course not available under localhost on a remote computer.

Regardless of whether the page is to be accessible from the entire network or only from the localhost, relative paths should be used.

Data completeness evaluation

As a user I want to get help how the my data can be improved.

• #690
• #692
• #691
• evaluate if the assessment can run async, i.e. later after processing and then be added to the other assessments

Example Item completeness:

• contact or owner
• team or contributors
• description or 1 link

Document journey of first use

Review the documentation, what is missing? What can be done to improve the experience of first use?

landscape with four items flickers

Reloading the graph.png renders different every time for 4 groups.

identifier: AdventOfCode
name: Advent Of Code App
contact: [email protected]


# Landscape configuration
config:
  groups:
    input:
      color: "0c8e3e"
    Frontend:
      color: "24a0ed"

  # https://jgraph.github.io/mxgraph/java/docs/com/mxgraph/layout/mxOrganicLayout.html
  jgraphx:
 #   triesPerCell: 8
 #   edgeLengthCostFactor: 0.0001
    nodeDistributionCostFactor: 900000.0
 #   borderLineCostFactor: 7.0

    #
    # for group alignment
    #

    # the higher, the longer the edges between groups
    forceConstantFactor: 4.0

    # higher value is cpu intensive, but can lead to better layouts
   # maxIterations: 1000

items:

  - identifier: user
    name: User
    type: humanuser
    group: input
    dataFlow:
      - target: fe-react
        description: User inputs data into React Frontend

  - identifier: fe-react
    name: React
    group: Frontend
    dataflow:
      - target: be-controller
        description: Using a REST-API, the data is sent to the backend

  - identifier: be-controller
    name: Controller
    group: Backend
    dataflow:
      - target: be-services
        description: Sends puzzle input to services
      - target: fe-react
        description: Sends puzzle output to frontent

  - identifier: be-services
    name: Services
    group: Backend
    dataflow:
      - target: be-persistence
        description: Sends puzzle results to the persistence layer
      - target: be-controller
        description: Sends puzzle results to the controller


  - identifier: be-persistence
    name: Persistence
    group: Backend
    dataflow:
      - target: postgres
        description: Data is stored in a PostgreSQL DB
      - target: be-service
        description: Returns retrieved data

  - identifier: postgres
    name: PostgreSQL DB
    group: Persistence
    type: database
    lifecycle: planned

Investigate interactive graph visualization as output

The architect loves to drag things around.

• https://github.com/neo4j-contrib/neovis.js
• https://orientdb.com/docs/2.0/orientdb-studio.wiki/Graph-Editor.html

Add plugin structure

Change the code (Osgi, Java modules?) to allow extended functionality at deploy/runtime.

For example the dashboard or other types of rendering could be added this way.

Examine https://github.com/pf4j/pf4j and https://github.com/spring-projects/spring-plugin

Improve short distance data flow bezier curves

x/y shift of the control point (current implementation) results in badly shaped curves for long distances. A better solution might be constructing an orthogonal vector on the edge between two services and using its end as control point.

Verify that docker image "latest" is based on develop branch only

GraphStream graph not rendered nicely with unconnected parts

If there is more than one graph or unconnected nodes the rendered output is ugly.

Yaml configuration autocomplete and suggestions

Provide support when writing nivio yaml configuration files by suggesting config entries and/or validating values. Possible solutions:

• MS Visual Studio Plugin
• Intellij Plugin
• Browser based YAML editor https://github.com/jaywcjlove/react-monacoeditor

• requires a schema of possible elements

Configurable greediness/gracefulness

Currently unknown Relation (dataflow, Provider) targets are ignored. The Landscape config should allow to add unknown Elements in the fly instead of ignoring them.

Affects:

• relations
• Machines?

• change default behaviour to be greedy, i.e. create everything instead of rejecting items

Source folder littered with files after docker run

I have been running Nivio via the "development setup" instructions and noticed that the source directory gets littered with image files and folders. These files remain even after terminating docker with ctrl+c (is this even the correct way?).

The image file names consist of the landscape name, followed by a random number of different lengths, followed by png, but with no extension. A new image file is created for every time, graph.png is requested.

The folders are named tomcat. followed by a random number, followed by a period and the port number. It contains the folder structure ./work/Tomcat/localhost/ROOT, but no files.

Folders named tomcat-docbase. followed by a random number, followed by a period and the port number are deleted after terminating docker.

Turn source references into string-string maps

The current implementation of references to sources (nivo, k8s, docker files) has a few fields like format, url, auth credentials. It might be more flexible to use flat string maps and have the implementations check the presence of keys and correct values. This way new fields in new format implementations do not affect existing impls.

Use error trackers for service status

In order to have just in time information on service "stability" (status), a status provider should be able to read from error trackers such as sentry.

Review service deletion

With several sources/source references describing the same services, review if/how services can be deleted from a landscape.

Reduce dataflow and providers to a single connection type

For automatic scanning of relations between landscape items it will often not be possible to find out the difference between provider connection and data flow. In this case it were better to be able to set a connection as-is, and maybe addeding some attribute to express the kind of connection.

Possible solution: replace dataflow and providers with "relation" and relation.type

Investigate graph DB inclusion

• allows to use all graph db features on a landscape

• https://neo4j.com/docs/java-reference/current/tutorials-java-embedded/include-neo4j/

• https://orientdb.com/docs/last/Memory-storage.html

Linked example image missing in documentation

The example image on the documentation page features seems to be missing. Chrome gives a 404 and firefox does not load "due to errors in the file".

Improve API with HAL / Hateoas

Enrich returned objects with Links, introduce more DTOs

Remove state providers as separate mechanic

State providers currently work as additional way to gather information, only as additive component, not as primary source of information.

Remove prometheus state provide and add it back as source format, i.e. creating landscape items on its own.

Meta nivio

Provide a Landscape of landscapes, e.g. when different departments Run their own nivio instance and an ivory tower instance collects Info from these Department instances.

Show scaling in rendered graph

Eg small circle with scale (if integer)

Blacklist stacks in (rancher) landscape

The architect may not want to see "infra" stacks containing rancher-internal stuff.

Scenarios

Provide a means to visualize different scenarios, like hiding end-of-life apps and showing planned apps if requested.

Possible solution: Filter rendering using service tags (negate/exclude) and a new property "scenarios", which works like tags, but hides a service by default if one or more scenarios are set.

relates to #98

Investigate kiali

https://github.com/kiali/kiali

Investigate Nivio as native image

At the moment Nivio needs Java or Docker to run. Complicated setups could be avoided if Nivio would provide native executables. Micronaut and GraalVM is an option here, because Micronaut is similar to Spring.

How much effort is needed to change the code?
Can GitHub compile GraalVM stuff?

Add sankey-like display of aggregating labels value

As a user I want to see where in my processes (#762) values appear and how the values contribute to a total. A sankey https://en.wikipedia.org/wiki/Sankey_diagram like visualisation of a process could be used to achieve this

Examples:

• running costs
• CO2 / pollution

Investigate whey large landscape with lots of unconnected groups renders strangely

Groups are distributed unevenly across image.

In comparison, items inside groups that are unconnected are distributed quite evenly.

Review aliases as Idea for connection matching

In order to find relations (#38) between items it might be useful to use more than just the identifier. Relation endpoints could be matched against aliases Like URLs or host names.

Support Interactive Application/Microservice Landscape Planning

Provide a means for quick and easy sketching of a landscape. For instance, automatically create and link services gracefully from unmatched provider/dataflow references.

JSON schema based forms provide an editor almost instantly. This generated JSON/YAML can be sent to the API, where a landscape is created or updated and then rendered instantly.

• integrate json schema based editor, optimize UI
• convert json to yaml and sent to API
• optional: allow sending JSON to API (makes sense?)
• create/update landscape
• integrate rendered svg as preview

Automatically generate and send PDF landscape reports for owners

Service owner should receive an up to date report regularly if wanted. Maybe introduce flag.

• pdf ?
• powerpoint ?

Status icons

Show UTF-8 Signs or favicons for service statuses instead of (or plus) abbreviations like "HEA" for health in landscape graph.

Find/Query by name

Relates to #41

In docker labels and rancher, an user might want to reference relation targets by their name, not by an internally created id (e.g. rancher service id).

Investigate service meshes, api gateways and proxies as information source

BLOCKED because the items below have low priority.

Find out how service meshes, api gateways and traffic proxies could be used to obtain landscape items and relations.

• nginx
• haproxy
• varnish
• ....

Relations in report showing "null"

Relations are displayed as "null"

Escape landscape identifier in landscape rendered graph file name

A landscape with identifier like "a:b" leads to an exception because of an invalid filename on Windows. Illegal chars need to be escaped.

Review ALM Paper

Read https://www.bptrends.com/publicationfiles/03-05-2013-ART-An%20Approach%20to%20Managing%20Appliction%20Landscape-Paradkar%20&_%20Kulkami%2027th%20Feb%202013.pdf and find low hanging fruits to get into low-fi EAM

Writing a lot png images to seed file containing folder and trying to read them

When refreshing through the Browser on e.g. http://localhost:8089/render/nivio:example/graph.png, nivio tries to parse the other files contained in the directory:

Maybe it should write them to a different directory or not try to parse them?

Add github actions

Build and test using github, publish/label test results if possible

Add kubernetes as information provider

The next step is to verify nivio works with k8s. How it is started is documented.

• take your time and read the fine manual (k8s)
• create a local cluster (e.g. using minikube) and load some apps using https://helm.sh/ and see what output is generated and what is missing, open tickets or fix yourself
• create a local cluster (e.g. using minikube) and load some services manually and see what output is generated and what is missing, open tickets or fix yourself

Read Rancher 1.6 API (like k8s)

• Read rancher 1.6 API
• index all services with labels
• use labels starting with nivio as fields

https://github.com/objectpartners/rancher-java-sdk

Group color always black in report

Infinite recursion in JSON relations

Relations contain source and target as objects. Instead endpoints should be referenced by their identifier or FQI

Seed url contains authority component

When SEED contains a Windows file location like "c:\abc.yml" it is not recognized as correct URI. Either catch the exception or correct the URL.

Consider removing JPA persistence

All graphs could be kept in memory without losing too much, except for state. The code would become a lot less complex.

• consider using cqengine to allow querying instead of built-in lists for service repo