Update admin documentation about dtaas

[prasadtalasila]

The users are need to run it from top-level. Please see this README.

source deploy/install.sh

The installation steps require strict sequence of steps. We will improve upon them to automate most of these steps in near future.

[nichlaes]

9. Update the servers/config/gateway/README.md for traefik-gateway launch command to use auth.

Updated on branch 96-update-admin-documentation

[prasadtalasila]

The Ubuntu desktop install is a good inspiration for structuring the install documentation.

The openai cookbook indexing and linking is a good structure to have. The linked pages are to be relative links pointing to our mkdocs pages. Another example is Devcookbook.

[nichlaes]

Make sure that all the code in deploy/ and script/ have been correctly ported from release-v0.2 branch to feature/distributed-demo. (cross-check PR #94)

a install script is located under /deploy/install.sh but the script uses scripts from the folder /scripts/. e.g.

bash script/env.sh || exit

Won't this be a problem, if you run it from the /deploy/ folder?

[nichlaes]

@prasadtalasila

Comments:

Admin section:

1. Add a general purpose section
   A section which shortly describes the role as an admin for the system and what the goal is. What am I installing? It relates to task/comment (2.), but maybe an even more general intro/overview of the admin role would make sense.

Questions:

Should I run test the 'admin doc install guide' on the DTaaS integration server or do you have another linux server available I can test on? My own PC is running windows machine.

4. The instructions and install.sh need to be updated for basepath. All the ml-workspace containers need to have basepath as well. See issue Update install instructions for basepath #88 for more information

I know me talked about this one, but I'm not sure what and where to update. Is this already done?

6. Add representative network diagrams on all the installation pages (the docs/developer/system/DTaaS.drawio)

Are the network diagram already made? If yes which one is it in the draw.io file, the first one?

7. Admin --> Installation --> Cookbook page

- create a `docs/admin/guides` to add notes on modifications to the standard deployment scenarios. Currently known scenarios:
- add more users
- add other services
- link services to local ports using SSH port mapping
- admin user with read-write permissions to common/ directory
- add self-signed (`ssl/`) / LetsEncrypt certificates
- add or remove TLS certificates to gateway
- hosting the site without SSL certificates

Is this already written somewhere or do I need to write this? If so I think this would take me some time, because I have a knowledge gap, and each scenario would need to be specified further in order for me to understand.

10. Document basepath installations and possibility of multiple installations. See issue #88

Same question as number 4.

11. One Mermaid diagram showing the installation of different software components

Not sure how to do this one and what the requirements are. Is it to take one of the draw.io diagrams and convert to Mermaid diagram?

[prasadtalasila]

@prasadtalasila

Comments:

Admin section:

1. **Add a general purpose section**
   A section which shortly describes the role as an admin for the system and what the goal is. What am I installing? It relates to task/comment (2.), but maybe an even more general intro/overview of the admin role would make sense.

The Ubuntu install procedure gives a template for what we can write about the installation steps. Let us follow that for now.

Questions:

Should I run test the 'admin doc install guide' on the DTaaS integration server or do you have another linux server available I can test on? My own PC is running windows machine.

The integration server

4. The instructions and install.sh need to be updated for basepath. All the ml-workspace containers need to have basepath as well. See issue Update install instructions for basepath #88 for more information

I know me talked about this one, but I'm not sure what and where to update. Is this already done?
Not done yet. The places where the changes may be required are:

• gateway config in deployment
• install script
• single script install script
• New documentation file for basepath
• Minor cleanup of client config in deploy and client/config

6. Add representative network diagrams on all the installation pages (the docs/developer/system/DTaaS.drawio)

Are the network diagram already made? If yes which one is it in the draw.io file, the first one?
No

7. Admin --> Installation --> Cookbook page

- create a `docs/admin/guides` to add notes on modifications to the standard deployment scenarios. Currently known scenarios:
- add more users
- add other services
- link services to local ports using SSH port mapping
- admin user with read-write permissions to common/ directory
- add self-signed (`ssl/`) / LetsEncrypt certificates
- add or remove TLS certificates to gateway
- hosting the site without SSL certificates

Is this already written somewhere or do I need to write this? If so I think this would take me some time, because I have a knowledge gap, and each scenario would need to be specified further in order for me to understand.

I have the steps written in a markdown file. We can discuss a way to get this done easily

10. Document basepath installations and possibility of multiple installations. See issue #88

Same question as number 4.

11. One Mermaid diagram showing the installation of different software components

Not sure how to do this one and what the requirements are. Is it to take one of the draw.io diagrams and convert to Mermaid diagram?
The mermaid is a JS package to draw UML diagrams. It is already integrated into the documentation pipeline. Please see this example diagram for library microservice. You will probably be making a deployment diagram for this. If mermaid is not capable of doing this, we will use the drawio.

Hope the explanation helps. We will meet soon anyway and so we can discuss more soon. Thanks.

[nichlaes]

@prasadtalasila
I have started restructuring some of the documentation.

I have noted some reflections while trying the trial installation.

• When the installation is complete (e.g trial), how do I know that it is complete? Should I just access the site, or should I also try out a DigitalTwin/algorithm.

• When accessing the website, I am prompted by a username and password, this is not mentioned in the installation guide?

• Does the OAuth Provider have to be GitLab or could it be any OAuthProvider?

[prasadtalasila]

@nichlaes am not sure if you received answers to these questions. In case you did not receive them, please see below.

@prasadtalasila I have started restructuring some of the documentation.

I have noted some reflections while trying the trial installation.

* When the installation is complete (e.g trial), how do I know that it is complete? Should I just access the site, or should I also try out a DigitalTwin/algorithm.

For now, just access the website and check all the steps outlined in docs/user/website/index.md.

* When accessing the website, I am prompted by a username and password, this is not mentioned in the installation guide?

This is the password check made by Traefik gateway. It should be made obvious in the installation documentation.

Your idea of post-install checklist is a good one. Can you please add a page to the admin documentation section describing the checklist steps and the expected results?

* Does the OAuth Provider have to be GitLab or could it be any OAuthProvider?

It can be any OAuth provider. But the gitlab is also integrated into other components of DTaaS software. Thus having another OAuth provider will be extra work for user. This question and the explanation is a good candidate for admin cookbook section.

[prasadtalasila]

Adding more users

Create new account with on gitlab

Within DTaaS codebase

## create required files
cd DTaaS/files
cp -R user2 <username>

## start new workspace for user
cd ..
docker run -d \
 -p <port>:8080 \
  --name "ml-workspace-<username>" \
  -v "${TOP_DIR}/files/<username>:/workspace" \
  -v "${TOP_DIR}/files/<username>:/workspace/common" \
  --env AUTHENTICATE_VIA_JUPYTER="" \
  --env WORKSPACE_BASE_URL="<username>" \
  --shm-size 512m \
  --restart always \
  mltooling/ml-workspace:0.13.2

## add route in gateway
cd servers/config/gateway
htpassword auth <username>

vi dynamic/fileConfig.yml

http:
  routers:
    ....
    <username>:
      entryPoints:
        - http
      rule: 'Host(`foo.com`) && PathPrefix(`/<username>`)'
      middlewares:
        - basic-auth
      service: <username>

  services:
    ...
    <username>:
      loadBalancer:
        servers:
          - url: 'http://localhost:<port>'

Add new services

Provide instructions for installing MongoDB

link services to local ports using SSH port mapping

do SSH-link from workspace to remote services
ssh -fNT -L 15672:localhost:15672 [email protected]
ssh -fNT -L 5672:localhost:5672 [email protected]

Set read only access to common asset area:

docker run -d \
 -p 8091:8080 \
  --name "ml-workspace-user2" \
  -v "${TOP_DIR}/files/user2:/workspace" \
  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
  --env AUTHENTICATE_VIA_JUPYTER="" \
  --env WORKSPACE_BASE_URL="user2" \
  --shm-size 512m \
  --restart always \
  mltooling/ml-workspace:0.13.2 || true

Hosting site without https

correct the following sentence on deploy/admin/overview.md

If you do not have a reverse proxy, please replace https://foo.com with http://foo.com in client .env file and in OAuth registration. Other installation configuration remains the same.

provide correct sample client env.js and OAuth table

[prasadtalasila]

PR #300 completes most of this work.