pcuzner / fioloadgen Goto Github PK

The UI currently shows the completion time only. This enhancement would provide a duration field following the completion time stamp as follows;

Completed: 2021/03/18 09:12:47 (10m)

By adding a kebab component, the row can have actions that are specific to the job state

queued -> delete
complete -> rerun, export, show
failed -> show
started -> show

Both the UI and CLI should allow jobs in a complete or failed state to be deleted

./fiodeploy.sh -s
Deployment will use worker pods based on yaml/fio.yaml
get_environment:read:2: -p: no coprocess

• defaulting to 'fio' namespace
  get_environment:read:7: -p: no coprocess
  Worker count must be within the range 1-20

When you have queued jobs, the job summary tab is broken... This is because of this code:

In this call (and the similar one after)

t[n].toString().localeCompare(e[n].toString()) 

You are comparing the start times of the jobs.

If a job is queued, it will look like this:

{
    "id": "734af13f-69a0-4185-8171-38b39828a81a",
    "title": "ups_4k_50.fio",
    "profile": "ups_4k_50.fio",
    "status": "queued",
    "started": null
}

because started is null, the toString() method will break and produces this error:

bundle.js:formatted:31740 Error: TypeError: Cannot read property 'toString' of null
    at bundle.js:formatted:31731
    at Array.sort (<anonymous>)
    at bundle.js:formatted:31727

For the user the site stays blank

When the statefulset is scaled the new workers count is not passed to the profiles component, so you can only run jobs up to the limit set at deployment time

The refresh call is syncing the database to the profiles in the local filesystem - removing any newly added profiles! Profiles that we added still appea rin the profiles list, but without content and the log shows a key error when the related db row is accessed.

Python dependency cherrypy

./fioservice.py start
Traceback (most recent call last):
  File "./fioservice.py", line 11, in <module>
    from fiotools.server import FIOWebService
  File "/root/fioloadgen/fiotools/server/__init__.py", line 1, in <module>
    from .web import FIOWebService  # NOQA:F401
  File "/root/fioloadgen/fiotools/server/web.py", line 3, in <module>
    import cherrypy
ModuleNotFoundError: No module named 'cherrypy'

and requests

./fiocli.py profile --ls
Traceback (most recent call last):
  File "./fiocli.py", line 4, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

Also need an fio project

Error from server (NotFound): namespaces "fio" not found

By doing this, the door is open to changes being made in the UI as well as ensuring that the jobs that actually run are the same as those represented in the UI

Capture a GUI workflow as an animated GIF, and embed in the readme with comments for context.

The presence of .py on the fioservice and fiocli commands should be removed.

Using affinity rules we can tell Kubernetes to try to spread the fio workers on different nodes:
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#an-example-of-a-pod-that-uses-pod-affinity

Being able to display the fio job spec that was executed would help understand the results, and also protects against future changes that may change the profile in the profile table.

Update the jobs table

• remove uuid
• add profile name
• change show to show output in kebab menu

In the native deployment model, the fioservice could handle the deployment of the workers itself - which would simplify the initial deployment workflow. This would also allow multiple storageclass to be utilised by the same fioservice, so you could easily perform a bake-off test against different storage providers from the same fioservice instance.

To reproduce:

• Have completed and queued items in the summary list
• Check some items in the summary list
• Wait until some fio runs have completed in the background
• Click on "refresh"
  --> This re-sorts the list, but the previously selected rows are still selected, details are not refreshed and still show the old details

communication between cli/web client and the server should be over https. this will help position for the fioservice running remotely

out-of-the-box profiles should be consistent with options and not have commented out options

The profiles available shouldn't be limited, and the user should be able to add more. This will require additional API and CLI interactions to support profile upload and delete functionality

I don't necessarily want to only limit the web frontend to my workstation, but collaborate on the results...

the fiomgr is collating data from the clients, but at the time the task completes it may not have written to disk so when the read of the file comes in it only gets an empty file causing json errors when the db is updated.

profiles are currently presets loaded into the fioservice database. This feature would enable the UI to create a custom profile within the UI to support a performance run

• fio client and servers must be the same version
• update the text when seeding in deploy (please be patient, this may take some time)
• rename baner to fioloadgen

by creating multiple groups of workers the tests can be performed against different storageclasses for direct comparison, avoiding saving and import results to combine the data

You may add jobs to the queue at the CLI, but they will operate serially. If you change your mind about the jobs, there is currently no way to remove a job from the pending queue.

This needs to be implemented within the API, and then exposed in the CLI and GUI

by implementing an ssh server, the interaction between the API backend handler and the fiomgr can be ssh based and avoid using the oc command

this would entail

• install ssh server in the fiotester pod
• update the deploy script to create the ssh key-pair for the fiomgr
• include a parameter and config setting to point to the specific ssh pub key to use
• change tiomgr logic to use sshd as the active task
• update the handler code to use ssh calls

To avoid scrolling when looking at 3 or more jobs, it would be useful to be able to zoom in to that pane, and then return to the split view.

add the ability to import an exported database record

When showing the details of a queued job, fiocli breaks:

./fiocli.py job --show 7ea6a777-c567-4e08-b1ec-533a17c1ef33
Id: 7ea6a777-c567-4e08-b1ec-533a17c1ef33
Title: ups_512k_10.fio
Traceback (most recent call last):
  File "./fiocli.py", line 226, in <module>
    args.func()
  File "./fiocli.py", line 195, in command_job
    show_summary(r)
  File "./fiocli.py", line 130, in show_summary
    print("Run Date: {}".format(datetime.datetime.fromtimestamp(data[k]).strftime('%Y-%m-%d %H:%M:%S')))  # NOQA
TypeError: an integer is required (got type NoneType)

This is probably similar to #3

At the moment the deploy process expects to create the ns

Allow a user to export rows from the jobs table, so they can be later imported or even service as a backup tool.

Got this issue when using fiodeploy.sh:

➜  fioloadgen git:(master) ./fiodeploy.sh -s
Deployment will use worker pods based on yaml/fio.yaml
What namespace should be created for the workload test [fio]? 
- defaulting to 'fio' namespace
How many fio workers [2]? 3
Checking access to kubernetes
Setting up the environment
Creating the namespace (fio)
Deploying the mgr
pod/fiomgr created
Deploying the workers
persistentvolumeclaim/fioworker1 created
pod/fioworker1 created
persistentvolumeclaim/fioworker2 created
pod/fioworker2 created
persistentvolumeclaim/fioworker3 created
pod/fioworker3 created
Waiting for worker pods to enter a running state
	 - waiting for pods to reach 'Running' state (1/12)
	 - waiting for pods to reach 'Running' state (2/12)
	 - waiting for pods to reach 'Running' state (3/12)
	 - waiting for pods to reach 'Running' state (4/12)
Removing residual IP information from data directory
Fetching the worker IP information (pod and host)
./fiodeploy.sh: line 124: lookup: bad array subscript
./fiodeploy.sh: line 129: lookup[${hostIP}]: bad array subscript

This is on commit 3527478 (current master)

My system is up-to-date Fedora 31 with ZSH shell (which shouldn't matter, since you select the shell in the script header)

add a syntax check to the fio jobs during fiodeploy and the fioservice

By including the spec in the record, we position for customizable one-off fio runs that don't have a profile loaded in the profile table

The namespace given is just deleted, so a mistype could lead to unintended deletion. The -d workflow should

• check the namespace contains only the fioloadgen resources, and proceed
• if the check fails, the deletion is aborted

In addition, the cleanup should also handle any port-forward or local fioservice daemon to return the users desktop to its original state.

The workflow is missing a number of features

• summary information before deployment
• consistent use of tick an colors
• checking availability of the port that the fioservice needs
• providing pid information for port-forward actions (so the user can clean up easier)

This would remove the warning during deployment:

Transfering worker IP addresses (pod and host), plus fio job specs to the fiomgr pod
WARNING: cannot use rsync: rsync not available in container

and should make the overall transfer quicker

When you tick a row to show the analysis reports, and then choose delete. The row is removed, but the graphs remain

When fio has issues during the run, they are saved with the json output... breaking the frontend when it tries to parse this :/

Result:

• Website shows a blank screen when selecting the a completed job
• CLI shows this:

./fiocli.py job --show 3e11d521-8cc9-441a-b49f-9c4416be4e1c --raw
Id: 3e11d521-8cc9-441a-b49f-9c4416be4e1c
Title: ups_4k_50.fio
Run Date: 2020-03-25 10:29:44
Profile: ups_4k_50.fio
Workers: 1
Status: complete
Summary:
  Clients: 1
  Total_Iops: 7694.399723
  Read Ms Min/Avg/Max: 1.94/1.94/1.94
  Write Ms Min/Avg/Max: 6.13/6.13/6.13
Traceback (most recent call last):
  File "./fiocli.py", line 226, in <module>
    args.func()
  File "./fiocli.py", line 198, in command_job
    js = json.loads(jstr)
  File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Providing fioservice as a container avoids dependency issues

• containerize the fioservice
• update the deployment to offer container based model
• update fiodeploy to handle the port-forward