atlarge-research / opendc

Collaborative Datacenter Simulation and Exploration for Everybody

OpenDC

Collaborative Datacenter Simulation and Exploration for Everybody

---

This repository is the home of the OpenDC project, a free and open-source platform for cloud datacenter simulation.

Latest Release

• General Availability (GA): OpenDC v2.0 (May 10, 2021)
• Preview (Release Candidate): OpenDC v3.0-rc1 (Jan 27, 2023)

Documentation

You can find the OpenDC documentation on the website. The documentation is divided into several sections:

• Getting Started
• Tutorials
• Advanced Guides
• Where to Get Support
• Contributing Guide

The source code for the documentation is located in site.

Contributing

Questions, suggestions and contributions are welcome and appreciated! Please refer to the contributing guidelines for more details.

License

OpenDC is distributed under the MIT license. See LICENSE.txt.

opendc's Issues

Task runtime scaling

Add the ability for tasks to be scaled resulting in faster execution when possible.

OpenDC location in keys.json file is redundant

The location of opendc inside a container is decided in the Dockerfile. But, this location also needs to be present in the keys.json. Mismatch between these locations leads to errors.

Make docker location in keys.json optional so that users don't need to enter it.

Support affinity scheduling

Allow the notion of tasks (virtual machines) having an affinity to a certain type of resource (or even processor). This requires modeling affinities on the workload-side and making the scheduler aware of those affinities, trough affinity-aware policies.

Depending on how granular we want to make this, we also might need to support inividual processing assignment (as opposed to general machine-based assignment).

Implement a method of giving schedulers the possibility of selecting which servers to schedule first

Add tests for opendc-trace-api

Upload experiment results via OpenDC CLI

Users should be able to run experiments locally and upload their results to opendc.org via the OpenDC command line interface, so that they can also utilize the visualization tools of the OpenDC web interface.

Sidebars hide zoom buttons, export buttons and scale

When extended, both sidebars hide the content below them.
This includes the zoom and export buttons on the left and the scale indicator on the right (see images).

I would expect the elements to move towards the middle of the screen when the sidebars are extended

Sidebars extended, buttons hidden by them

Sidebars collapsed, buttons visible in lower corners

Web server build fails inside Docker

I'm getting an error when running docker-compose build. It gets to the frontend build but then gives me this:. I'll include some of the lines just before it stopped.... Any suggestions?

Writing /tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/setup.cfg
Running Jinja2-3.0.0a1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/egg-dist-tmp-Dk6W_f
Traceback (most recent call last):
  File "/opendc/opendc-web-server/setup.py", line 44, in <module>
    'six==1.11.0'
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 436, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 680, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 727, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 772, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 846, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1118, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1130, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 699, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 725, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 906, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1145, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1131, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 258, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 169, in save_modules
    saved_exc.resume()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 144, in resume
    six.reraise(type, exc, self._tb)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 157, in save_modules
    yield saved
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 255, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 285, in run
    return func()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 253, in runner
    _execfile(setup_script, ns)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/setup.py", line 6, in <module>
    # Get the long description from the README file
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 421, in _open
    return _open(path, mode, *args, **kw)
TypeError: 'encoding' is an invalid keyword argument for this function
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c python /opendc/opendc-web-server/setup.py install 	&& chmod 555 /opendc/build/configure.sh 	&& cd /opendc/opendc-frontend 	&& rm -rf ./build 	&& rm -rf ./node_modules 	&& npm install 	&& export REACT_APP_OAUTH_CLIENT_ID=$(cat ../keys.json | python -c "import sys, json; print json.load(sys.stdin)['OAUTH_CLIENT_ID']") 	&& npm run build' returned a non-zero code: 1

Add monitor that ends after all tasks are finished

In the current implementation of OpenDC, a trace runs for a set timespan.
However, there should be a method to keep running until all tasks are finished.

Update metrics

• Change metrics to similar units
• Look at the datetime metrics

Model machine resource startup and clean-up time

Currently, machines are assumed to be instantly running, without any delay. However, especially once we start supporting Kubernetes-like pods or functions, this needs to become more nuanced, since a machine needs to time to start and clean up, as well.

Unable to create machines on lower shelves on rack without populating higher ones

Steps to reproduce

1. Create a new topology.
2. Create a new room in that topology.
3. Add a rack to the new room.
4. Attempt to populate a slot other than the top slot, for example scroll down to the slot named 1U.
5. Press + on the 1U Empty Slot
6. Select Machine with no units

Expected

I should be able to populate the machine just as I would if I selected the top shelf of the rack.

Actual

The entire UI just disappears, left with just the gray background color.

The following error is logged in the console.

TypeError: can't access property "id", machine is undefined
    NextJS 39
react_devtools_backend.js:4012:25

Look at the functionality of the different power models

Add spotless apply for java

Currently only kotlin and gradle are checked for spotless code.
Java code should also be checked

Parquet Exporter exports 0.0 for all non-host values

Re-design workflow service

This issue is concerned with making several requested improvements to the current opendc-workflow module:

• Split API and service implementation.
  A consumer of the workflow service in OpenDC should not need to depend on the service, but only on the API client. In this way, we could also provide alternative or mocked API implementations.
• Represent workflows as graph
• Enable easier construction of workflows with mutable builders

Process

• Design initial API specification
• Implement initial API
• Port existing codebase to new API

Add CI to github

Add tests for opendc-experiment-compute

Server and simulator stop working due to database locking

During the course on 27/2, this issue repeatedly occured, forcing a restart of the entire server. The python server would throw an error that the database was already locked. It is not clear what exactly causes this issue.

Fix host failures

Update all tests

Incorperate features from RaDiCe into master branch

Unable to delete rack from floor plan

Steps to reproduce

1. Create a new topology and select it
2. Create a new room
3. Add a rack to that room
4. Attempt to delete the rack (Press the Delete this rack button)

Expected

The rack should get deleted and the modal should close.

Actual

Nothing happens in the UI, and the modal is left open.
In the console the following error occurs:

TypeError: can't access property "rack", state.topology.tiles[tileId] is undefined
    NextJS 95
react_devtools_backend.js:4012:25

Add documentation about traces and metrics

Look at the functionality of interference

The interference tests currently result in very different results than before.
Make sure this is correct

Add virtual machine migration

Given that the simulator now contains a hypervisor implementation, the simulated resource manager should be adapted to be able to migrate them. At this point, this means a simplistic model of instantaneous migrations from one machine to the other, without looking at transfer times.

This includes:

• Researching and selecting a set of migration policies that we want to implement
• Implement these policies in the scheduler
• Support migration of VMs through simple manipulation of the physical properties included within the logical model (i.e. modifying the machine a certain VM has been assigned to)

Expand Scenarios

Add all variables of an OpenDC run to the scenario dataclass.
Most variables should have default variables.

Workflows

Add workflows to OpenDC

Add test for the parquet exporter

Add support for fault traces

Add support for providing traces that describe when different failures should occur

Change the format of host_id and server_id in the Parquet Exporter to string

List dependencies in the readme

I see that docker is required for opendc to run, it might be useful to add all dependencies that are required to be listed in the readme.

Perhaps even an apt-get line for linux distro's and links + versions for mac/windows/etc.?

Google login broken

Logging into the opendc.org website using Google no longer works.

I believe I submitted a ticket about this - which was resolved - but the ticket is now also gone.

I have tried both Chrome and Firefox, have cleared cookies and tried again, etc. No luck.

Be great to get this resolved. :-)

Migrate to GitHub Container Registry

Docker Inc is sunsetting free team organizations for the Docker registry, which our organization is one of. Instead, a paid subscription is now required to maintain the organization.

Given our relatively small usage of the account, it makes more sense to start publishing the container images on the GitHub Container Registry, since it is free for open source projects and integrates well with GitHub Actions.

Support multiple workload types in OpenDC frontend

We should support uniform handling of both types of traces we currently support (workflow and VM traces) in the simulator. Currently, we only support VM traces in the frontend.

ComputeServiceHelper.registerHost allows to add multiple hosts with same UUID

registerHost allows to add arbitrary hosts to the hosts set.
This includes hosts, that share the same UUID, but are not the same. Functions that rely on the UUID of the host for identification therefore produce unreliable results.

Add tests for opendc-compute-api

Improve Telemetry

Create a new method of exporting data. It should be a simple class in which three things are decided by the user:

• When is a sample taken
• How is the data aggregate (or not aggregated)
• How is the data exported (parquet, csv, console output)

Internal server error when attempting to delete topology referenced in scenario

Steps to reproduce

1. Create a topology
2. Create a portfolio
3. Select your new portfolio
4. Create a new scenario, in the modal that appears, specify the topology from step 1
5. Navigate back to project details
6. Attempt to delete your topology

Expected

A warning that it cannot be deleted because it's referenced in a portfolio/scenario, and that either the scenario must be deleted first. (Or the entire portfolio the scenario is in.)

In the network tab I should something more like a 400 (bad request) or 403 (forbidden).

Actual

Nothing happens in the UI, but in Network tab I can see a 500, internal server error.

Change output timestamp to absolute and relative

Currently, the timestamp in the output parquet files are relative (IE. The time since the start of the experiment).

Add a relative timestamp to the output that is based on given workload.

UI disappears when adding CPU/Memory without selecting a unit

Steps to reproduce

1. Create a new topology
2. Create a new room in that topology
3. In the floor plan, create a room, then add a rack to that room
4. Start the flow to create a new machine on the top shelf of the rack
5. When asked to specify a GPU or Memory, do not select a unit. Just click + Add.

Expected

The app should display a warning that selecting a unit is mandatory to use the + Add button.

Actual

The entire UI disappears, no action is performed, and an error is logged in the console.

TypeError: can't access property "energyConsumptionW", cpus[id] is undefined
    NextJS 79
react_devtools_backend.js:4012:25

Uncaught TypeError: can't access property "energyConsumptionW", cpus[id] is undefined
    NextJS 79
38.eec3bf165761755e.js:1:5546

OpenDC has problems compiling on windows

Building OpenDC on windows causes problems due to pathing

Use setup.py in Docker setup

The web-server actually has a setup.py file, so we should use it in the setup process!

Fix fault injection

Fault injection when used together with Traces result in a crash

Update topology format

Change the format of the topology file provided by the user to json.

New floor selection offset from cursor when dragging

Steps to reproduce

1. Create a new topology and select it.
2. Navigate to the Floor Plan tab.
3. Select + Construct a new room
4. Drag the view using the mouse. (Hold primary-click and move the cursor so that the grid moves.)
5. Attempt to select a square in the grid.

Expected

When I hover over a square, it should be highlighted and easy to select.

Actual

The grid and cursor fall out of sync. I have to select the top left (where my grid was before) to select the first item, for example.

In all the screenshots, the red dot indicates my cursor location.

Without dragging the grid, the placement is fine.

Now that I've dragged the grid around, it's offset.

If I try to actually click the top-left grid item.

Limit simulation time per user

At the moment, the hosted OpenDC service does not place any restrictions on the amount of simulation time consumed by their experiments. In case of a malicious user (or misconfiguration), this could lead to one user consuming all compute resources available for the hosted service.

This issue proposes adding limits to the amount of simulation time that can be consumed per user per time period (e.g., month). In this way, abuse or misconfiguration should be prevented.

Add tutorial(s) for using OpenDC for research

The repository should contain one or preferably more (5/6) step tutorials for using OpenDC for research activities. These tutorials may for example go into detail on how to reproduce several of our works in which OpenDC was used.

Implement cost models

A large factor in real-world decision-making is, naturally, the financial picture. Buying/leasing servers comes with certain costs, just as power supply does. The former can even be interpolated to compute the per-hour cost of a certain resource, to accurately be able to quantify the cost of running certain workloads or introducing certain resources to the mix. Adding client-side cost models (how much customers are charged) would make this even more realistic, but this also adds a significant amount of complexity. SLA penalty calculations could also be added for more realism, using the available failure models.

This includes:

• Similarly to how power consumption is be modeled, each rack and machine should not only compute the power it uses, but also its cost of ownership. For this, the cost of each resource type needs to be recorded
• For each unit of power consumption, a cost needs to be accumulated
• Nice to have: implementing client-side models
• Nice to have: implementing SLA penalties

Add total counts to Service output

Currently there is no indication of the total number of servers executed, this might be nice to have