Coder Social home page Coder Social logo

atlarge-research / opendc Goto Github PK

View Code? Open in Web Editor NEW
67.0 9.0 42.0 27.32 MB

Collaborative Datacenter Simulation and Exploration for Everybody

Home Page: https://opendc.org

License: MIT License

Shell 0.01% Dockerfile 0.10% JavaScript 9.91% Kotlin 43.34% Java 18.89% CSS 0.53% MDX 0.85% Jupyter Notebook 26.38%
simulator datacenter tu-delft data-center opendc cloud-computing big-data serverless cloud-simulation vu-amsterdam

opendc's Introduction

OpenDC logo

OpenDC

Collaborative Datacenter Simulation and Exploration for Everybody

MIT License Documentation GitHub release Build


This repository is the home of the OpenDC project, a free and open-source platform for cloud datacenter simulation.

Latest Release

Documentation

You can find the OpenDC documentation on the website. The documentation is divided into several sections:

The source code for the documentation is located in site.

Contributing

Questions, suggestions and contributions are welcome and appreciated! Please refer to the contributing guidelines for more details.

License

OpenDC is distributed under the MIT license. See LICENSE.txt.

opendc's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

opendc's Issues

New floor selection offset from cursor when dragging

Steps to reproduce

  1. Create a new topology and select it.
  2. Navigate to the Floor Plan tab.
  3. Select + Construct a new room
  4. Drag the view using the mouse. (Hold primary-click and move the cursor so that the grid moves.)
  5. Attempt to select a square in the grid.

Expected

When I hover over a square, it should be highlighted and easy to select.

Actual

The grid and cursor fall out of sync. I have to select the top left (where my grid was before) to select the first item, for example.

In all the screenshots, the red dot indicates my cursor location.

image

Without dragging the grid, the placement is fine.

image

Now that I've dragged the grid around, it's offset.

image

If I try to actually click the top-left grid item.

Task runtime scaling

Add the ability for tasks to be scaled resulting in faster execution when possible.

Unable to delete rack from floor plan

Steps to reproduce

  1. Create a new topology and select it
  2. Create a new room
  3. Add a rack to that room
  4. Attempt to delete the rack (Press the Delete this rack button)

Expected

The rack should get deleted and the modal should close.

Actual

Nothing happens in the UI, and the modal is left open.
In the console the following error occurs:

TypeError: can't access property "rack", state.topology.tiles[tileId] is undefined
    NextJS 95
react_devtools_backend.js:4012:25

UI disappears when adding CPU/Memory without selecting a unit

Steps to reproduce

  1. Create a new topology
  2. Create a new room in that topology
  3. In the floor plan, create a room, then add a rack to that room
  4. Start the flow to create a new machine on the top shelf of the rack
  5. When asked to specify a GPU or Memory, do not select a unit. Just click + Add.

Expected

The app should display a warning that selecting a unit is mandatory to use the + Add button.

Actual

The entire UI disappears, no action is performed, and an error is logged in the console.

TypeError: can't access property "energyConsumptionW", cpus[id] is undefined
    NextJS 79
react_devtools_backend.js:4012:25

Uncaught TypeError: can't access property "energyConsumptionW", cpus[id] is undefined
    NextJS 79
38.eec3bf165761755e.js:1:5546

Sidebars hide zoom buttons, export buttons and scale

When extended, both sidebars hide the content below them.
This includes the zoom and export buttons on the left and the scale indicator on the right (see images).

I would expect the elements to move towards the middle of the screen when the sidebars are extended

image
Sidebars extended, buttons hidden by them
image
Sidebars collapsed, buttons visible in lower corners

OpenDC location in keys.json file is redundant

The location of opendc inside a container is decided in the Dockerfile. But, this location also needs to be present in the keys.json. Mismatch between these locations leads to errors.

Make docker location in keys.json optional so that users don't need to enter it.

Expand Scenarios

Add all variables of an OpenDC run to the scenario dataclass.
Most variables should have default variables.

List dependencies in the readme

I see that docker is required for opendc to run, it might be useful to add all dependencies that are required to be listed in the readme.

Perhaps even an apt-get line for linux distro's and links + versions for mac/windows/etc.?

Upload experiment results via OpenDC CLI

Users should be able to run experiments locally and upload their results to opendc.org via the OpenDC command line interface, so that they can also utilize the visualization tools of the OpenDC web interface.

Limit simulation time per user

At the moment, the hosted OpenDC service does not place any restrictions on the amount of simulation time consumed by their experiments. In case of a malicious user (or misconfiguration), this could lead to one user consuming all compute resources available for the hosted service.

This issue proposes adding limits to the amount of simulation time that can be consumed per user per time period (e.g., month). In this way, abuse or misconfiguration should be prevented.

Update metrics

  • Change metrics to similar units
  • Look at the datetime metrics

Support affinity scheduling

Allow the notion of tasks (virtual machines) having an affinity to a certain type of resource (or even processor). This requires modeling affinities on the workload-side and making the scheduler aware of those affinities, trough affinity-aware policies.

Depending on how granular we want to make this, we also might need to support inividual processing assignment (as opposed to general machine-based assignment).

Google login broken

Logging into the opendc.org website using Google no longer works.

I believe I submitted a ticket about this - which was resolved - but the ticket is now also gone.

I have tried both Chrome and Firefox, have cleared cookies and tried again, etc. No luck.

Be great to get this resolved. :-)

Model machine resource startup and clean-up time

Currently, machines are assumed to be instantly running, without any delay. However, especially once we start supporting Kubernetes-like pods or functions, this needs to become more nuanced, since a machine needs to time to start and clean up, as well.

Internal server error when attempting to delete topology referenced in scenario

Steps to reproduce

  1. Create a topology
  2. Create a portfolio
  3. Select your new portfolio
  4. Create a new scenario, in the modal that appears, specify the topology from step 1
  5. Navigate back to project details
  6. Attempt to delete your topology

Expected

A warning that it cannot be deleted because it's referenced in a portfolio/scenario, and that either the scenario must be deleted first. (Or the entire portfolio the scenario is in.)

In the network tab I should something more like a 400 (bad request) or 403 (forbidden).

Actual

Nothing happens in the UI, but in Network tab I can see a 500, internal server error.

image

Web server build fails inside Docker

I'm getting an error when running docker-compose build. It gets to the frontend build but then gives me this:. I'll include some of the lines just before it stopped.... Any suggestions?

Writing /tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/setup.cfg
Running Jinja2-3.0.0a1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/egg-dist-tmp-Dk6W_f
Traceback (most recent call last):
  File "/opendc/opendc-web-server/setup.py", line 44, in <module>
    'six==1.11.0'
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 117, in do_egg_install
    cmd.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 436, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 680, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 727, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 772, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 846, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1118, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1130, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 699, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 725, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 906, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1145, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1131, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 258, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 169, in save_modules
    saved_exc.resume()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 144, in resume
    six.reraise(type, exc, self._tb)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 157, in save_modules
    yield saved
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 198, in setup_context
    yield
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 255, in run_setup
    DirectorySandbox(setup_dir).run(runner)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 285, in run
    return func()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 253, in runner
    _execfile(setup_script, ns)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 47, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-L5ljjq/Jinja2-3.0.0a1/setup.py", line 6, in <module>
    # Get the long description from the README file
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 421, in _open
    return _open(path, mode, *args, **kw)
TypeError: 'encoding' is an invalid keyword argument for this function
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c python /opendc/opendc-web-server/setup.py install 	&& chmod 555 /opendc/build/configure.sh 	&& cd /opendc/opendc-frontend 	&& rm -rf ./build 	&& rm -rf ./node_modules 	&& npm install 	&& export REACT_APP_OAUTH_CLIENT_ID=$(cat ../keys.json | python -c "import sys, json; print json.load(sys.stdin)['OAUTH_CLIENT_ID']") 	&& npm run build' returned a non-zero code: 1

Add virtual machine migration

Given that the simulator now contains a hypervisor implementation, the simulated resource manager should be adapted to be able to migrate them. At this point, this means a simplistic model of instantaneous migrations from one machine to the other, without looking at transfer times.

This includes:

  • Researching and selecting a set of migration policies that we want to implement
  • Implement these policies in the scheduler
  • Support migration of VMs through simple manipulation of the physical properties included within the logical model (i.e. modifying the machine a certain VM has been assigned to)

Add tutorial(s) for using OpenDC for research

The repository should contain one or preferably more (5/6) step tutorials for using OpenDC for research activities. These tutorials may for example go into detail on how to reproduce several of our works in which OpenDC was used.

Implement cost models

A large factor in real-world decision-making is, naturally, the financial picture. Buying/leasing servers comes with certain costs, just as power supply does. The former can even be interpolated to compute the per-hour cost of a certain resource, to accurately be able to quantify the cost of running certain workloads or introducing certain resources to the mix. Adding client-side cost models (how much customers are charged) would make this even more realistic, but this also adds a significant amount of complexity. SLA penalty calculations could also be added for more realism, using the available failure models.

This includes:

  • Similarly to how power consumption is be modeled, each rack and machine should not only compute the power it uses, but also its cost of ownership. For this, the cost of each resource type needs to be recorded
  • For each unit of power consumption, a cost needs to be accumulated
  • Nice to have: implementing client-side models
  • Nice to have: implementing SLA penalties

Re-design workflow service

This issue is concerned with making several requested improvements to the current opendc-workflow module:

  • Split API and service implementation.
    A consumer of the workflow service in OpenDC should not need to depend on the service, but only on the API client. In this way, we could also provide alternative or mocked API implementations.
  • Represent workflows as graph
  • Enable easier construction of workflows with mutable builders

Process

  • Design initial API specification
  • Implement initial API
  • Port existing codebase to new API

Migrate to GitHub Container Registry

Docker Inc is sunsetting free team organizations for the Docker registry, which our organization is one of. Instead, a paid subscription is now required to maintain the organization.

Given our relatively small usage of the account, it makes more sense to start publishing the container images on the GitHub Container Registry, since it is free for open source projects and integrates well with GitHub Actions.

Unable to create machines on lower shelves on rack without populating higher ones

Steps to reproduce

  1. Create a new topology.
  2. Create a new room in that topology.
  3. Add a rack to the new room.
  4. Attempt to populate a slot other than the top slot, for example scroll down to the slot named 1U.
  5. Press + on the 1U Empty Slot
  6. Select Machine with no units

Expected

I should be able to populate the machine just as I would if I selected the top shelf of the rack.

Actual

The entire UI just disappears, left with just the gray background color.

The following error is logged in the console.

TypeError: can't access property "id", machine is undefined
    NextJS 39
react_devtools_backend.js:4012:25

Improve Telemetry

Create a new method of exporting data. It should be a simple class in which three things are decided by the user:

  • When is a sample taken
  • How is the data aggregate (or not aggregated)
  • How is the data exported (parquet, csv, console output)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.