A collaborative platform for rapid and reproducible research.
Home Page: https://beaker.org
License: Apache License 2.0
Beaker is a collaborative platform for rapid and reproducible research.
Documentation can be found here.
👀 Looking for a Python client? See beaker-py.
Please refer to our installation guide.
To run your own Beaker cluster, please refer to our self-host guide
If the auth token is invalid, beaker session create fails with the following error:
Error: invalid image: could not resolve image "ai2/cuda11.2-ubuntu20.04": invalid authentication token
It should prompt for a user token like other commands do when the token is invalid.
One of our users has to use the web API for:
downloading individual files of a dataset
Get to know all the datasets for an experiment
We should record all of these use cases and make the APIs more user friendly
There's a couple of security alerts associated with dependencies that need to be updated:
I was looking at this experiment/task in the UI: https://beaker.org/ex/ex_ufzij9wc4h7u/tasks/tk_71k4zznd2amm and trying to see the result metrics with the CLI:
The UI shows a successful experiment; both tasks completed. But the CLI reports "pending" for both tasks/executions.
Follow up to https://github.com/allenai/beaker-docs/pull/5
If someone uses the VSCode Server to remote connect to machines, but doesn't manually clean up after themselves, there will be orphaned processes and an unnecessary folder left over. It'd be nice to have a cron job that automatically cleans up these in a similar fashion to what VSCode provides in the "Remote-SSH: Uninstall VS Code Server from Host..." command.
The M1 Macbooks will be in circulation at AI2 soon. We need to build a version of the Beaker client for the architecture, otherwise people who get M1 machines won't be able to use Beaker.
Discussion in Slack: https://allenai.slack.com/archives/C6MN19S05/p1620159452182800?thread_ts=1620153910.177000&cid=C6MN19S05
Example from console:
From Chandra:
beaker session create --image gcr.io/ai2-beaker-core/public/c28p88ardlu9jnadldbg --gpus 2
Scheduling session 01F4WGG3QQNEFZ56TS09J7BWQR with at least 2 GPUs... (Press Ctrl+C to cancel)
Reserved 2 GPUs, 11.25 CPUs, 91148704358 B memory
Pulling image...
Error: Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
From Michael:
michaels@ai2-mosaic-server2 ~ $ docker tag gcr.io/ai2-beaker-core/public/c28p88ardlu9jnadldbg foo
michaels@ai2-mosaic-server2 ~ $ beaker session create --image foo --gpus 2
Scheduling session 01F4WGPKGV6C6F1TT3Y05S7S66 with at least 2 GPUs... (Press Ctrl+C to cancel)
Reserved 2 GPUs, 11.25 CPUs, 91148704358 B memory
Pulling image...
Error: Error response from daemon: pull access denied for foo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
I've use beaker cli to submit image, but it reported "Error: *** is not authorized to create workspaces", how can I solve it? Thanks.
Presently one needs to click into each experiment and manually sum the cost of each.
Currently the leaderboard links out to Beaker, but in some cases users will get a 404 because they are not logged into Beaker. We don't think there's a great fix on the leaderboard side since it's not aware of whether the user is logged into Beaker and has permissions to view the page, but we would like an update to the current 404 page.
https://user-images.githubusercontent.com/954798/58968946-b8038680-876b-11e9-93f0-5c8ba047440d.png
We should have something that looks less like something went terribly wrong ("undefined -- Error: Request failed with status code 404") and instead say something friendly like "Page Not Found! The page may not exist or you do not have access to it." or, if you are logged out, "Page Not Found! Please try again after logging in. The page may not exist or you do not have access to it."
Hi, I downloaded the newest release and extract it to \usr\local\bin,
when I want to set my user token, the error comes up.
Error: open /home/xxx/.beaker/config.yml: no such file or directory
can you help me ?
When I run the example beaker config YAML shown in https://github.com/beaker/docs/blob/main/docs/start/run.md#creating-a-beaker-specification-for-your-experiment, I see
$ beaker experiment create beaker-conf.yaml
Error: yaml: line 4: did not find expected '-' indicator
If I un-indent everything after tasks:, it runs successfully.
The output will always be non-JSON
I am trying to set up Beaker to submit a model to the aNLI leaderboard, but running into some trouble with the installation. I’ve followed all the installation steps, but there seems to be an issue with a missing config file that I’m having trouble resolving.
I am on a Mac and I’ve installed Release v20200402. When I try to set my user token I receive the following error:
beaker config set user_token xxxxx
Error: open /Users/alineberry/.beaker/config.yml: no such file or directory
I can’t find any instructions for how to set up this config.yml file. I also tried installing via Brew with the same result.
Any help in this matter would be greatly appreciated.
asdfasdf
Presently I need to drill down and add all the sizes of the files. I'd like to have this at least on the datasets page (e.g. https://beaker.org/ds/ds_eawz8kfvv7mp/details).
Presently many people erroneously create multiple accounts. Some of these accounts don't have an associated email address and others contains duplicate email addresses. It would be easier to troubleshoot leaderboard issues if the email address were required and unique.
In the Getting Started guide for Beaker, we ask users to request that their account be elevated to the "Scientist" role.
We don't, however, tell them how to request this. We should document the email address they should contact alongside this recommendation.
https://github.com/beaker/docs/blob/main/docs/start/run.md
When setting up a first experiment in Beaker, the reader is told to install a NVIDIA toolkit download which is only for Linux. Consulted with @mkusold and we decided to raise the issue since this seems unnecessary & confusing for those using Macs. Since we all use mac os, @aimichal do you think it's necessary to keep these instructions?
Running beaker experiment create --format=json doesn't change the output (it's still text).
It should output a JSON struct that explains what occurred, maybe something like:
{ "experiment_id": "...", "url": "..." }Or possibly the result of beaker experiment get $id --format=json for the newly created experiment.
https://beaker.org/clusters reports ai2/on-prem-aristo-server1 has 0 jobs running whereas https://beaker.org/cl/us_wvnghctl47k0/01DRYHCNMZD37W759TXABXSBVW reports 1 job running.
Are there any plans to open-source Beaker server side?
Or make it available for in-house deployment in some other way.
UI looks good and integration with AllenNLP is really nice.
It would be great to run it locally and use it for personal and internal projects.
It looks like the latest version is 1.0.19, from February 4?
$ beaker --version
beaker version Beaker 1.0.19 ("68ccc70cffc7550ef79353752f76be99ef56097c")
$ brew upgrade beaker
Warning: allenai/beaker/beaker 1.0.19 already installed
$ brew info beaker
allenai/beaker/beaker: stable 1.0.19
Beaker command-line tool.
https://beaker.allenai.org
/usr/local/Cellar/beaker/1.0.19 (3 files, 12.1MB) *
Built from source on 2021-07-13 at 14:41:59
From: https://github.com/allenai/homebrew-beaker/blob/HEAD/beaker.rbSome commands seem to fail with the current version, e.g.,
$ beaker cluster list maxf
Error: invalid version; please update your clientAt https://github.com/beaker/docs/blob/main/docs/start/install.md, the link at the bottom on the text "your first experiment" 404s.
As a followup question, I noticed that both allenai/beaker and beaker/docs have installation instructions. Does it make sense to de-dup.?
I reproduced this issue in https://github.com/mbforbes/beaker-docker, where I use the Dockerfile from https://github.com/beaker/docs/blob/main/docs/start/run.md. The issue also happens to me when I start from an nvidia/cuda image (nvidia/cuda:11.4.1-cudnn8-runtime-ubuntu20.04).
Copying the output from there:
$ docker --version
Docker version 20.10.8, build 3967b7d
$ docker build -t my-experiment .
# ...
$ docker run --rm -it my-experiment nvidia-smi
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:380: starting container process caused: exec:
"nvidia-smi": executable file not found in $PATH: unknown.
$ docker run --rm -it --gpus all my-experiment nvidia-smi
Mon Aug 23 17:35:13 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.42.01 Driver Version: 470.42.01 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... Off | 00000000:00:04.0 Off | 0 |
| N/A 37C P0 51W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+When trying to run on Beaker, I see the same error:
StartError: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "nvidia-smi": executable file not found in $PATH: unknown
This StackOverflow answer says
Since Docker
19.03, you need to installnvidia-container-toolkitpackage and then use the--gpus allflag.
What is confusing is that @ckarenz was able to run nvidia-smi successfully in another experiment on the same cluster running directly from the base image.
So, does this mean there is a difference in how the images are built that is changing the requirement for --gpus all?
+cc @schmmd
There's several containder security issues that require updating the dependency:
The cluster create command can create clusters in the cloud or on-premise. The decision is made implicitly, where a cluster is created in the cloud if any autoscale resource (CPU, GPU, memory) is set.
We should instead have an explicit flag which is required to pass any of those autoscale parameters, or even a separate command which attaches an autoscale policy to an existing cluster.
There was a problem verifying your default org.
Set the default organization in your config in the format default_org: <org_name>. Note that the name may be different from the name displayed in beaker UI.
Error: : organization not found
I'm wondering if this error lead to the failure in creating the beaker image. And I'm unable to find instructions about how to fix it.
Thanks!
The command to stop a session is beaker session update --cancel <id>. This is a precise transliteration of the REST API, but we should expose something more natural like beaker session stop <id>
We have at least one unresolvable parsing issue with Kingpin (- special flag argument doesn't work). We could contribute a fix, but we've been moving to Cobra in other repositories.
Below are some files to include Cobra without adding Viper and its dependencies.
module github.com/allenai/beaker
// version, imports...
replace github.com/spf13/viper => ./viperstub
// Package viper exists purely to remove the unused dependency from spf13/cobra.
// This reduces cobra's (therefore our) transitive dependencies significantly.
package vipermodule github.com/spf13/viper
go 1.13
Right now, to rerun a task with changed params, you have to change the spec and create a new experiment. There may be a use case to simply rerun the experiment without creating a whole new experiment.
How can I get the authorization to use beaker?
Thank you!
I get this sometimes when I'm click happy:
Hi there,
Beaker's argument parser seems to escape < , i.e., < is translated into <.
I dug into the code. Replacing html/template by text/template might resolve the issue, or unescape the buffer by html.UnescapeString(buf.String()) before https://github.com/allenai/beaker/blob/10b63ac8cf54e6ab2d0cbb13b33868b468bfa7ae/cmd/beaker/experiment.go#L393
When running beaker cluster create -h
You get:
Usage:
beaker cluster create [command]
Available Commands:
cloud Create a new cloud cluster with autoscaling
on-prem Create a new on-premise cluster without autoscaling
It should be
beaker cluster create [type]
The source code appears to be trying to set [type] as well but perhaps there's a cobra bug or something? https://github.com/allenai/beaker/blob/156c4c1c7cba94c78c64a4e9dd6ce91694336aa8/cmd/beaker/cluster.go#L31
It would be really useful to see how utilized each node in a cluster is through the beaker CLI. This would allow us to programmatically select which cluster to run a job on based on the availability of GPUs within each node.
In tango, for example, we're running GPU tests by submitting Beaker jobs via GitHub Actions. Right now we run these tests on a scalable cloud cluster, which is relatively cheap, but would be even cheaper if we could run these jobs on on-prem clusters when they have GPUs available, while only using the cloud cluster as a fallback.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.