lightning-universe / training-studio_app Goto Github PK
View Code? Open in Web Editor NEWLightning HPO & Training Studio App
Home Page: https://lightning-ai.github.io/lightning-hpo/
License: Apache License 2.0
Lightning HPO & Training Studio App
Home Page: https://lightning-ai.github.io/lightning-hpo/
License: Apache License 2.0
View the logs for a particular sweep or notebook
Following example_4 the trainer app
The database tab is not loading
Steps to reproduce the behavior:
1- follow readme to install package and requirements
2- run lightning run app examples/4_app_sweeper.py
the page shows a spinner
There's no CI setup in the repo and the package building
Steps to reproduce the behavior:
conda
, pip
, source):Right now, the Optimizer API requires to be passed a BaseObjective Work. However, this isn't 100 % required in most case.
Instead, we could have this alternative API.
Optimizer(
framework="pytorch_lightning|keras|tensorflow|sklearn"
distributions={"lr": Uniform(0, 1)}
)
TODOS:
lightning_hpo/commands/tensorboard/run.py
run_tensorboard_handler
in the TensorboardController
API:
lightning run tensorboard --sweep_id=..
Started a tensorboard server with id: {,,,}
lightning stop tensorboard {TENSORBOARD_ID}
Refused to frame 'https://wandb.ai/' because an ancestor violates the following Content Security Policy directive: "frame-ancestors 'self'".
lightning run app app.py --cloud --env xxxx --env xxx
TODOS:
Command to list sweeps / notebooks with their status
Something like:
first connect to app, then:
lightning cmd show sweeps
TODOS:
API:
lightning run notebook ... --drives {S3_URL}:{MOUNT_DIR} (--env {AWS_CREDENTIALS})
Launch a notebook from the CLI:
As a user, I want to list of the available notebooks with who created them.
TODOS:
Pseudo-Code:
class NotebookController(...):
def list_notebooks_handler(self) -> List[str]:
return list(self.notebooks.keys())
def configure_commands(self):
return [{"list notebooks": self.list_notebooks_handler}]
int HPO param is passed as float causing jsonparsearg to reject --trainer.max_epochs=3.0
train.py: error: Configuration check failed :: Parser key "trainer.max_epochs": Value "3.0" does not validate against any of the types in typing.Union[int, NoneType]:
- Expected a <class 'int'> but got "3.0"
- Expected a <class 'NoneType'> but got "3.0"
train.py: error: Configuration check failed :: Parser key "trainer.max_epochs": Value "9.0" does not validate against any of the types in typing.Union[int, NoneType]:
- Expected a <class 'int'> but got "9.0"
- Expected a <class 'NoneType'> but got "9.0"
lightning run app examples/2_app_pytorch_lightning.py
Wandb recently released a new way to generate sweep graphs using their API Client.
api = wandb.Api()
wandb.require('report-editing:v0')
report = api.create_report(project='pytorch-sweeps-demo')
report.title = 'A fabulous title'
report.description = 'A descriptive description'
panel_grid = wb.PanelGrid()
run_set = wb.RunSet()
run_set.entity = '_scott'
run_set.project = 'pytorch-sweeps-demo'
panel_grid.runsets = [run_set]
coords = wb.ParallelCoordinatesPlot(columns=[wb.reports.PCColumn('batch_size'), wb.reports.PCColumn('epoch'), wb.reports.PCColumn('loss')])
panel_grid.panels = [coords]
run_set.set_filters_with_python_expr('User == "_scott"')
report.blocks = [panel_grid]
report.save()
api = wandb.Api()
project = 'pytorch-sweeps-demo'
wandb.require('report-editing:v0')
run_set = wb.RunSet(project=project)
coords = wb.ParallelCoordinatesPlot(columns=[wb.reports.PCColumn('batch_size'), wb.reports.PCColumn('epoch'), wb.reports.PCColumn('loss')])
panel_grid = wb.PanelGrid(runsets=[run_set], panels=[coords])
report = api.create_report(project='pytorch-sweeps-demo', blocks=[panel_grid])
report.save()
``
### Motivation
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
### Pitch
<!-- A clear and concise description of what you want to happen. -->
### Alternatives
<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
Add __init__.py in lightning_hpo.algorithm and lightning_hpo.distribution to support installing with
pip install git+https://github.com/Lightning-AI/lightning-hpo.git
Enable use in personal projects and begin working towards full release on pypi.
Add blank __init__.py in both directories.
Leave as is.
None
Release lightning-notebook to PiPy so we can use its component.
Adding a notebook through the UI will cause an error if the works aren't ready yet. We should probably have a spinner / countdown in the app which waits for the works to become ready before letting you in
Steps to reproduce the behavior:
conda
, pip
, source):Command to download an artefact after having listed them
TODOS:
TODOS:
TODOS:
API:
lightning run sweep ... --drives {S3_URL}:{MOUNT_DIR} (--env {AWS_CREDENTIALS})
TODO: Add support for multi include or exclude.
conda
, pip
, source):In the More
dropdown for notebooks, should be a button to stop / start the notebook
TODOS:
Fixes #69
TODOS:
class Tensorboard(LightningWork):
def run(self, drive):
Popen(f"tensorboard --log_dir={drive.url}", env=os.environ)
Add the start time (since linux epoch) to the NotebookConfig. This would enable displaying the notebook runtime in the UI
error:
[flow] 2022-09-01T08:53:04.580Z ERROR: Found an exception when loading your application from examples/1_app_agnostic.py. Please, resolve it to run your app.
[flow] 2022-09-01T08:53:04.580Z Traceback (most recent call last):
[flow] 2022-09-01T08:53:04.580Z File "examples/1_app_agnostic.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.580Z from lightning_hpo import Sweep
[flow] 2022-09-01T08:53:04.580Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/__init__.py", line 22, in <module>
[flow] 2022-09-01T08:53:04.580Z from lightning_hpo.components.sweep import Sweep # noqa: E402
[flow] 2022-09-01T08:53:04.580Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/components/sweep.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.580Z from lightning.app.components.python.tracer import Code
[flow] 2022-09-01T08:53:04.580Z ImportError: cannot import name 'Code' from 'lightning.app.components.python.tracer' (/content/venv/lib/python3.8/site-packages/lightning/app/components/python/tracer.py)
[flow] 2022-09-01T08:53:04.583Z ERROR: Found an exception when loading your application from examples/1_app_agnostic.py. Please, resolve it to run your app.
[flow] 2022-09-01T08:53:04.583Z Traceback (most recent call last):
[flow] 2022-09-01T08:53:04.583Z File "examples/1_app_agnostic.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.583Z from lightning_hpo import Sweep
[flow] 2022-09-01T08:53:04.583Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/__init__.py", line 22, in <module>
[flow] 2022-09-01T08:53:04.583Z from lightning_hpo.components.sweep import Sweep # noqa: E402
[flow] 2022-09-01T08:53:04.583Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/components/sweep.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.583Z from lightning.app.components.python.tracer import Code
[flow] 2022-09-01T08:53:04.583Z ImportError: cannot import name 'Code' from 'lightning.app.components.python.tracer' (/content/venv/lib/python3.8/site-packages/lightning/app/components/python/tracer.py)
[flow] 2022-09-01T08:53:04.692Z ERROR: Found an exception when loading your application from examples/1_app_agnostic.py. Please, resolve it to run your app.
[flow] 2022-09-01T08:53:04.692Z Traceback (most recent call last):
[flow] 2022-09-01T08:53:04.692Z File "examples/1_app_agnostic.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.692Z from lightning_hpo import Sweep
[flow] 2022-09-01T08:53:04.692Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/__init__.py", line 22, in <module>
[flow] 2022-09-01T08:53:04.692Z from lightning_hpo.components.sweep import Sweep # noqa: E402
[flow] 2022-09-01T08:53:04.692Z File "/content/venv/lib/python3.8/site-packages/lightning_hpo/components/sweep.py", line 5, in <module>
[flow] 2022-09-01T08:53:04.692Z from lightning.app.components.python.tracer import Code
[flow] 2022-09-01T08:53:04.692Z ImportError: cannot import name 'Code' from 'lightning.app.components.python.tracer' (/content/venv/lib/python3.8/site-packages/lightning/app/components/python/tracer.py)
[flow] 2022-09-01T08:53:05.113Z INFO: Found dead components ['Flow', 'Server', 'root.hi_plot'], Exiting execution!!!
lightning run app examples/1_app_agnostic.py --cloud
Command to list artefacts with filtering by sweep or notebook
TODOS:
lightning_hpo/commands/artefacts/view.py
. Simply use the LightningClient and list the artifacts. Use this reference.Rich Tree example:
# https://github.com/Textualize/rich/blob/master/examples/tree.py
"""
Demonstrates how to display a tree of files / directories with the Tree renderable.
"""
import os
import pathlib
import sys
from rich import print
from rich.filesize import decimal
from rich.markup import escape
from rich.text import Text
from rich.tree import Tree
def walk_directory(directory: pathlib.Path, tree: Tree) -> None:
"""Recursively build a Tree with directory contents."""
# Sort dirs first then by filename
paths = sorted(
pathlib.Path(directory).iterdir(),
key=lambda path: (path.is_file(), path.name.lower()),
)
for path in paths:
# Remove hidden files
if path.name.startswith("."):
continue
if path.is_dir():
style = "dim" if path.name.startswith("__") else ""
branch = tree.add(
f"[bold magenta]:open_file_folder: [link file://{path}]{escape(path.name)}",
style=style,
guide_style=style,
)
walk_directory(path, branch)
else:
text_filename = Text(path.name, "green")
text_filename.highlight_regex(r"\..*$", "bold red")
text_filename.stylize(f"link file://{path}")
file_size = path.stat().st_size
text_filename.append(f" ({decimal(file_size)})", "blue")
icon = "๐ " if path.suffix == ".py" else "๐ "
tree.add(Text(icon) + text_filename)
try:
directory = os.path.abspath(sys.argv[1])
except IndexError:
print("[b]Usage:[/] python tree.py <DIRECTORY>")
else:
tree = Tree(
f":open_file_folder: [link file://{directory}]{directory}",
guide_style="bold bright_blue",
)
walk_directory(pathlib.Path(directory), tree)
print(tree)
โ โ โ โ โ โโโ ๐ test_scripts.cpython-38-pytest-7.1.2.pyc (2.1 kB)
โ โ โ โ โฃโโ ๐ __init__.py (0 bytes)
โ โ โ โ โโโ ๐ test_scripts.py (1.1 kB)
โ โ โ โโโ ๐ __init__.py (0 bytes)
โ โ โฃโโ ๐ core_features_app
โ โ โ โฃโโ ๐ __pycache__
โ โ โ โ โโโ ๐ app.cpython-38.pyc (845 bytes)
โ โ โ โโโ ๐ app.py (422 bytes)
โ โ โฃโโ ๐ custom_work_dependencies
โ โ โ โฃโโ ๐ __pycache__
โ โ โ โ โฃโโ ๐ __init__.cpython-38.pyc (193 bytes)
โ โ โ โ โโโ ๐ app.cpython-38.pyc (2.7 kB)
โ โ โ โฃโโ ๐ __init__.py (0 bytes)
โ โ โ โโโ ๐ app.py (2.2 kB)
โ โ โฃโโ ๐ idle_timeout
โ โ โ โฃโโ ๐ __pycache__
โ โ โ โ โฃโโ ๐ __init__.cpython-38.pyc (181 bytes)
โ โ โ โ โโโ ๐ app.cpython-38.pyc (2.8 kB)
โ โ โ โฃโโ ๐ __init__.py (0 bytes)
โ โ โ โโโ ๐ app.py (2.8 kB)
โ โ โฃโโ ๐ __init__.py (0 bytes)
โ โ โฃโโ ๐ conftest.py (1.2 kB)
โ โ โฃโโ ๐ layout.py (544 bytes)
โ โ โฃโโ ๐ pickle_or_not.py (565 bytes)
โ โ โฃโโ ๐ test_argparse.py (1.8 kB)
โ โ โฃโโ ๐ test_boring_app.py (1.1 kB)
โ โ โฃโโ ๐ test_collect_failures.py (1.2 kB)
โ โ โฃโโ ๐ test_commands_and_api.py (1.5 kB)
โ โ โฃโโ ๐ test_core_features_app.py (647 bytes)
โ โ โฃโโ ๐ test_custom_work_dependencies.py (675 bytes)
โ โ โฃโโ ๐ test_drive.py (541 bytes)
โ โ โฃโโ ๐ test_idle_timeout.py (573 bytes)
โ โ โฃโโ ๐ test_multi_node.py (746 bytes)
โ โ โฃโโ ๐ test_payload.py (514 bytes)
โ โ โฃโโ ๐ test_quick_start.py (2.4 kB)
โ โ โฃโโ ๐ test_template_react_ui.py (1.0 kB)
โ โ โฃโโ ๐ test_template_streamlit_ui.py (1.1 kB)
โ โ โโโ ๐ test_v0_app.py (2.4 kB)
Stop notebook given the UID.
You are connected to the local Lightning App.
๐ root
โฃโโ ๐ artifacts
โ โโโ ๐ drive
โ โโโ ๐ code
โ โโโ ๐ root.file_server
โ โโโ ๐ thomas-0ed22042
โฃโโ ๐ sweep_controller.sweeps.thomas-1fdc34f9.w_0.ws.0
โ โโโ ๐ Users
โ โโโ ๐ thomas
โ โโโ ๐ Documents
โ โโโ ๐ GitHub
โ โโโ ๐ LAI-lightning-hpo-App
โ โโโ ๐ wandb
โ โโโ ๐ lightning_logs
โ โโโ ๐ trial_0
โ โโโ ๐ 1z93ywba
โ โโโ ๐ checkpoints
โ โโโ ๐ epoch=0-step=20.ckpt
โฃโโ ๐ w_0.ws.0
โ โโโ ๐ Users
โ โโโ ๐ thomas
โ โโโ ๐ Documents
โ โโโ ๐ GitHub
โ โโโ ๐ LAI-lightning-hpo-App
โ โโโ ๐ wandb
โ โโโ ๐ lightning_logs
โ โโโ ๐ Optimizing a Simple CNN over MNIST with Lightning HPO
โ โโโ ๐ version_None
โ โโโ ๐ checkpoints
โ โฃโโ ๐ epoch=0-step=20-v7.ckpt
โ โโโ ๐ epoch=1-step=40.ckpt
โฃโโ ๐ w_0.ws.1
โ โโโ ๐ Users
โ โโโ ๐ thomas
โ โโโ ๐ Documents
โ โโโ ๐ GitHub
โ โโโ ๐ LAI-lightning-hpo-App
โ โโโ ๐ lightning_logs
โ โฃโโ ๐ version_11
โ โ โโโ ๐ checkpoints
โ โ โโโ ๐ epoch=0-step=20.ckpt
โ โโโ ๐ version_12
โ โโโ ๐ checkpoints
โ โโโ ๐ epoch=1-step=40.ckpt
โฃโโ ๐ w_1.ws.0
โ โโโ ๐ Users
โ โโโ ๐ thomas
โ โโโ ๐ Documents
โ โโโ ๐ GitHub
โ โโโ ๐ LAI-lightning-hpo-App
โ โโโ ๐ wandb
โ โโโ ๐ lightning_logs
โ โโโ ๐ Optimizing a Simple CNN over MNIST with Lightning HPO
โ โโโ ๐ version_None
โ โโโ ๐ checkpoints
โ โโโ ๐ epoch=0-step=20-v9.ckpt
โโโ ๐ w_1.ws.1
โโโ ๐ Users
โโโ ๐ thomas
โโโ ๐ Documents
โโโ ๐ GitHub
โโโ ๐ LAI-lightning-hpo-App
โโโ ๐ lightning_logs
โโโ ๐ version_13
โโโ ๐ checkpoints
โโโ ๐ epoch=0-step=20.ckpt
Steps to reproduce the behavior:
conda
, pip
, source):When a notebook is stopped, all its artifacts are stored in the persistent storage. When restarting a notebook, we can simply re-download them all within the work.
TODOS:
There is duplicate code in lightning_hpo.loggers.wandb
at ln47
Download the logs for a particular sweep
TODOS:
lightning_hpo/commands/sweep/delete.py
and add a delete_sweep_handler in the SweeperController. If the Sweep exists, we need to:In the More
dropdown for notebooks should be a button to open the notebook in a tab
Needs a way to give the TB work access to the logs for that run / sweep (i.e. may need to merge multiple logs folders).
Add a few (3-4) pages with a basic explanation of this app...
Following example_4 the trainer app
The dashboard is not loading
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
Traceback:
File "/Users/mansy/src/lightning-hpo/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 556, in _run_script
exec(code, module.__dict__)
File "/Users/mansy/src/lightning-hpo/venv/lib/python3.9/site-packages/lightning_app/frontend/streamlit_base.py", line 35, in <module>
main()
File "/Users/mansy/src/lightning-hpo/venv/lib/python3.9/site-packages/lightning_app/frontend/streamlit_base.py", line 31, in main
render_fn(flow_state)
File "/Users/mansy/src/lightning-hpo/lightning_hpo/controllers/sweeper.py", line 106, in render_fn
resp = requests.get(state.db_url + "/general/", data=GeneralModel.from_cls(SweepConfig).json())
Steps to reproduce the behavior:
1- follow readme to install package and requirements
2- run lightning run app examples/4_app_sweeper.py
the dashboard shows
Creating a Shared Drive objects already works within a notebook through the AppState, but this won't worked with s3 Mount Drive.
from lightning.app.utilities.state import AppState
state = AppState()
print(state.component_name.drive.list())
Provide utilities to a list of app drives to help users investigate their artifacts
TODOS
References:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.