ploomber / debuglater

Store Python traceback for later debugging. 🐛

Tip

Deploy AI apps for free on Ploomber Cloud!

Join our community | Newsletter | Contact us | Docs | Blog | Website | YouTube

Ploomber is the fastest way to build data pipelines ⚡️. Use your favorite editor (Jupyter, VSCode, PyCharm) to develop interactively and deploy ☁️ without code changes (Kubernetes, Airflow, AWS Batch, and SLURM). Do you have legacy notebooks? Refactor them into modular pipelines with a single command.

Installation

Compatible with Python 3.7 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Try the tutorial:

Community

• Join us on Slack
• Newsletter
• YouTube
• Contact the development team

Main Features

⚡️ Get started quickly

A simple YAML API to get started quickly, a powerful Python API for total flexibility.

get-started.mp4

⏱ Shorter development cycles

Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last execution.

shorter-cycles.mp4

☁️ Deploy anywhere

Run as a shell script in a single machine or distributively in Kubernetes, Airflow, AWS Batch, or SLURM.

deploy.mp4

📙 Automated migration from legacy notebooks

Bring your old monolithic notebooks, and we’ll automatically convert them into maintainable, modular pipelines.

refactor.mp4

I want to migrate my notebook.

Show me a demo.

Resources

• Documentation
• PyData Chicago talk (covers motivation and demo)
• Develop and deploy an ML pipeline in 30 minutes (EuroPython 2021)
• Guest blog post on the official Jupyter blog
• Examples (Machine Learning pipeline, ETL, among others)
• Blog
• Comparison with other tools
• More videos

About Ploomber

Ploomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.

Whatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!

Click here to know how you can contribute to Ploomber.

debuglater's Issues

IPython integration

To integrate this with papermill, we need to modify the IPython debugging feature, so it stores the dump instead of starting a pdb session. Here's how to patch the debugger:

import types
import IPython
term = IPython.get_ipython()

def debugger(self, force: bool = False):
    print('hello!')

term.debugger = types.MethodType(debugger, term)

Then run:

%pdb on

1/0

Reference

the function that is executed when an error happens

the pdb magic logic

the line that starts the debugger

We don't need a magic since we'll inject this code on behalf of the user, something like this:

import pydump
pydump.patch_ipython()

But it'd be nice for people using Jupyter interactively to have this feature. We can also do a magic like:

%pydump on

serializing any exception

the current implementation (unless using IPython/Jupyter) requires users to add a try ... catch ... statement. We got a question if it's possible to serialize non-handled exceptions. I see two ways of approaching this:

1. patch the exception hook

We can patch Python's section hook; this will allow us to serialize any exception without the user having to do to anything:

from debuglater import patch_excepthook
patch_excepthook()

Note: I think this would affect IPython as well, so perhaps we can get rid of the patch_ipython

2. decorator

Alternatively, users could decorate functions in case they only want to serialize exceptions in certain places:

@debuglater
def something():
    pass

Hertem

Hertem

pass traceback object to allow catching all exceptions using sys.excepthook

If you add a parameter tb in function run in pydump.py:

def run(filename, echo=True, tb=None):
    out = Path(filename).with_suffix('.dump')
    if echo:
        print(Fore.RED + f'Exception caught, writing {out}\n')
    save_dump(out, tb)
    if echo:
        print(f'To debug, run:\n  dltr {out}')
        print(Style.RESET_ALL)

then you can catching all exceptions like:

import sys

def my_except_hook(exctype, value, traceback):
    if exctype == KeyboardInterrupt:
        sys.exit("Ctrl-C pressed")
    else:
        import debuglater
        debuglater.run(__file__, tb=traceback)
        sys.__excepthook__(exctype, value, traceback)

sys.excepthook = my_except_hook


if __name__ == '__main__':
    x = 1
    y = 0
    x / y

validate cli arg

validate that the argument passed to the CLI has .dump extension - otherwise users might pass the source file and get a weird pickle error message