everruler12 / roam2github

Automatic backups from Roam Research into your private GitHub repo

Roam2Github

Click here to view guide on setting up free, unlimited, automatic Roam backups

Click here for extra settings

---

This project was inspired by https://github.com/MatthieuBizien/roam-to-git

Roam-to-git has offered me great peace of mind knowing my Roam data is safe. However, my backups regularly failed with unknown errors multiple times a week. People were emailing me with the same issues, and I couldn't help. Then it got to the point on 2021-01-28 where all my backups were failing. Roam-to-git's creator didn't seem active with addressing the issues, and I don't know enough Python fix his code. So I decided to write my own backup solution from scratch using Node— with clearer logging to make troubleshooting easier.

Differences from roam-to-git

• Uses Node (rather than Python)
• Supports EDN in addition to JSON and Markdown (not formatted markdown though)
• Multiple graph backups in the same repo
• Better error debugging and active support from the developer (Erik Newhard @everruler12) to get your backups running smoothly and error-free

Future Plans

• New, full guide with step-by-step screen recordings
• Update code to run asynchronously, instead of linearly, to cut down on run time
• Use fipp for faster EDN formatting
• EDN support (2021-01-31)
• Multi graph support (2021-02-01)
• Markdown support (2021-02-04)
• Allow setup of public repo for running Actions and committing to private repo for backup, in order to bypass minute limit for private GitHub Actions (2021-02-18)

EDN Backups are live!

The backup has a check to make sure the formatted EDN (which only adds extra linebreaks and indentation) can be parsed back to match exactly with the original before saving it. It will exit with an error if it can't, so you can rest assured that the formatting doesn't mess with the file integrity. I also tested that the formatted EDN can be used to successfully restore graphs.

2021-01-31 It took all day to figure out how to use ClojureScript to prettify EDN. It was a daunting task, never having dealt with Clojure before, much less compiling it into JavaScript. But I did it! This is necessary because the exported EDN data from Roam is all in one line, meaning GitHub would have to save the entire file each time, instead of just the new lines. This would eat up the storage pretty quickly if run every hour, as unchanged notes would be duplicated each time. And you wouldn't be able to see line-by-line changes in the git history.

Multi Graph Backups in Same Repo

You can now backup multiple graphs without having to create a new GitHub repo for each one. Just add them to your R2G_GRAPH Secret in separate lines, or separated by commas.

Markdown support added

2021-02-04 Markdown is now supported. Worked all day to get filename sanitization working. My backup script can even export markdown from the official Roam help database and Roam book clubs error-free! I have added several measures to prevent errors:

• / slashes are replaced with full-width versions ／
• illegal filename characters are replaced with �
• Page titles longer than 255 characters are automatically truncated (though they lose the .md extension)
• no subdirectories
• no blank files
• The logs will list the files that have been renamed or overwritten. The logs no longer display file names, as this would be a privacy concern for the new way to run Actions publicly.

Unfortunate side-effect with markdown backups: files with duplicate names are overwritten (like [[test]] and [[Test]]). (This was also present in roam-to-git)

Separate backup save location and backup script actions

It is possible now to run the script actions from a public repo, to not be limited by 2000 minutes/month, and save the backup to a private repo. Note that Secret names have changed with this update. (The guides at the top have been updated with the new Secret names and main.yml. The old version is still up, but will no longer be updated.)

Support / Donations

If you experience any issues or errors with my backup script, let me know! Either post as a GitHub issue here, or send me a message at my support email:

[email protected]

You can also check here: Common error causes and their solutions

---

Some very generous people have been asking how to donate. If you like my work, I won't refuse your support!

PayPal: https://paypal.me/eriknewhard

Bitcoin (BTC) address: bc1qsa3l8lraa3rjj6wyc7zdlv5z2xnlunppavtxw0

roam2github's Issues

Backup will fail if Quick Capture is not synced

Hi. Thank you very much for making such a useful library.

There seems to be a problem with backup. If there are any Quick Capture Notes that you have not synced, a pop-up will appear on your Roam.

This caused the backup to fail. When I close this window, the backup now succeeds.

Every job was stopped by Checking for "Export All" option.

Logs

Run cd ..

> [email protected] install /home/runner/work/roambackup/roam2github/node_modules/puppeteer
> node install.js


Chromium (818858) downloaded to /home/runner/work/roambackup/roam2github/node_modules/puppeteer/.local-chromium/linux-818858
added 66 packages from 83 contributors and audited 66 packages in 10.854s
found 0 vulnerabilities

2021-02-07 05:05:01.067 R2G Create browser
2021-02-07 05:05:01.364 R2G Login
2021-02-07 05:05:01.367 R2G - Navigating to login page
2021-02-07 05:05:02.537 R2G - Checking for email field
2021-02-07 05:05:02.672 R2G - (Wait 10 seconds)
2021-02-07 05:05:12.676 R2G - Filling email field
2021-02-07 05:05:12.753 R2G - Filling password field
2021-02-07 05:05:12.830 R2G - Checking for "Sign In" button
2021-02-07 05:05:12.836 R2G - Clicking "Sign In"
2021-02-07 05:05:13.575 R2G Login successful!
2021-02-07 05:05:13.575 R2G Open graph s******a
2021-02-07 05:05:13.575 R2G - (Wait 1 second)
2021-02-07 05:05:14.576 R2G - Navigating away to 404 (workaround)
2021-02-07 05:05:14.628 R2G - (Wait 1 second)
2021-02-07 05:05:15.628 R2G - Navigating to graph
2021-02-07 05:05:16.193 R2G - astrolabe spinning...
2021-02-07 05:05:26.431 R2G - astrolabe spinning stopped
2021-02-07 05:05:26.478 R2G Graph loaded!
2021-02-07 05:05:26.478 R2G Export JSON
2021-02-07 05:05:26.483 R2G - Clicking "..." button
2021-02-07 05:05:26.531 R2G - Checking for "Export All" option
Error: The operation was canceled.

Any way to specify which type of backup to download?

Looks like for now you can specify which backup type to download by editing roam2github.js and modifying backup_types.

Easy in the short term but would be nice to have a more friendly way to specify which backups to make. Could help some people if they hit the minute long limit. Perhaps another secret? Something like R2G_BACKUP_TYPES = "json, edn, markdown"

R2G ERROR - EDN formatting error: mismatch with original

I have been seeing roam2git fail for two consecutive nights with the error:

R2G ERROR - EDN formatting error: mismatch with original

I’m reporting this per the instructions in Common error causes.

The backup in question is for a group, the Yak Collective, that believes in working in public. As such, both our Roam graph (https://roamresearch.com/#/app/ArtOfGig) and the backup (https://github.com/The-Yak-Collective/backups) are publicly accessible (note that the repo backs up two graphs; the failure is occurring on the ArtOfGig graph).

Happy to try provide additional information or help test things, though I’m hoping that the public nature of both the graph and the backup will make it easier to diagnose this issue.

multigraph backup goes wrong

I am think you for your excellent work, it help me a lot. work so fine and new never got failed messages as before.

today i tried the new function to backup multigraph, and proceeded show this

2021-02-04 10:05:56.034 R2G - Saving Markdown 2021-02-04 10:05:56.936 R2G Open graph F*********l 2021-02-04 10:05:56.936 R2G - (Wait 1 second) 2021-02-04 10:05:57.938 R2G - Navigating away to 404 (workaround) 2021-02-04 10:05:58.047 R2G - (Wait 1 second) 2021-02-04 10:05:59.048 R2G - Navigating to graph 2021-02-04 10:05:59.949 R2G - astrolabe spinning... Error: The operation was canceled.

just let you know.

Error: Input required and not supplied: token

I walked through the step-by-step instructions and it seemed all straight forward.

my understanding cron “0 * * * *” means top of the hour - and nothing happened.

so I went into the workflow and found the error:

Run actions/checkout@v2 
 with: 
   ssh-strict: true 
   persist-credentials: true 
   clean: true 
   fetch-depth: 1 
   lfs: false 
   submodules: false 
Error: Input required and not supplied: token

in main.yml:

- uses: actions/checkout@v2
   with: 
     repository: ${{ secrets.BACKUP_REPO }}
     token: ${{ secrets.ACCESS_TOKEN }}

I was only prompted to insert 3 keys during setup: ROAM_{EMAIL, PASSWORD, GRAPH}

why is my ACCESS_TOKEN null?
who defines BACKUP_REPO and ACCESS_TOKEN?

I found something called GITHUB_TOKEN - not sure if this is something completely different as I’m a bit of a noob when it comes to workflows (and am finding it neat :)

Is it possible to extract markdown in user specific folder.

I am using your script. But I want to extract markdown to my _notes folder of Jekyll based project. Is it possible ?

No formatting needed. Just extract or move extracted markdown files to specific location.

Thanks a lot in advance.

Google sign-in to Roam

Is it possible to use this if I use my google account to sign in to Roam instead of using email + password?

[ - /roam2github-actions] Run failed: Roam Research backup - main ( - )

Hi Eric, my backup run started failing around 8 hours ago for no apparent reason. I am not aware of any changes that might have led to this. Any ideas about how to troubleshoot? I have it set to run every 30 mins. Thanks in advance

Error today: Node with given id does not belong to the document

After many successful runs, this error happened today:

2021-02-12 18:27:05.568 R2G GitHub Actions path found
2021-02-12 18:27:05.571 R2G Create browser
2021-02-12 18:27:06.308 R2G Login
2021-02-12 18:27:06.362 R2G - Navigating to login page
2021-02-12 18:27:07.647 R2G - Checking for email field
2021-02-12 18:27:10.035 R2G ERROR - Error: Protocol error (DOM.resolveNode): Node with given id does not belong to the document
    at /tmp/roam2github/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:208:63
    at new Promise (<anonymous>)
    at CDPSession.send (/tmp/roam2github/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:207:16)
    at ExecutionContext._adoptBackendNodeId (/tmp/roam2github/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:299:47)
    at ExecutionContext._adoptElementHandle (/tmp/roam2github/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:314:21)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Frame.waitForSelector (/tmp/roam2github/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:846:24)
    at async /tmp/roam2github/roam2github.js:173:13
R2G Exit after: 4.473s
Error: Process completed with exit code 1.

I'm hoping the next run will be fine, but figured you would want to see it anyway.

Operation Times Out

After a successful login and loading at the Astrolabe logo, for whatever reason this thing drops off after 10 minutes. I've never had it take 10 minutes on my own browser, and my graph is 14 MB right now. Log is as follows:

2021-02-26 04:22:16.695 R2G Detected GitHub Actions path
2021-02-26 04:22:16.698 R2G Create browser
2021-02-26 04:22:18.041 R2G Login
2021-02-26 04:22:18.107 R2G - Navigating to login page
2021-02-26 04:22:19.347 R2G - Checking for email field
2021-02-26 04:22:19.458 R2G - (Wait for auto-refresh)
2021-02-26 04:22:21.656 R2G - Filling email field
2021-02-26 04:22:21.746 R2G - Filling password field
2021-02-26 04:22:21.852 R2G - Checking for "Sign In" button
2021-02-26 04:22:21.857 R2G - Clicking "Sign In"
2021-02-26 04:22:22.743 R2G Login successful!
2021-02-26 04:22:22.802 R2G Open graph ***
2021-02-26 04:22:22.802 R2G - Navigating to graph
2021-02-26 04:22:23.695 R2G - astrolabe spinning...
Error: The operation was canceled.

Specific error is:

The job running on runner Hosted Agent has exceeded the maximum execution time of 10 minutes.

Can roam2github give backups of offline graphs?

Can roam2github give backups of offline graphs?

Disk usage is near full.

Hi, Erik, thanks to your script roam is now backed up successfully, and I am relieved from losing data.
Github says yesterday disk usage is coming near to the upper limit.
I am wondering if it is possible to change and reduce the back-up timing of a day or delete old backup automatically.
Should I periodically delete old files by manual?

Error: fatal: couldn't find remote ref refs/heads/main

I followed the setup instructions but when I run my workflow I get this: " Error: fatal: couldn't find remote ref refs/heads/main" attached is a zip of my logs
logs_5.zip

roam2github workflows disable after 60 days

roam2github runs dutifully for 60 days and because I don’t make any changes to the the actions repo, the workflow gets disabled.

The message in my actions tab:

“This scheduled workflow is disabled because there hasn’t been activity in this repository for at least 60 days”.

Is there an easy to prevent this from happening?

I missed an email notification I was supposed to get 7d prior to workflow deactivation - that was 8 months ago and just noticing this now! grin

(sorry I don’t have a screenshot, I thought of this after reenabling the workflow!)

Backup fails when Roam subscription lapses

Hi,

Thanks for creating a stable backup solution for Roam. Here is a specific issue that I have encountered recently: When the Roam subscription lapses, the backup script will start failing on every run. This is probably due to a modal popup with a subscription reminder that appears while opening the Roam database in question. However, it is still possible to export the database, so the problem can likely be worked around. I plan to renew my subscription imminently, but I would be happy to provide the information needed to handle this scenario.

Kind regards,
Ketil

Started seeing consistent backup failures yesterday

It looks to me like the npm install is failing on Chromium, but I haven't spent any time digging into it.

Started 16 hours ago, or around when 016ce8c was merged.

I'm seeing these logs in the Run Backup step

Run cd ..
  cd ..
  git clone -q https://github.com/everruler12/roam2github-test.git roam2github
  cd $_
  npm i
  npm run start -s
  shell: /usr/bin/bash -e {0}
  env:
    R2G_EMAIL: ***
    R2G_PASSWORD: ***
    R2G_GRAPH: ***

> [email protected] install /home/runner/work/roam-backups/roam2github/node_modules/puppeteer
> node install.js


Chromium (818858) downloaded to /home/runner/work/roam-backups/roam2github/node_modules/puppeteer/.local-chromium/linux-818858
added 66 packages from 83 contributors and audited 66 packages in 10.564s
found 0 vulnerabilities

Error: Process completed with exit code 1.

Pretty sure my secrets are correct - the secrets page looks like this

And none of those values have changed in Roam in the last day.