#31 #267

Two edge cases:

• Existing page: #61
• Daily Notes: #66

Demo

Link Filter

• get list of links
• implement in three places (see screenshot)
  • page-level
  • linked/unlinked references
  • sidebar for pages and linked/unlinked references
• implement devcards #185 #186

Locations

Original Post

READER BEWARE: this is a rabbithole and perhaps the most exciting part about knowledge graphs. Continue at your own risk.

I'm skeptical any of these would be quite good enough out of the box for the Roam/Athens use case. But at the implementation level, these are some usable libraries today:

• Roam uses cytoscape, I believe.
• Org-roam uses Graphviz.
• More JS libraries: https://neo4j.com/developer/tools-graph-visualization/

In terms of what I want from a graph in general, and not just what is available today, here is a rambling list of stuff:

Read-level features for a global overview, e.g. https://twitterverse.net/:

• can visualize a large number of nodes in a tractable way (Roam already cannot do this), probably with multiple algorithms and views
• many variables could be used as parameters to the algorithm
  • number of inbound and/or outbound links (neighbors)
  • content size: word count or number of blocks
  • content itself (i.e. NLP and sentiment analysis)
• based on these parameters, algorithm would be able to cluster nodes, depict node size, and zoom
• zoom (no not that Zoom!)
  • ZUIs are cool
  • PA3D
  • Eagle Mode
  • Pad++
• perhaps switch between 2D/3D/VR

Read+Write at the local page level:

• page preview of course
• to incrementally expand a page's "peripheral vision" to 1/2/3/N hops away
• to arbitrarily query subgraphs using set/boolean logic. I believe this combined with previous point would feed into complex permissions, e.g. share [[Page X]] and all nodes at most 3 hops away excluding [[Page Y]]

Links, mainly from Discord. Some more relevant than others.

1. visualizations using Wikipedia data
2. Codex for literature analysis
3. dimensonality reduction
4. is it possible that a hypergraph is a better data model
5. noahtren: plotting based on NLP
6. Go Links
7. Hive Plots
   • http://bl.ocks.org/RichMorin/raw/2117857/
   • http://egweb.bcgsc.ca/
8. Les Misérables Co-occurrence matrix
9. Walrus
   • https://www.caida.org/tools/visualization/walrus/
   • https://jovianlin.io/content/images/2018/04/FPTreeWaterVis_Final-3336-2012-04-5-08-38-2.png
10. RemNote Tree & Graph Viz by David Dohan
11. Kantian corpus viz
12. Bible corpus viz
13. Elm lang dependency viz
14. https://www.yworks.com/products/yed
15. http://www.biofabric.org/gallery/pages/SuperQuickBioFabric.html
16. git contributor visualization: https://www.youtube.com/watch?v=P_02QGsHzEQ
17. https://quid.com/
18. https://infranodus.com/
19. Garnet https://news.ycombinator.com/item?id=11230547
20. cola.js https://ialab.it.monash.edu/webcola/
21. https://www.anishathalye.com/2019/12/12/constraint-based-graphic-design/

For v1, the only remaining task is drag and drop to re-order sidebar.

Tasks

• Drag to re-order shortcuts. see #183
• create Athena affordance. Not entirely sure what this interaction would look like
• expand/collapse toggle
• import into actual project - #125
• mock datoms #118
• buttons #119
• - collapsed stick menu #134
• expanded #122
• ~ Appearance - screenshot and explanation below sticking with one appearance for v1
• ~ Recents - UI is same as Shortcuts. Could be persisted in localStorage as described in Appearance now showing recents in Athena: #208
• ~ Settings - TBD what settings will do or what they will look like can't work on this until we have settings

• Colors
• Typography
• import to project #125
• Breakpoints - we will let this emerge, as discussed below

Downhill Tasks (good for contributors)

Sorted easiest to hardest (as I perceive)

• depths / shadows - seems like a very straightforward implementation. Just create various +box-shadow-X functions like [here].(https://github.com/athensresearch/athens/blob/master/src/cljs/athens/style.cljs#L45-L46)
• Icons - #129 #133
• colors & opacities. #148 Explained below

Colors & Opacities

06-08-18 Edit: @shanberg found stylefy lib and is refactoring relevant code in #136. This solves inline pseudoselectors, but not the opacity abstraction out of the box.

Problem

• Hexcodes are a poor way to identify colors because they are machine-, not human-readable.
• HSL is also not great because it doesn't use opacities directly the way rgba or Figma does.
• We want to stick to the design system's opacity levels: 0.1, 0.25, 0.5, 0.75, 1
• Defining a function for each (Color OR Cackground Color) x Opacity leads to many permutations, e.g. +primary-.1 +secondary-.5 +warning-bg-.75...
• Furthermore, with-styles also generates a style map, e.g. {:style {:key "val"}}. But oftentimes these different shades/opacities happen on the hover pseudoselector, which cannot be supplied through inline styles the way we've been doing with with-styles.

Solution?

• need to easily create any permutation of Color x Opacity and Background Color x Opacity
• need to easily merge into a flattened map to add to style-guide-css

Design Language / Style Guide:

If this has already been discussed elsewhere, please feel free to close.

I believe a full stack framework with graphs and graph queries at its core is an ideal fit for this project. It is agnostic to backend DB, but offers a well developed and tested frontend DB and data sync abstraction, as well as a beautiful model for front end composition. A UI with potentially duplicated information such as backlinks and sidebar views would fit fulcro's model well in my opinion.

If this is a direction you wish to explore, I would be interested in spiking out a branch to get to feature parity to current state with fulcro.

Feature

Usually, blocks are rendered as divs and spans. In order to change their contents (i.e. :block/string), you must click on the block. This changes the block to an editable <textarea>. When you click out, the block turns back into divs and spans.

Demo

CONTRIBUTING.md lacks sufficient guidelines. Areas to be improved:

• Clojure styleguide
• dev environment setup instructions
• Git(hub) workflow: how to write issue/commits, how to merge PRs, purpose of labels. See bitcoin and status.im
• add a section on governance (meritocratic, BD), particularly on roles/responsibilites and how decisions are made

This feature is for parsing block-level syntax – e.g. • here is [[a link]] – and rendering it to the correct HTML via Hiccup.

We use the instaparse library.

This file may be useful for nested links: https://github.com/athensresearch/athens/blob/master/data/nested-link.hiccup

#5
#25
#44

Is your feature request related to a problem? Please describe.
This might be a bit too early in the project, but are there plans to make this more performance-friendly? Right now self-hosting takes up more than 500 MB of RAM which is a bit much for someone that don't have the best spec like me.

Describe the solution you'd like
I'm not sure what the solution would be, but certain Electron web apps I've tried can be (relatively) lightweight, so maybe it's possible to make this into one?

Tasks

#183 #221 #224 #209 #226

• tests for all this
• copy and paste
• ctrl-z
• context menu
• block selection
• text selection
• cmd+b bold
• cmd+i italics
• cmd+enter to cycle through TODOs
• [[]] and (()) creates auto-complete (get-caret)
• / creates slash commands menu (get-caret #59 )
• backspace
• drag and drop #241
• tab, shift-tab
• enter
• arrow-keys
• debounce

Original Post

Creating an issue for how to work with parsing the markup: how to turn plaintext like I __very__ much like [[Athens Research]] into a semantic structure that we can understand.

Context & prior work

• Our very own parser.cljs seems to be a bit of a work in progress
• static-roam relies on serial regex replacement, and does not go through an AST

Ideas

I think we'll be able to find an elegant solution to this. Instaparse is powerful. I don't think we should try to "hack some regexes", as that will cause us problems with recursion.

Design proposal:

• Define our AST structure with Clojure Spec. That way, we get validation & generators
• Define symmetric functions so that (= source (-> source serialize deserialize))
• Then we should be able to generate ASTs from Spec generators, run them through a serialize & deserialize sequence, and check that they stay the same.

Hoping that we can reflect the same structure in our Instaparse parser and in our specs.

Thoughts?

Challenges

• How do we represent incomplete ASTS? We need to allow for opening and not closing brackets. Or do we? How do we put I like [[Athens Research (missing ]]) in an AST?

Mostly concerned with writing issues, PRs, and commits clearly for posterity. (I'm bad at this myself.)

See https://github.com/jethrokuan/org-roam/blob/master/.github/CONTRIBUTING.md

Tasks

• get caret position (necessary for in-line search as well with [[]] and (())]) tutorial
• use arrow keys to scroll through options
• typeahead filter that narrows possible options
• on enter or click, expand text
• determine when to show/close typeahead (compare Roam and Notion)
• determine best way to query (by index of input value or separate string?)

Many small interaction / implementation questions. Lots of overlap with inline-search - [[]] (()). Slack and Discord are not great examples because they only allow commands if you start input with a slash.

Original Post

(I've applied for a Sentry opensource license 🤞)

Sentry's error tracking is useful to catch errors in production (artificacts) early. This can monitor regressions and help us determine the general health of the application.

https://sentry.io/organizations/athensresearch/

Keybindings

• ctrl-z : undo
• ctrl-shift-z : redo
• cmd-k : toggle athena
• ctrl-g : toggle devtools
• ctrl-l : toggle left sidebar
• ctrl-r : toggle right sidebar

06-28-2020

Implementation at time of this update. Mapping have not been thought about deeply. But it shouldn't be too difficult to change/customize.

This update is focused on global keybindings. Most other keybindings require block-level context. Thinking about putting block-level keybindings in the #97 issue, because there are many edge cases based on selection, location of block, location of cursor, etc. Copy and paste require some block-specific context, but they are magical so just acknowledging that.

05-15-2020

Most of Roam's keybindings are not super necessary. The important ones that stand out to me are:

• backspace - deletes block if empty
• arrow keys
• indent and shift-indent
• ctrl-o: open link
• ctrl-b: bold
• cmd-z: undo
• cmd-shift-z: undo
• cmd-v: paste (this is its own, lots of magic here)

Do consider how this could fit into a plugin architecture: #63.

Example keybindings

Non-root page example: #/page/AVOHMK6bK

My guess for why this is is that when the app is mounting, the datoms have not yet been loaded from Github. The solution would probably be to show a loading UI, and let all datascript queries wait in a queue until the datoms have been loaded.

I'd like to join in and chat, but the discord link currently in README.md leads to an error page!

Demo

Requirements:

• A transport layer, probably websockets. Sente appears to be the most popular WS library.
• A service layer. Sente supports http-kit, Immutant v2+, nginx-clojure, node.js, Aleph.
• A router: reitit is already being used on the front-end, and is Purely Functional's Top Choice.
• A persistence layer: probably going with local Datomic, which runs on-top of a storage service like PostgresQL, so that we can have the same data model on the backend (Datomic) and frontend (Datascript), which are datoms.
• Assuming Datomic, we need to write synchronization logic or use a library such as datsync.

Note: these libraries/requirements are more of a suggestion and starting point than a final decision. These requirements can and may very well evolve or change.

Primarily interested in creating a sidebar and rendering favorites inside it.

If an entity has a :page/sidebar attribute, it is in the sidebar. The value is an integer that represents the order in the sidebar list, indexed from 0.

Don't need to worry about write operations, e.g. adding or removing shortcuts, or rearranging items in the list. In terms of the links above the shortcuts, don't need to worry about Daily Pages until Phase 2, but there should be a link to All Pages which already exists.

Don't need to worry about showing/hiding the left sidebar, by default just show. Wouldn't mind a logo at the bottom. 😉

Demo

You will probably use the following attributes.

:block/children
:block/open
:block/order

See Datascript FAQ for more background on all attributes.

Demo in Roam

I'm not that familiar with Clojurescript. I downloaded, did yarn install, and was expecting npm start to work - or to find some scripts in package.json, but didn't find any.

Currently, some Roam users download the browser extension roam-toolkit to get additional features such as custom CSS overrides and additional keybindings.

Given that Roam is a text editor, it makes more sense to me that customizations be a native experience. Can we take inspiration from Atom and VS Code to see what a modern plugin architecture looks like? This architecture ideally supports a variety of use cases:

• custom themes
• custom keybindings (e.g. Vim, Emacs)
• arbitrarily executable and composable code

Some Inspirations

• VS Code Extension API
• org-mode
  • scheduling and planning your GTD agenda
  • tracking time
  • native capture templates (as opposed to text expanders)
  • export (to HTML, PDF, LaTex, slideshow)
  • literate programming
• Smallest Federated Wiki
  • data viz with d3.js. With sci.web, we can import arbitrary npm packages on the client. A use case for this would be importing data visualization libraries and having ObservableHQ workflows inside your Athens. See #67.
  • arduino sensor data. Imagine ingesting actionable live data from IFTTT or Zapier into your Athens.
• https://www.dokuwiki.org/plugin:struct
• https://joplinapp.org/plugins/
• roam/css and roam/js
• https://tiddlywiki.com/static/Plugins.html

version

all productions builds e.g. 76d0881 (artifact here)

platform

Production build, Chrome 81.0.4044.138

problem

Page doesn't load

Under water:

For some reason the ?s here in this query is not (always) a string. In dev this doesn't seem to matter or isn't the case

repro

Click on one of the page links or go to index.html#/page/04-02-2020

expected behavior

Display the page (like in dev)

DevCards isolates components so you can iterate on components in isolation from the rest of the app.

It's ClojureScript's Storybook.js, a development-time Design System.

Testable Clojurescript apps by Juxt goes deeper on dev/testing tools.

Also looking into Dirac and Debux.

Tasks

• delete page
• add to shortcuts
• merge pages if editing a non-date page. show confirmation alert #61 #62
• hide linked/unlinked refs if in Daily Pages view #66
• prevent title editing if date page. show warning alert
• button to create a new page? can already create pages in Athena
• filters #65

Most (if not all) of Clojure's options for developments environments allow the option of connecting your editor to the running instance of the program through an editor-integrated REPL. Do we have an option for this?

For athens, this would mean that we could do interactive queries of our roam datascript database from within the editor.

Some references:

• https://www.youtube.com/watch?v=3Ogei0FX4AI
• https://shadow-cljs.github.io/docs/UsersGuide.html#browser-repl
• https://clojure.org/guides/repl/introduction
• https://youtu.be/FihU5JxmnBg

Datascript is set-based so there is no implicit ordering to return values when you query for a collection. Therefore, if a block or a page has :block/children, each of these children will have a :block/order attribute with a natural number value. Only top-level children are being sorted now, but they should be sorted recursively.

See https://github.com/athensresearch/athens/blob/master/src/cljs/athens/subs.cljs#L63-L64

Downhill Tasks (good for contributors)

• datetime formatter for Modified and Created columns. Library discussion: #66 (comment)
• responsive widths. see Collapsed section on #99

UI

Don't worry about showing links within a page.

clj-kondo is helping us to keep code quality high. Right now, there are still many warnings. Let's fix this. Current output can be viewed here

If everything is resolved, the message here will disappear:
https://github.com/athensresearch/athens/runs/656191957#step:4:46

After resolving this, we can configure the linter to fail the build.

As follow up of #15, in tandem or maybe even beforehand, set up CI.

Github Actions (free for opensource) is the easiest, as it doesn't require any extra setup or new accounts.

Roam has been brought to its knees by the influx of new users. This means the documents linked from here to Roam are hard to access. Would it be possible to export them as markdown, and host them locally here on github?

Instaparse is the parsing library Athens and Roam use.

The code for Athens's parser can be found in src/cljs/athens/parser.cljs

I watched these videos to learn about EBNF Syntax, the parsing syntax used by Instaparse.

This blog shows an example of instaparse that might be helpful for the link parsing.

There are many useful test cases for parsing at http://localhost:3000/#/page/fxi42G2zA, which is the, the page [[Nested [[Links]]]]

Nested links

Both Tachyons and Tailwind are more so libraries (i.e. a composable design language) than they are frameworks (e.g. not composable Bootstrap or Bulma). Functional programmers probably like the former more 😄.

Basic markdown syntax: ![description](url)

See the linked md libraries.

Tricky part might be making the HTTP request.

No need to worry about resizing image since that requires writing datoms (e.g. Roam writes the height and width to :block/props). Can do that in Phase 2.

According to OSS Watch, "It is never too soon to define a suitable governance model."

Things this governance document should include:

• Athens's objectives
• the software license (EPL right now) and their implications
• roles and responsibilities, e.g.
  • user
  • contributor
  • maintainer
  • committer
• decision-making process

Model

OSS Watch depicts projects in two dimensions:

• X: BD to Meritocratic
• Y: Cathedral to Bazaar

What matters most to me beyond the categories and labels are transparency, inclusivity, and collective intelligence. I like to think we've done a good job of this so far.

How do we improve ClojureFam? Let's decentralize the process.

Tasks

• calendar button to select any date
• show alert when user tries to edit title
• hide linked/unlinked references if there are none #191
• ~ notional dates
• ~ jump to Timeline
• scroll to create new page #212

Original Post

When trying to run this on Windows with lein dev I'm getting a lot of OutOfMemoryError. When I visit localhost:3000, there's only a blank page (but with the correct tab title and icon). Is my computer simply not powerful enough to run this?

Also, the blank page's source code says this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset='utf-8'>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <title>athens</title>
  </head>
  <body>
    <noscript>
      athens is a JavaScript app. Please enable JavaScript to continue.
    </noscript>
    <div id="app"></div>
    <script src="js/compiled/app.js"></script>
  </body>
</html>

Requested by @anshbansal to test parser functionality.

Resources:

• https://clojurescript.org/tools/testing
• https://shadow-cljs.github.io/docs/UsersGuide.html#_testing
• https://clojure.org/guides/spec#_testing
• https://github.com/lambdaisland/kaocha
• https://github.com/redplanetlabs/specter
• https://github.com/clojure/test.check
• https://www.clojure-toolbox.com/ ctrl-f test

Did I do something wrong in setting it up or is athens really read-only for now?

#132 https://athensresearch.github.io/athens/cards.html#!/athens.devcards.db_boxes

#91 https://athensresearch.github.io/athens/cards.html#!/athens.devcards.sci_boxes

Using sci, allow users to query their database in the browser. Datalog is a very powerful language and could one day be as common knowledge as SQL.

Core meat of this issue is writing logic to scan through user's :node/titles and :block/strings when the input is changed, and showing first N recommendations. Don't worry too much about nice looking dropdown and other UX things. Can be handled by later issues when making user-friendly.

Search with [[]] and (()) are actually subproblems of the search bar, since the search bar queries titles and blocks.

Demo

Tasks

• ~ drag and drop to re-order. We are using re-frame to manage right sidebar state as a hashmap. We want to be using datalog. This will allow us to persist multiple right sidebar states at once (these might be better called "Workspaces" and requires design). As it stands now, this is already better than Roam's. See #183 for an idea of how to re-order indices with datalog.
• fix the flicker on start. The right sidebar has a conditional class based on the open state (see code below). This is in contrast to the left sidebar, which has two completely distinct components based on open state. The good part about the former is that it gives us more control over animations via CSS keyframes. The downside is that we have to write more conditions within the component, which is more complex and is harder to maintain. This seems to be an engineering-design tradeoff, though there may be better workarounds. UPDATE: closed by #238

;; right sidebar 
  [:div (use-style sidebar-style {:class (if open? "is-open" "is-closed")})

...

;; left sidebar
    (if (not @open?)

Original Post

Assigned to @jeroenvandijk

This issue covers importing only Roam's JSON export.

Requires generating :block/uids for pages since Roam's export lacks them... 🤷‍♂️🧐

Also, there are some properties in the JSON that may need to be omitted otherwise the datascript transaction throws an exception. Consider using clojure.spec, specter, or something like that to ignore non-essential attributes for now.

• map JSON fields to attributes, using https://github.com/athensresearch/athens/blob/master/src/cljs/athens/db.cljs#L8-L25
• add to or refactor https://github.com/athensresearch/athens/blob/master/src/cljs/athens/db.cljs#L30-L39
• file upload should accept multiple file formats and shapes of data. Again, something like clojure.spec can be used to determine the shape of the data: https://github.com/athensresearch/athens/blob/master/src/cljs/athens/db.cljs#L57-L63
• Attribute reference: https://www.notion.so/athensresearch/Datascript-Attribute-Table-afd0e5c9556b404880a0b149f87e9d6d

06-07-2020 Update

• so what is the difference between posh and datascript practically? We use both interchangeably in the application