petterik / lajt Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 0.0 215 KB

DataScript parser for om.next like libraries

License: Eclipse Public License 1.0

Clojure 99.51% Shell 0.49%

lajt's People

Contributors

Stargazers

Watchers

lajt's Issues

Implement dependencies for lajt.read.ops

Checkout stuartsierra/dependency: https://github.com/stuartsierra/dependency

It's what stuartsierra/component is using to resolve dependencies.

Check the notes app for more stuff

Split parsing and calling of reads/mutates apart

Parsing can be split apart to a function that takes a query and returns a data structure that reads and mutates will be called on.
Pro: This data structure can be changed other functions, such as adding or removing reads.

Spec recursive queries

Test to make sure we're not running the same plugin more than once

When our parser is called recursively, make sure our plugins aren't re-initialized.

Return the remote query when :target is set

Handle remote returns.

Async/suspended parsing/rendering

Could we achieve suspended rendering, where we only return the new values if everything (every read) is committed?

Index the pull pattern if it's in the find pattern

Use case:

(defmethod lajt-read :query/current-route
  [_]
  {:query '{:find  [(pull ?e [:ui.singleton.routes/current-route
                              :ui.singleton.routes/route-params
                              :ui.singleton.routes/query-params]) .]
            :where [[?e :ui/singleton :ui.singleton/routes]]}
   :after [:result #(clojure.set/rename-keys % {:ui.singleton.routes/current-route :route
                                                :ui.singleton.routes/route-params  :route-params
                                                :ui.singleton.routes/query-params  :query-params})]})

Make the whole read-map be dynamic

Let the values of the lajt-reads be a function taking the env. Right now it's defined as maps everywhere. Example where the query changes depending on params:

(defmethod lajt-read :query/store-items
  [_]
  (fn [env]
    (merge-with merge
                {:remote true
                 :query  {:find '[[?e ...]]}}
                (if (seq (get-in env [:route-params :navigation]))
                  {:query  '{:where [[?s :store/items ?e]
                                     [?e :store.item/section ?n]
                                     [?n :store.section/path ?p]
                                     [?e :store.item/name _]]}
                   :params {'?s [:route-params :store-id]
                            '?p [:route-params :navigation]}}
                  {:query  '{:where [[?s :store/items ?e]
                                     [?e :store.item/name _]]}
                   :params {'?s [:route-params :store-id]}}))))

sulolive tests timeout

lajt

git checkout -b petter/sulo-integration
mvn install

sulolive:

git checkout -b petter/lajt-integration
lein repl
eponai.repl=> (start-reloading)
eponai.repl=> (require '[eponai.fullstack2.tests])
eponai.repl=> (clojure.test/run-tests 'eponai.fullstack2.tests)

Implement optional caching for each op's stage

Datascript queries should take advantage of caching, trying to avoid querying or trying to incrementally update the past result.

I'd be really nice if pull also could use caching to try to incrementally update its result.

The goal is to have a fast re-read of all reads, making it possible to always perform re-render from the root-query (making other parts of the UI infra easier to implement).

Spec the read-map

Rewrite sulo reads such that they re-use other queries

See :query/store :query/store-items and :query/store-item-count.
There may be more reads like this.

Optimize localized ui re-render

When lajt discovers changes in the ::pull op, make it easy to get the focused query for the data that has changed, such that one can map the focused query to the deepest component in the UI tree.

Example:

;; PeopleList query:
[{:people (om/get-query PersonItem)}]
;; PersonItem query:
[:person/primary-email]

If PeopleList has rendered some people and the :person/primary-email changes for any of those people, the ::pull op should recognize that [:person/primary-email] in query [{:people [:person/primary-email]}] belongs to a component of PersonItem, and it should be able to re-render the specific one that has changed (maybe by ident?).

Define dispatches on the keys in the map such that it can be extended

Index :where clauses with (some common) functions

Use case:

(defmethod lajt-read :query/categories
  [_]
  {:query '{:find [[?e ...]]
            :where [[?e :category/path _]
                    ;; TODO lajt: Must be able to index where clauses
                    [(missing? $ ?e :category/_children)]]}})

If the indexing stuff doesn't recognize the function, it can just bail out from caching?

Let queries have outer query-params

Let queries look like this:

([:read] {:param-that-goes-into-env 1})

It's like a query-context.

Prefer interceptors/plugins over middleware

Parser, read and mutate middlewares should be split out into interceptors.

Either a function

(fn 
  ([env k p] :before)
  ([env k p result] :after))

Or a map

{:before (fn [env k p])
 :after  (fn [env k p]) ;; with result in env?

Add dependent's where clauses into the pull pattern

Introduce :custom when no :query can be made

Sometimes there's no good way to do a query or lookup-ref. Not sure what to do about caching.

Use case:

Use case:
```clj
(defmethod lajt-read :query/navigation
  [_]
  {:remote     true
   :depends-on (fn [{:keys [query]}]
                 [{:query.navigation/genders query}
                  {:query.navigation/categories query}])
   :custom     (fn [env]
                 (vec
                   (concat
                     (assoc-category-hrefs
                       (get-in env [:depends-on :query.navigation/genders]))
                     (assoc-category-hrefs
                       (get-in env [:depends-on :query.navigation/categories])))))})

Join and Union query-plugins

These plugins would flatten the query at the joins and unions.

If these plugins are run before dedupe-query-plugin, then that plugin wouldn't need the conf/read-plugins. The dedupe thing would be much simpler. Could even use the lazy-parser-plugin directly.

Pull type is a closed system

For every new action, it needs to add itself to the ::pull code.

Open it up some how?

Execute queries only when all params are set

It was common in sulo that we did a when-let around the route params like this:

(defmethod client-read :query/store
  [{:keys [db query target route-params] :as env} _ _]
  (when-let [store-id (:store-id route-params)]
  ;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    (if target
      {:remote true}
      {:value (db/pull db query store-id)})))

What if we only execute the query if none of the values are set to nil? The lajt query for this example would be:

(defmethod lajt-read :query/store
  [_]
  {:remote true
   :query '{:find [?e .]}
   :params {'?e [:route-params :store-id]}})

Introduce :after

Implement :after a key that's called with the result after keys like :query or :lookup-ref.

Not sure what to do about caching 🤔

Example:

(defmethod lajt-read :query.navigation/categories
  [_]
  {:query (db/merge-query
            '{:find [?top]}
            (products/category-names-query {:top-category ["home" "art"]}))
   :after [:result (partial mapv assoc-category-hrefs)]})

Send only reads with (:target env) true, returned by mutate's :remote

Use case:

{:mutate 
  (fn [env k p] 
    (when (and (= 'foo/conj k) (:target env))
      [true :query/foo]))
 :read
  (fn [env k p]
    (when (and (= :query/foo k) (:target env))
      false))}

((parser *1) {} '[(foo/conj)] :remote)
;; should equal '[(foo/conj)]
;; not '[(foo/conj) :remote]
;; because :query/foo is not a remote read
;; even though the mutate suggested that it is.

Strategy for scalability with multiple peer updates

I've been skimming through resources/private/notes/initial-idea.md. I've completely possible I've missed something, but what kind of mechanism do you plan to use to support multiple peers updating the same entity? (some are listed here)

Anyway, the general idea looks very promising, I'd love to get updates from this project :-)

UPD: err, by the way, do you know about replikativ/replikativ?

Include :sort's :key-fn in remote pull-pattern

When sorting on a :key-fn and it's a keyword, we'll want to make sure that it's in the remote pull-pattern.
Use-case:

(defmethod lajt-read :query/store-vods
  [_]
  {:remote true
   :query  '{:find  [[?e ...]]
             :where [[?e :vod/store ?store-id]]}
   :params {'?store-id [:route-params :store-id]}
   :sort   {:order :decending
            :key-fn :vod/timestamp}})

Merge queries with recursion

Get expound and clojure.spec to work with stuff

I haven't gotten any value out of spec, but I've written spec.

Start using expound to see where specs or tests go wrong.

Cache :query's :remote

Use core.cache?

Create a function memoize-1, using a :lifo checking only the previous inputs.

Since most :query's are static, we can cache the input.

Introduce mutations

:depends-on can pass pull-patterns

Use case:

Use case:
```clj
(defmethod lajt-read :query/navigation
  [_]
  ;; ...
  {:depends-on (fn [{:keys [query]}]
                 [{:query.navigation/genders query}
                  {:query.navigation/categories query}])
  ;; ...
  })

Remote key should be able to return a vector with queries to run

Use case:

(defmethod lajt-read :query/stripe-country-spec
  [_]
  {:remote [:query/stripe-country-spec]
   :query  '{:find  [?e .]
             :where [[?e :country-spec/id]]}})

The :query/stripe-country-spec when going remote, shouldn't have a query (for some reason).
In addition to just specifying the queries as a vector, it should be able to be a function taking env, so it has access to the query passed to it.

Modifying the remote query with spec would also be nice, most common use case in sulo being to add a parameter to the query (before we realized we could send all the :route-params).

Support arbitrary functions in the find-pattern

Pattern found in sulo is that a query depends on another read and executes a function on those entities. Example:

(defmethod lajt-read :query/store-item-count
  [_]
  {:remote     true
   :depends-on [:query/store-items]
   :query {:find '[(count ?e) .]}
   :params {'[?e ...] [:depends-on :query/store-items]}})

Find patterns should be able to contain these function calls. Also, I think every case that followed this pattern wanted to do a count on the entities. Might be needed to have a :count key? Dunno.

read, mutate, query plugins should provide some way to prevent the action

In petterik/lajter I found myself wanting to not execute a read. (Reads do not need to be executed when doing transact!).

Rename :dispatched-by -> :expr

Dependencies should be included when there's a :target

Use case

{:common-read {:remote true}
 :read1       {:remote     true
               :depends-on [:common-read]}
 :read2       {:remote     true
               :depends-on [:common-read]}}
;; read:
[:read1 :read2]
;; with :target :remote
[:common-read :read1 :read2]

Macro for :base and :case

The spec for :case is quite gross to write for hand. It'd be nice with a macro.

Update readme with goals

Add goals with check boxes (something like: ☑️ ⏹)

Where we are and where we want to go. Include sulo integration.

Implement :initially

Fulcro allows components to define their initial-state. For lajt, I think it makes sense to put this logic in the parser/reads, as the parser is the thing that deals with all the data.

Fulcro mostly doesn't need to deal with a parser(?), so it puts the initial-state with the component, which makes sense to me.

I don't think this feature is needed too much when one's got server side rendering (SSR), as the SSR will render the actual initial state. But, for both testing, mock UI and apps without SSR, initial state makes sense.

It'd be nice if there was a lajt.read.op for :initially so one could define a query like this:

{:query '{:find [[?e ...]]
          :where [[?e :person/first-name _]]}
 :initially ...
;; oh wait. Doh.

This approach is weird because one doesn't know what the pull-pattern is like. The component knows this, which is why this logic should be put there? Ah..

Re-write the ops and how they're executed

Notes from the WIP implementation:

(def-operation :sort
  :op.stage/setup
  (fn [env value]
    )
  :op.stage/transform
  :op/dependents [:whatever]
  (fn [env value]
    ))

{:op/key    :sort
 :op/stages [{:op/stage      :op.stage/setup
              :op/dependents []
              :op/depends-on []
              :op/fn         (fn [env v])}
             {:op/stage      :op.stage/transform
              :op/dependents []
              :op/depends-on []
              :op/fn         (fn [env v])}]}
;; This is all we need for an op right now.
;; Need to know when it should be run (type).
;; We need to know if it should depend on other ops.
;; Need to know if it has other ops depending on it (dependents).
;; We need to be able to call the op.

Remove :read and :mutate from env

Implement :base and :case keys

One pattern found in sulo was that sometimes the query changes depending on which params are present. Example:

(defmethod client-read :query/store-items
  [{:keys [db query target route-params]} _ _]
  (let [{:keys [store-id navigation]} route-params]
    (when (some? store-id)
      (if target
        {:remote true}
        {:value (do
                  (db/pull-all-with db query (if (not-empty navigation)
                                               {:where   '[[?s :store/items ?e]
                                                           [?e :store.item/section ?n]
                                                           [?n :store.section/path ?p]]
                                                :symbols {'?s store-id
                                                          '?p navigation}}
                                               {:where   '[[?s :store/items ?e]]
                                                :symbols {'?s store-id}})))}))))

Since this is common, maybe we want to implement as cases, something like this:

(defmethod lajt-read :query/store-items
  [_]
  {:base {:remote true
          :query  {:find '[[?e ...]]}
          :params {'?s [:route-params :store-id]}}
   :case {[[:route-params :navigation seq]]
          {:query  '{:where [[?s :store/items ?e]
                             [?e :store.item/section ?n]
                             [?n :store.section/path ?p]
                             [?e :store.item/name _]]}
           :params {'?p [:route-params :navigation]}}

          [[(constantly true)]]
          {:query '{:where [[?s :store/items ?e]
                            [?e :store.item/name _]]}}}})

Remote parsing should process the :pre ops

The :remote key might be within an expansion op, like :case.

We should always be running the :pre ops before trying to return the :remote query.

:depends-on should be able to be a function

Use case:

(defmethod lajt-read :query/navigation
  [_]
  {:depends-on (fn [{:keys [query]}]
   ;; See, a fn ^^^^^^^
                 [{:query.navigation/genders query}
                  {:query.navigation/categories query}])
  })

Implement :lookup-ref to be used instead of :query

Example from sulo:

(defmethod lajt-read :query/store-has-streamed
  [_]
  {:remote true
   :lookup-ref [:ui/singleton :ui.singleton/state]})

Parsing query with same key but different params

What to do about parsing deduped queries like this one:

    '[({:read [:a]} {:param 1})
      ({:read [:a]} {:param 2})]

It should never happen for local reads as query params is pretty much useless (as they are static). Dynamic query params are not allowed.

It did happen for remote queries when working on sulolive. The solution we did back then was to wrap these queries with the designated join-namespace, that have different names. This solution didn't have any problems, as the result of the query was just merged into the app state, and no one was trying to access a specific key in the query.

petterik / lajt Goto Github PK

lajt's People

Contributors

Stargazers

Watchers

lajt's Issues

Recommend Projects

Recommend Topics

Recommend Org