Coder Social home page Coder Social logo

porsas's Introduction

porsas cljdoc badge

Nopea kuin sika pakkasella

Spike to see how fast we can go with both Clojure + JDBC & Async SQL. Highly Experimental.

Related dicsussion: https://clojureverse.org/t/next-jdbc-early-access/4091

Latest version

Clojars Project

Basics

porsas provides tools for precompiling the functions to convert database results into Clojure values. This enables basically Java-fast database queries using idiomatic Clojure.

SQL queries are executed against a Context, which caches compiled row transformation functions based on database metadata.

There are different Context implementations:

Currently, only eager query-one and query functions are supported.

Performance

At least an order of magnitude faster than clojure.java.jdbc, see the tests for more details.

Usage

JDBC

With defaults:

(require '[porsas.jdbc])

(def ctx (jdbc/context))

(jdbc/query-one ctx connection ["select * from fruit where appearance = ?" "red"])
; {:ID 1, :NAME "Apple", :APPEARANCE "red", :COST 59, :GRADE 87.0}

Returning maps with qualified keys:

(def ctx
  (jdbc/context
    {:key (jdbc/qualified-key)}))

(jdbc/query-one ctx connection ["select * from fruit where appearance = ?" "red"])
; #:FRUIT{:ID 1, :NAME "Apple", :APPEARANCE "red", :COST 59, :GRADE 87.0}

Generate Records for each unique Resultset, with lowercased keys. NOTE: this feature uses runtime code generation, so it doesn't work under GraalVM:

(def ctx
  (jdbc/context
    {:row (jdbc/rs->compiled-record)
     :key (jdbc/unqualified-key str/lower-case)}))

(jdbc/query-one ctx connection ["select * from fruit where appearance = ?" "red"])
; ; => #user.DBResult6208{:id 1, :name "Apple", :appearance "red", :cost 59, :grade 87.0}

Context can be omitted, bypassing all caching. Can be used when performance doesn't matter, e.g. when exploring in REPL:

(jdbc/query-one connection ["select * from fruit where appearance = ?" "red"])
; {:ID 1, :NAME "Apple", :APPEARANCE "red", :COST 59, :GRADE 87.0}

Async SQL

Uses non-blocking vertx-sql-client and can be used with libraries like Promesa and Manifold.

(require '[porsas.async :as async])

(def ctx (async/context))

;; define a pool
(def pool
  (async/pool
    {:uri "postgresql://localhost:5432/hello_world"
     :user "benchmarkdbuser"
     :password "benchmarkdbpass"
     :size 16}))

(-> (async/query-one ctx pool ["SELECT randomnumber from WORLD where id=$1" 1])
    (async/then :randomnumber)
    (async/then println))
; prints 504

A blocking call:

(-> (async/query-one ctx pool ["SELECT randomnumber from WORLD where id=$1" 1])
    (async/then :randomnumber)
    (deref))

With Promesa

(require '[promesa.core :as p])

(-> (pa/query-one ctx pool ["SELECT randomnumber from WORLD where id=$1" 1])
    (p/chain :randomnumber println))
; #<Promise[~]>
; printls 504

With Manifold

(require '[manifold.deferred :as d])

(-> (pa/query-one ctx pool ["SELECT randomnumber from WORLD where id=$1" 1])
    (d/chain :randomnumber println))
; << … >>
; printls 504

next.jdbc

Using porsas with :builder-fn option of next.jdbc:

(require '[porsas.next])
(require '[next.jdbc])

(def builder-fn (porsas.next/caching-row-builder))

(next.jdbc/execute-one! connection ["select * from fruit where appearance = ?" "red"] {:builder-fn builder-fn})
; #:FRUIT{:ID 1, :NAME "Apple", :APPEARANCE "red", :COST 59, :GRADE 87.0}

More info

There is #sql in Clojurians Slack for discussion & help.

Roadmap as issues.

License

Copyright © 2019 Metosin Oy

Distributed under the Eclipse Public License, the same as Clojure.

porsas's People

Contributors

ikitommi avatar imrekoszo avatar janosmeszaros avatar jonnik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

porsas's Issues

Bug in async namespace

Handlers for functions query-one and query in async context are equals therefore both functions returns only first row of result set.

Syncing up with recent changes in next.jdbc

This is mostly a heads up that next.jdbc row builders have a couple of extra expectations compared to when you first created porsas. Illustrated by the updated record-based builder in the test suite here: https://github.com/seancorfield/next-jdbc/blob/develop/test/next/jdbc/result_set_test.clj#L380-L406

The two main points:

  • There was an implicit assumption that all builders would support :cols and :rsmeta to return the vector of column names and the ResultSetMetaData but that has been made explicit and all the built-in builders now explicitly support clojure.lang.ILookup for those keys,
  • A new, low-level method has been added to RowBuilder -- with-column-value -- that is assumed to just do the "assoc" or "conj" equivalent, and is now used from most implementations of with-column; it is explicitly used by the new next.jdbc.result-set/builder-adapter and the existing as-*-adapter functions have been rewritten in terms of that.

Pipelining support

I see that setting the pipelining limit of PgPools is supported but I don't see it ever being used.

According to the vertx docs one has an option to create a pooled client as opposed to a pool and query using that to opt into pipelining. Querying using a pool as done here will not be pipelined.

Is this something you think porsas could support and perhaps accept a PR for? If not, could you please share the reasons why not?

Use Bounded Cache with Context

The default implementation stores all unique SQL sentences on a HashMap, which is a potential memoery leak. The default should be a fast bounded cache.

cves in dependencies via vertx

Using nvd-clojure:

; clojure -Sdeps '{:aliases {:nvd {:replace-paths [], :replace-deps #:nvd-clojure{nvd-clojure #:mvn{:version "RELEASE"}}, :exec-fn nvd.task/check :jvm-opts ["-Dclojure.main.report=stderr"]}}}' -X:nvd :classpath '"'"$(lein with-profile -base,-system,-user,-provided,-dev classpath)"'"' 2>/dev/null

We get

Checking dependencies for stdin  ...
  using nvd-clojure:  and dependency-check: 6.5.3
+----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| dependency                       | status                                                                                                                                    |
+----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| netty-transport-4.1.69.Final.jar | CVE-2021-43797                                                                                                                            |
| vertx-pg-client-4.2.1.jar        | CVE-2016-7048, CVE-2016-0766, CVE-2015-0244, CVE-2015-3166, CVE-2019-1021... |
+----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+

37 vulnerabilities detected. Severity: HIGH

Examining the dep tree:

; lein with-profile -base,-system,-user,-provided,-dev deps :tree

It can be seen these come via vertx-pg-client

 [io.vertx/vertx-pg-client "4.2.1"]
   [io.vertx/vertx-core "4.2.1"]
     [com.fasterxml.jackson.core/jackson-core "2.11.4"]
     [io.netty/netty-buffer "4.1.69.Final"]
     [io.netty/netty-codec-http2 "4.1.69.Final"]
     [io.netty/netty-codec-http "4.1.69.Final"]
     [io.netty/netty-common "4.1.69.Final"]
     [io.netty/netty-handler-proxy "4.1.69.Final"]
       [io.netty/netty-codec-socks "4.1.69.Final"]
     [io.netty/netty-handler "4.1.69.Final"]
       [io.netty/netty-codec "4.1.69.Final"]
     [io.netty/netty-resolver-dns "4.1.69.Final"]
       [io.netty/netty-codec-dns "4.1.69.Final"]
     [io.netty/netty-resolver "4.1.69.Final"]
     [io.netty/netty-transport "4.1.69.Final"]
   [io.vertx/vertx-sql-client "4.2.1"]
 [org.postgresql/postgresql "42.3.1"]
   [org.checkerframework/checker-qual "3.5.0" :scope "runtime"]

I'll submit a PR to bump the dependency.

Protocol to implement db functions

Currently, A CompiledQueries record is used to host different functions like :query and :query-one. It should be a protocol so we would get static analysis on the arities etc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.