borkdude / api-diff

Print API diffs between library versions

api-diff

This tool prints the diff of breaking changes between two Clojure library versions.

Status - Alpha

Api-diff is in the early stages of development - any interfaces can be expected to change.

We encourage you to try the tool out and greatly appreciate any feedback you might share.

Raise an issue or drop by and chat in the Clojurians Slack #clj-kondo channel.

Installation

To invoke via -M:api-diff add this alias to your deps.edn:

{:aliases
 {:api-diff
  {:replace-deps
   {borkdude/api-diff {:git/url "https://github.com/borkdude/api-diff"
                       :git/sha "<latest-sha>"}}
   :main-opts ["-m" "borkdude.api-diff"]}}}

Or install as tool:

clj -Ttools install com.github.borkdude/api-diff '{:git/tag "<latest-tag>"}' :as api-diff

Usage

Arguments for comparing two mvn libs:

• :lib: fully qualified symbol
• :v1: the older version as mvn lib
• :v2: the newer version as mvn lib

Arguments for comparing two directories:

• :path1: the file or directory with older
• :path2: the file or directory with newer

This tool currently only prints breaking changes: removed vars or removed arities. To see what was added in a newer version, just swap :v1 and :v2 (for now):

clj_kondo/core.clj:205:1: error: clj-kondo.core/resolve-config was removed.
clj_kondo/core.clj:213:1: error: clj-kondo.core/config-hash was removed.
clj_kondo/impl/analyzer.clj:1473:1: error: clj-kondo.impl.analyzer/analyze-ns-unmap was removed.

Comparing two jars locally:

clj -M:api-diff :path1 ./my-jar-v1.2.jar :path2 ./my-jar-v1.3.jar

or via tool usage:

$ clj -Tapi-diff api-diff :lib clj-kondo/clj-kondo :v1 '"2021.09.25"' :v2 '"2021.09.15"'

Optional exclusions:

• :exclude-meta: exclude namespaces and vars with specified metadata keyword, repeat for multiple

Some libraries use :no-doc metadata to mark which namespaces and vars are not part of their documented public API. Use :exclude-meta to diff only the public API of these libraries:

$ clojure -M:api-diff :lib zprint/zprint :v1 0.4.16 :v2 1.1.2 :exclude-meta no-doc
zprint/rewrite.cljc:29:1: warning: zprint.rewrite/prewalk now has meta [:no-doc].
zprint/rewrite.cljc:44:1: warning: zprint.rewrite/get-sortable now has meta [:no-doc].
zprint/rewrite.cljc:54:1: warning: zprint.rewrite/sort-val now has meta [:no-doc].
zprint/rewrite.cljc:90:1: warning: zprint.rewrite/sort-down now has meta [:no-doc].

Other libraries use :skip-wiki. Let's first compare 2 versions of spec.alpha with no exclusions:

$ clojure -M:api-diff :lib org.clojure/spec.alpha :v1 0.1.108 :v2 0.2.194 
clojure/spec/alpha.clj:348:1: error: clojure.spec.alpha/map-spec was removed.
clojure/spec/alpha.clj:1360:1: error: clojure.spec.alpha/amp-impl arity 3 was removed.

Now let's repeat the run with the knowledge that this library use metadata to exclude vars from its public API:

$ clojure -M:api-diff :lib org.clojure/spec.alpha :v1 0.1.108 :v2 0.2.194 \
   :exclude-meta skip-wiki :exclude-meta no-doc
clojure/spec/alpha.clj:348:1: error: clojure.spec.alpha/map-spec was removed.

Now we only see the relevant changes to the public API.

How it works

To discover APIs, api-diff uses clj-kondo's data analysis feature.

For reasons of safety and speed, clj-kondo uses static analysis. This means it reads, but does not run, source code.

There are some libraries generate their APIs at runtime. Clj-kondo has built-in support to apply the effect of potemkin import-vars and can be configured to apply the effect of other macros that manipulate APIs at runtime.

api-diff's Issues

Distinguish lang (aka platform aka dialect, i.e. Clojure, ClojureScript)

Scenario
One var can be defined differently for different langs.

A contrived example:

#?(:cljs (defn mixed [a cljs sig])
   :clj  (defn mixed [a different clj sig]))

A real-world example might be the different signatures for assert-expr between Clojure and ClojureScript.

Api-diff currently does not consider the lang when comparing APIs, and will currently sometimes target the wrong v2 target.

Proposal
The unique key for a var is currently :ns + :name.
Make it :ns + :name + :lang.
Report on lang.

Reporting option ideas:

1. A line for each diff:

   test-resources/older/example.cljc:11:1:clj error: example/y was removed.
   test-resources/older/example.cljc:11:1:cljs error: example/y was removed.
   

   Accurate, but could be considered noisy.

2. A line for each unique diff.
   We could consider optimizing this to a single line by listing platforms when diff finding is same for all platforms for single var:

   test-resources/older/example.cljc:11:1:clj,cljs error: example/y was removed.
   

3. Omit lang from reporting when vars compared are Clojure only or ClojureScript only.

Option 3 can be combined with option 1 or 2.
I'm leaning toward only option 1 for an initial release. What do you think?
(Any edn #16 return would be the verbose finding for each diff)

Other info

• Somewhat related to #5, at least in topic
• Clojure documented platforms are :clj :cljs :cljr and :default.
• We'll just pass along what clj-kondo provides us (including :bb etc)
• We'll infer :lang when clj-kondo does not fill it by looking at filename extension.

Next steps
After we agree on an approach, I can take work on a PR.

Ignore private vars when comparing

Proposal
Private vars should always be excluded when comparing APIs.

How does it work now?
If we update test-resources/older.clj to add pf and p:

(ns example)

(defn- pf [])
(def ^:private p 3)
(def x 1)
(def y 2)
(def z 3)

And the run an api-diff like so:

❯ clojure -M -m borkdude.api-diff :path1 test-resources/older.clj :path2 test-resources/newer.clj
test-resources/older.clj:3:1: warning: example/pf was removed.
test-resources/older.clj:4:1: warning: example/p was removed.
test-resources/older.clj:6:1: error: example/y was removed.
test-resources/older.clj:7:1: warning: example/z was deprecated.

You'll notice that pf and p are reported as removed.

How I propose that it works
I would expect p an pf to not be reported:

❯ clojure -M -m borkdude.api-diff :path1 test-resources/older.clj :path2 test-resources/newer.clj
test-resources/older.clj:6:1: error: example/y was removed.
test-resources/older.clj:7:1: warning: example/z was deprecated.

Next steps
I can follow up with a PR if you agree.

Mention static analysis in README?

Should we mention that api-diff uses clj-kondo static analysis and therefore will not pick up APIs that are generated at load time? (We can mention it can compensate via custom hooks and has built in support for potemkin import-vars).

(Amazonica is a good example of a lib that does load time API generation.)

If you think this is a good thing to mention I can add in.

Clarification: What is considered a breaking change?

It would be nice to add to the README a few words about what is considered a breaking change.

Consider reporting on variadic arity

When v1 has

(defn variadic-loss [a b c & more])

And v2 has

(defn variadic-loss [a b])

The loss of the variadic signature could be reported, maybe like so:

test-resources/older/example.clj:x:y: error: example/variadic-loss variadic arity was removed.

If v1 and v2 were swapped, we do have a signature difference but don't technically have a signature breakage and so would not report.

Optionally compare by Clojure dialect

Proposal
Some libraries expose a different API for different dialects of Clojure.
It could be interesting to allow comparison of API by Clojure dialect.

Vested interest examples:

• How does the rewrite-clj v1 API differ for clj and cljs?
• How does rewrite-cljs API differ from rewrite-clj v1 cljs API?
• How does the rewrite-clj v0 API differ from rewrite-clj v1 clj API?

Some other details
I did support such a feature in lread/diff-apis, but my implementation is, IMHO, unnecessarily complex, and I'd like to migrate to this project, if I can.

Implementation ideas
We already have :path1 :path2 and :v1 and v2.
I suppose we could add :lang1 :lang2?

For same lib comparison :v1 and :v2 (or :path1 and :path2) could specify the same target:

clj -M:api-diff :lib rewrite-clj/rewrite-clj :v1 1.0.699-alpha :v2 1.0.699-alpha :lang1 cljs :lang2 clj

Compare rewrite-cljs to rewrite-clj v1:

clj -M:api-diff :path1 <path to rewrite-cljs jar> :path2 <path to rewrite-clj jar> :lang1 cljs :lang2 cljs

(I suppose we could introduce :lib1 and :lib2 but I won't scope creep that in for this issue).

Compare rewrite-clj v0 to rewrite-clj v1:

clj -M:api-diff rewrite-clj/rewrite-clj :v1 0.6.1 :v2 1.0.699-alpha :lang1 clj :lang2 clj

Next steps
I could take a stab at this, if this idea makes sense and is of interest.

Sample invocation not working for me

I have added this alias to my ~/.clojure/deps.edn:

    :api-diff {:replace-deps {borkdude/api-diff {:git/url "https://github.com/borkdude/api-diff"
                                                 :git/sha "7149bccc6a9f7a5734c3413e6bb4faa18c8d90c3"}}
                             :main-opts ["-m" "borkdude.api-diff"]}

Then, trying the sample invocation from README.md:

clj -Mapi-diff api-diff :lib clj-kondo/clj-kondo :v1 '"2021.09.25"' :v2 '"2021.09.15"'
Execution error (ExceptionInfo) at clojure.tools.deps.alpha.extensions.maven/get-artifact (maven.clj:153).
Could not find artifact clj-kondo:clj-kondo:jar:"2021.09.25" in central (https://repo1.maven.org/maven2/)

I'm not sure if this is because I don't have something set up to correctly resolve Clojure deps from Clojars?

Consider sharing reason why target is no longer visible.

Proposal
When comparing APIs, a target var can be reported as removed.
It would be nice to know the reason why. Here are current possibilities:

1. deleted - the var no longer exists
2. made private - the var was public, but has become private
3. excluded - the filtering specified to api-diff excluded the target var (we currently only have :exclude-meta but might add more in the future)

Other details
I might have inadvertently undone some private reporting via #8, but I think api-diff might have been comparing privates to privates? If so, this would be a refinement.

Implementation ideas
As of this writing, we have:

test-resources/older/example.clj:6:1: error: example/becomes-private was removed.
test-resources/older/example.clj:8:1: error: example/becomes-nodoc was removed.
test-resources/older/example.clj:11:1: error: example/y was removed.

If we were to add the reason:

test-resources/older/example.clj:6:1: warn: example/becomes-private was made private.
test-resources/older/example.clj:8:1: warn: example/becomes-nodoc was excluded from diff (meta :no-doc).
test-resources/older/example.clj:11:1: error: example/y was deleted.

Future thought: If we do end up with multiple exclusion filters, and a single target was excluded for multiple reasons we'd only show 1 reason.

Next steps
I could take this on, if nobody beats me to it.

Consider showing additions

Proposal
Api-diff currently shows what has been removed.
The (very worthy) focus, it seems, is on discovering breakages.

It can also be interesting to see what was added to an API.
What new things can I do?

This can be achieved by swapping x1 and x2 in the diff comparison, but this, to me, seems awkward because the diff is still described in terms of removals.

Implementation ideas
Show the user:

• new vars
• new arities

What might we report? Maybe something like:?

x/y/z.cljc:10:1: info: x.y.z/new-def-x was added.
x/y/z.cljc:20:1: info: x.y.z/changed-defn-y Arity 10 was added.
x/y/z.cljc:30:1: info: x.y.z/new-defn-z was added.
x/y/z.cljc:30:1: info: x.y.z/new-defn-z Arity 1 was added. 

We would also need to include reason of additions, see #13.

test-resources/older/example.clj:6:1: info: example/becomes-public now public.
test-resources/older/example.clj:8:1: info: example/loses-nodoc now included in diff (meta :no-doc).

To work out

1. Is this idea interesting?
2. Optionally or always show additions?
3. Or only show one of: additions or removals?
4. Separate additions from removals in reporting? One tells about breakages, the other about opportunities.

Next steps
Refine with anybody who might be interested in this feature.

Linter for avoiding inflicting breakage

Context

It is a common desire/need to avoid inflicting breakage to one's downstream consumers.

It can be hard to consistently avoid this in authoring / code review phases.

Idea

Create a linter that inspects git diffs and determines whether any new breakages have popped up, failing builds accordingly.

...Perhaps parsing diffs would be unnecessary - one can instead compare trees using api-diff's usual API. The only interesting part of the diffs would be their contained file names.

WDYT?

Cheers - V

Consider optionally returning result as edn

Proposal
Optionally return api-diff result as a map.

• the code would benefit from separating reporting from analysis
• the user would benefit from being able to use the result as they pleased

Initial Ideas
Command line would include optional :format which would default to :text but also allow for :edn.

The api-diff fn would no longer print and instead return the a result map.
A separate report function would spit out result in requested format.

Next steps
What would map contain?

• opts used for diff
• same information as our current text only result but probably also complete clj-kondo v1 and v2 var info.

Consider reporting on callable <-> not-callable change

If v1 has

(defn becomes-def [])

And v2 has:

(def becomes-def 42)

An arity 0 deletion is currently reported. This is an indicator of breakage and might be good enough.

But if we swap v1 and v2 we get no report which is not ideal.

Perhaps we could instead report when something has become callable (or not callable) as a breakage.

test-resources/older/example.clj:x:y: error: example/becomes-def was callable.
test-resources/older/example.clj:x:y: error: example/becomes-defn was not callable.

(BTW: I'm not suggesting that we evaluate based on def vs defn but rather on the presence or absence of any call signatures found by clj-kondo).

Optionally compare only documented APIs

Proposal
Many libraries deliberately indicate which vars and namespaces are private via :no-doc metadata.
When :no-doc is applied to a namespace, the entire namespace is not documented as part of the public API.
When :no-doc is applied to a var, the var is not documented as part of the public API.

This convention was popularized by Codox has been adopted by cljdoc.

It would be interesting if api-diff optionally supported comparing only documented (i.e. public) APIs.

Some other details

• In the very old days some libraries used :skip-wiki as well.
• Codox (by default) and cljdoc (always) also excludes record constructor functions (ex. ->Foo and map->Foo), but since api-diff only does static analysis, these shouldn't be a concern.

Implementation ideas
Perhaps a new :exclude-with-meta option would make sense?
Usage might look like:

clj -M:api-diff :lib clj-kondo/clj-kondo :v1 2021.09.25 :v2 2021.09.15 :exclude-with-meta :no-doc

Or maybe we should allow for a sequence instead?:

clj -M:api-diff :lib clj-kondo/clj-kondo :v1 2021.09.25 :v2 2021.09.15 :exclude-with-meta [:no-doc :skip-wiki]

Next steps
I could take a stab at this PR, if it is of interest.