ClojureScript support? about mulog

[BrunoBonacci]

Hi @filipesilva ,

At the moment I don't have a ClojureScript use case for it, but I will look into this and come back with a design.
The CLJS side is more complex as you have to relay the events via a server endpoint. Because of the variety of client/server stack combinations, it is not trivial.
come back with a reasonable design.

For everyone else: please vote the issue if you think that μ/log on ClojureScript would be useful to you.

[wilkerlucio]

Hello Bruno, I'm on my path to pick a log library to use in some of my libraries like Pathom. I love the concepts of mulog, but since my libraries are cljc I can't use mulog with the lack of Clojurescript support.

How hard you think would be to give just a minimal implementation for CLJS? I mean, it doesn't need to support any fancy storing mechanism, the first version can have only the console.log option in CLJS.

What do you think?

[filipesilva]

Hi @BrunoBonacci, thanks for describing the expectations and hard parts in so much detail. I'll try to provide some input on them.

Flakes

If performance is the main consideration here, I don't quite see why it'd be any different than the performance hit of everything else on ClojureScript/Javascript. I scanned https://github.com/BrunoBonacci/mulog/blob/master/mulog-core/java/com/brunobonacci/mulog/core/Flake.java a bit and didn't see anything there that'd make me think it's a problem in Javascript. There is a notable exception though, the nanosecond precision. Don't think it's really possible in the current JS engines.

Thread-local context

I think the lack of multithreading on javascript means this is just a matter of storing state as usual. It's possible that a JS app would use web workers, but they share no state and communicate via message passing, so they'd just be another separate client.

Event propagation & Propagate tracing IDs

I think the problems described in these two sections hinge heavily on the premise of integrated logging between client and server apps, as you described in (2). But it is not clear to me that this is a first class problem, because I disagree that not solving it means that μ/log would then only log to the console.

ClojureScript apps today already use third-party logging services like Amplitude and Posthog. My (rough, maybe ignorant) expectation is that'd I should be able to make a μ/log publisher myself that'd mimic what the JS clients for those services do: collect the logging items, batch them, send them to some configured backend.

I think https://github.com/BrunoBonacci/mulog/blob/master/doc/publishers/slack-publisher.md is a good example. This looks like the kind of producer that would be the same in CLJS. I understand that there are concerns about using secrets and auth on CLJS apps, but those problems are outside μ/log (e.g. firebase does just fine with js config, I can configure my server to only accept calls from my official domain, etc).

I can also imagine that the integrated logging CLJS producer could just be another producer, meaning the special endpoint you mentioned does not need to be encoded in the CLJS μ/log design but rather deferred to producer implementations. Deferring this integrated producer looks like it'd leave CLJS μ/log aligned with the current CLJ μ/log design.

[BrunoBonacci]

Hi all,

There are a number of hurdles to overcome for a ClojureScript implementation, I will expose here my thoughts so that I can gather some feedback from your side.

Expected behaviour

μ/trace provides a primitive to capture the execution of some important operation and the relationship with other operations in the same context.

For example, assuming I want to trace the execution of product-availability, wrapping the function call with μ/trace will provide quantitative measure of the execution time, whether the execution raised any exception or it was successful, and the acutal exception value. In addition to that, it will also maintain the relationship with other calls to other functions wrapped in μ/trace.

(μ/with-context {:order order-id, :user user-id}
  (μ/trace ::availability
    [:product-id product-id]
    (product-availability product-id)))

Let's assume that the process-order function makes a call to the availability function and that the availability function, in turn, makes a call to warehouse-availability, then to shopping-carts and finally to availability-estimator, we will end up with a call-tree that looks like:

(process-order)
└── (availability)
    ├── (warehouse-availability)
    ├── (shopping-carts)
    └── (availability-estimator)

If all the functions above are wrapped in a μ/trace call you will be able to see the relationship between the calls and their outcome and duration in a distributed tracing tool (like Zipkin), here is an example:

(1) A reasonable assumption would be that if μ/trace is used on the client-side (ClojureScript) the generated trace would include the client call as well.

(2) Secondly, every call to μ/log and μ/trace made on the client-side should generate events that somehow are published through the publisher's infrastructure of the backend processes. If this wasn't true, then client-side μ/log calls would only be logged (if any) in the browser console (or node application console).

In my opinion, the above two requirements are key to make the ClojureScript μ/log extension useful.

Challenges

Flakes

Flakes are unique identifiers, in a single application context they are monotonic, extremely cheap to create and homomorphic in respect to the ordering in their string representation.
For performance reasons, they are implemented in Java (Flake.java).
It is not difficult to make a ClojureScript implementation, but performances will be far from the Java version one.

Thread-local context

μ/log on the JVM makes used of Thread-local variables to store the local context. A ClojureScript implementation would need a different approach.

Event propagation

In order for the μ/log's events generated on the client to end up in the backend publishers and centralised logging systems, the client-side will need to piggyback on a "special" backend endpoint.
Events will need to be collected in the client application and at regular intervals published to the backend "special" endpoint.
Here the challenge is related to the fact that in order to provide the special endpoint μ/log on the server-side will need to expose the endpoint using a number of different libraries: ring, yada, pedestal, reitit etc using an even bigger range of client-side libraries to post the events.
I'm not sure that having just one way to achieve this will be acceptable. Here more investigation is required.

Propage tracing IDs

Similarly to the previous point, in order for the distributed tracing to work correctly, the client-initiated calls will need to pass tracing headers back to the server application on every interaction. Again, due to the many different ways/libraries which can be adopted on the client-side this challenge isn't easy.

Summary

All the above challenges are fairly easy to solve in a specific context. When a specific set of libraries is defined, it is fairly easy to plug this in. However, provide a general approach that will work on all/most of the commonly used libraries on both: client-side and server-side is not an easy challenge.

If you have ideas on how to solve the above challenges, feel free to comment below. I'm interested to see whether there is a common solution that can be adopted.

[wilkerlucio]

One comment on the nanosecond precision, this is doable in the browser, but the accuracy vary from vendor to vendor. In Chrome for example its enabled by default and we can get nano time using (System/nanoTime). Firefox has it disabled by default but the user can enable via browser settings.

[xificurC]

ClojureScript apps today already use third-party logging services like Amplitude and Posthog

I think your request is not what Bruno is concerned about. There are 2 parts to mulog, logging and tracing. For just logging data from a browser, sure, you can define an endpoint and shovel the data there. But mulog also provides tracing support. Imagine the scenario where a user clicks a button and you initiate a request to the backend (which is awfully common). You want to end up with a trace like

<client> add-to-cart
  <server> add-to-cart
    <integration A> send
    ...

i.e. the trace that initialized in the browser is the parent trace of your backend trace. In order to achieve that you have to send the backend your client-side trace-id (otherwise the server's add-to-cart trace woudn't be able to reference the client-side trace as its parent). This is what distributed tracing is all about. For this to work you need to correctly assemble the client-server communication channel, which can be http through a number of libraries, but also e.g. websocket.

If we cut the scope of the request and you only wish to have cljs side logging then I think the task is simpler, since you only need to get mu/log working.

[xificurC]

As for tracing, OpenTelemetry is getting traction and I think sending and reading their standardized headers would enable interop. @BrunoBonacci if you'd choose to use their headers for sending over the tracing context you could let users leverage the ecosystem. You wouldn't need to add support for all the http libraries, just serialize the context into a clojure map of their http headers and let the users plug those headers into their requests.

[ignorabilis]

@BrunoBonacci - for the event propagation challenge - may be a silly question, but why bother with all the possible servers? Why not just make mulog on the frontend to require a server url during configuration - and then mulog sends all frontend events to that url, regardless of server implementation. Then on the server side you just provide a generic handler that can be used as a request handler.

flakes challenge - sure, performance is super important on the backend - if hundreds/thousands/more users are hacking and clicking and the system produces a gazillion log messages per second, then being able to log stuff confidently (knowing that the perf overhead is low) is a must; in cljs - not so much. Also completely agree with @filipesilva that not having nanosecond precision is a part of life when dealing with js (even if it could be enabled in the browser, etc.)

thread local variables - js being single threaded this is a non-issue; all web workers are isolated so variables are "thread local" by default (just repeating @filipesilva ).

propage tracing ids challenge - not sure about that one; if I'm understanding correctly you'll need the same trace id on the frontend (or whatever the js environment is) as well as on the backend; however that should only mean that mulog generates its own tracing ids and same as propagating events it just uses an endpoint on the server.

Let me know your thoughts, super interested in having a cljs implementation (without which it's not quite usable for certain classes of projects).