calculang / calculang

calculang is a language for calculations 🧮💬👩‍💻

calculang

calculang is a language for calculations.

motivation: separation of concerns

calculang is domain-specific to calculations and it proposes a separation of concerns: calculations separate from general programming.

This makes it easier to reason about calculations, empowers specialised tooling, and enables frictionless transparency and certainty about calculations.

A language for calculations also generalises more readily than more rigid systems or tools, enabling unity of calculations and process (all ends).

A language for calculations can integrate freely with established best-practice development, testing, and controls approach and tooling for languages.

🐣 early calculang highlights ⚡

Visit calculang.dev for the official website and examples!

I post about calculang and sometimes models using calculang on my CalcWithDec blog.

My (older) ObservableHQ collection including additional models including ferns, donuts and pi estimation models!

follow calculang 📫

For calculang updates follow @calculang on Twitter, or follow @[email protected] from any(!) Mastodon server 🐘

compiler

cul-js CLI compiles calculang into Javascript and provides information about the resulting model (introspection).

usage

Installation:

npm install --save-dev @calculang/calculang-js

Compilation:

cul-js compile entrypoint.cul.js

This creates entrypoint.js (alongside entrypoint.cul.js). This is a UMD Javascript bundle, with sourcemap entrypoint.js.map.

Use the --memo option to turn on memoization. Memoization typically dramatically helps execution of models.

Introspection:

cul-js introspect entrypoint.cul.js

Graph, using dot command from graphviz (install separately):

cul-js dot entrypoint.cul.js | dot -Tsvg > temp.svg
start temp.svg

design principles/features

Modularity:

A key design focus for calculang is scalability. To this end calculang models can be defined in modular fashion and recycled for varying usecases.

Models-of-models is a useful concept in considering model design. A design suggestion is to develop very general models and apply these in use- or application-specific models; in this way the models can support a wider range of usecases.

Input inference:

Inputs to function calls in calculang are inferred by the compiler, based on a graph of inputs and functions in the complete model being compiled.

This reduces code boilerplate, but more fundamentally it promotes recycling of models: we shouldn't code the wiring between functions manually because it will differ from one usecase to the next.

Inheritance of parent functions and overriding:

In modular development, calculang gives precedence for a given function call to functions defined closer to the entrypoint or model root, rather than closer to the call.

In this way "parent" models can override the functionality of "child" models, and this supports recycling of very general models.

The overriden functionality can be defined in terms of the child function (using an interim '_' modifier), for example to inflate the price in a 'base' model might involve

import { price_ as price_base } from './base.cul.js';

export const price = price_base() * 1.1;

Javascript:

In calculang-js implementation calculang models can interact with Javascript. You should keep Javascript-specific code in separate .js files/packages.

This interaction, done carefully, opens up many usecases. Not least, integration and co-ordination with (and of) other systems.

status

Initial implementation scales better conceptually vs. practically, but is nonetheless useful for an array of simple applications and carefully bounded problems beyond that.

This is another way of saying: some technical creativity helps to get the most out of the existing implementation. Careful interation with Javascript leaves enormous scope open here!

roadmap

v0.1.0-alpha1+ : some mini releases to introduce developed/proposed features, tests, cleanup code

v0.1+.0 : more tests, examples, integrations, documentation, logo, community focus, maybe REPL, website, blog, barebones browser extension, barebones VSCode extension, alt. bundler?, eventually show HN, review license/CLA/etc, CI/CD optimisation

Post: alt. implementations? e.g. Rust?

help wanted

Extension authors for browsers/VSCode. These can be independent projects. Similar Re integrations (incl. Excel, Python & notebooks, Julia & Pluto.jl).

Frontend developers for blog/websites and community model frontends including tools.

Community models and web apps, vizualisations/explorables (calculang.party?).

All developers to get feedback/discuss/think re motivation and implementation/technical improvements.

more motivation

calculang is motivated to make models: the workings of numbers, more accessible. Bret Victor included points around this in his blog post on What can a technologist do about Climate Change (a personal view), particularly relevant at Media for Understanding Situations.

Also good by Bret Victor, regarding interfaces to models: Explorable Explanations.

Many ideas along different lines of interaction with models become clearer when we separate concerns in the way that a language for calculations proposes.

contributing

CONTRIBUTING.md contains brief development/contributing notes including build steps.

license

Code in this repository is free software licensed under the GNU Affero General Public License Version 3.

calculang is made with ❤️ and 🧉

calculang's Issues

license clarifications

for:

• example/testcase calculang code
• observablehq notebook code

More permissive than AGPL, but still a few choices? MIT, ISC, BSDx, ...

example: work declaratively

declarative working example, candidates:

• ClimateMARGO model, although in Julia and available as a service it suits a particularly close integration
• lifelib actuarial models in Python, to extent the process I use needs input processes (csvs/spreadsheets)

Relevant: What processing do I want to do in calculang, for inputs and outputs?

remove introspection_info going into options of calculang-transform-loader

Requiring this to go in as an option makes it hard to do a simple webpack config, e.g. for calculang-hotreload-codesandbox

cul_functions includes non-exports with no indicator

This is especially visible with memoized models, where $-ending formulae appear in cul_functions: but they are not exported (just used within other code)

static analysis for recursion with missing bound cases

This bit me and led to stack error in memoisation code : w. no more indication

json (and csv) loader and json generator

example:

[
{product: 'PV panel', purchase_price: 1000, selling_price: 2000},
{product: 'battery', purchase_price: 300, selling_price: 500}
]

should be loaded effectively like:

export const pruchase_price = () => {
  if (product() == 'PV panel) return 1000
  else if (product() == 'battery') return 300
}
// similar for selling price

Also, we should be able to save or generate json from a model run. And possibly both simultaneously.

=> loading pre-generated results vs. loading model calcs may be just a flip of js/json extension (and can be seamless behind any interface)

override requirement, import new references (bug=no compile error)

This doesn't work, units is determined to have no inputs in introspection-api test (should have price_in):

import { revenue } from './base.cul';
export { revenue };

// demand-curve constraint:
export const units = () => 1000 - 200 * price();

does work with

- import { revenue } from './base.cul';
+ import { revenue, price } from './base.cul';

Maybe worth stating somewhere.

test modularity across packages

e.g. annuity model application.

no strict need to use npm.

Poss affected by #8

can I remove jest from individual packages?

& is it a good idea?

import * as base from './base.cul.js'

see also #13

Namespaced import, probably shouldn't inherit parent fns (=> apply '?-merge' in query string?), enables patterns like:

import * as base from './base.cul.js'
import * as proposal from './proposal.cul.js'
export const revenue_impact = () => (base.revenue() - proposal.revenue()) // this can also be auto-generated, e.g. potential cul-impact-loader?

be consistent about use of _in in api results

e.g. cul_functions inputs field, inputs excludes _in.

EDIT: I was wrong!

include option to generate a source map visualization link

Useful to use a source map visualizer to understand what compiler did.

Source map viz service: https://sokra.github.io/source-map-visualization

Can generate a link using base64 encoding. Drag bundle/sourcemap into interface and see "Link to this" link. Comes from https://github.com/sokra/source-map-visualization/blob/master/app/app.js#L282 Requires a little processing of sourcemap (app uses https://www.npmjs.com/package/source-map#examples)

We should be clear that this is insecure, so probably include that in the CLI option. "Insecure" because code goes over the wire to a remote service, when the link is accessed. (similarly when bundle/sourcemap dragged in)

Alternative approach? VS code extensions? Which we can easily link to?

Do/develop something local? Own visualizer? May be useful in the long run to include in a website or e.g. REPL interface.

Put sourcemap refs into introspection info (for function types AND call types)

Scope this out as an enabler for code overlays in vega vizes and website. Is this quick e.g. how to use refs?

Chk ability to traverse tree via same structure

monorepo build system : change?

npm workspaces? (I forget why I opted for lerna, maybe more relevant examples?)

nx? https://github.com/nrwl/nx/

csv test failing : inconsistent mapping

csv test failing here because of SOME inconsistent sourcemap mapping results

https://github.com/calculang/calculang/pull/54/checks

because of diff mappings calcd on my machine vs githubs? Possibly an impact from carriage return replacement messing ....

memo-loader implementation generates esm code that fails no-undef rule

This blocks #60 and other error-reporting issues that can be fixed by reliable eslints on esm.

META ISSUE: integrate an eslint check of esm output on build

simple eslint config pass on esm output closes #60 #49 and #31

Not immediate high priority because already getting full benefit (+flexibility) with a simple eslint config in model repos

creates a dependence on esm output, but this is OK

move away from dagre/graphlib

which is deprecated.

vega editor dataflow viewer moved to cytoscape? Does it have tree logic we need?

include compiler versions in bundle

hoist expensive/static inputs if it improves performance

data coming as an input may be expensive depending how javascript manages arguments and how deep passing goes.

must experiment or understand javascript behavior

// Is performance better if I hoist the table in a variable
// maybe a possibility if I did deeper calcs, lets juts see if it works.

export const numerator = ({ /* table_in, */ key_in }) => table({ /* table_in */ })[key({ key_in })].numerator;
export const denominator = ({ /* table_in, */ key_in }) => table({ /* table_in */ })[key({ key_in })].denominator;

let table_in; // we hoist table_in
export const ratio = ({ table_in: zzz, key_in }) => {
  table_in = zzz;
  return numerator({ /* table_in, */ key_in }) / denominator({ /* table_in, */ key_in });
}


export const key = ({ key_in }) => key_in;

export const table = ({ /* table_in */ }) => table_in;

use .mjs extension for esm output

rename extension in filename and in esm import ref substitution

Dont pollute _ namespace

The reason this is a Requirement is because a model that does

Import {price_}
Price = ...price_...

Breaks the possibility of other models overriding price again.

All_cul issue should NOT import any _s (so no problem about not knowing how to as them), they should always be explicit which is logical

exports issues?

I commented:

//import { revenue as r } from './base.cul';
//export { r as revenue }; need to create a link for this pattern to work // TODO

put calculang bundles up on api

I've hosted calculang models as netlify functions as a POC.

I'd prefer someone else give api a broader look, e.g. how to query multiple results (I didn't come up with a great solution when I thought about graphql).

Example which should move: https://observablehq.com/@declann/covid19-cases-deaths-link as this uses static model results vs interacting with model bundle.

fix VSCode call hierarchy

related: #6 especially ties in with graph potentially in there.

Call hierarchy doesn't understand overrides for instance.

Call hierarchy is very useful for modelling, so also check how can make call hierarchy more accessible (settings, dev?)

A prerequisite is UI or setting to select the entrypoint. Results will differ depending on selection.

Might be simple or difficult fix. Might be slow to re-populate (requiring an introspection-api call), but maybe can be kept to a minimum?

Revive (or do) a hot reload example for rapid dev workflow

Can be in codesandbox etc. Or codespaces (shouldnt be exclusive)

Is this calculang-js transform loader in a webpack app?
=> create react app(hmr?), npm i calculang-js and plug into webpack config

App can do almost anything, can evolve.

Hmr for non-webpack e.g. parcel, is out?? What about e.g. a watch mode?

Purpose is to suit a rapid dev workflow.

VSCode extension

A simple useful extension would include the graph, probably neater, perhaps interactive.

Access to calculang scope tree and function inputs (for a selected entrypoint) would be excellent and simple.

#10 would be useful

Inputs overlays would be useful and a pretty good/intuitive way to demo calculang. Other hint overlays?

Results views would eventually be useful #7

Integration with #4 eventually would be useful.

• snippets for cul/cul input is easy

Again lots of possibilities, it would be great to get something basic started.

Parentfn passing

CallExpression, where we do expansion i.e. in calculang-js, if fn depends on parentfn_in (for example) pass in the parentfn.

Usecase e.g. is incometax model with same 1 year input used for usc rates/bands and paye rates/bands, can be overridden by separate year inputs specific to the different functions that call year. e.g. facilitating a reconciliation model

don't leak directory locations

e.g. in a local project I just did npm i <local path to calculang/packages/calculang-js>

Now long relative paths leak into sourcemap. Maybe the command is considered the sources root, but the location of the entrypoint should be?

import all_cul from 'base.cul.js'

all_cul should be expanded into { units, price, revenue, ... }
If same fn is defined in current file the _ modifier should be applied

Does a circularity issue make this a little more complicated?

error: is there a (new?) problem with constants

memoization

Memoization is an obvious and huge performance win.

There is some code/basis to work from in memoloader.js

typings

Probably a good idea to produce typings for bundles, should consider this with a simple manual example first.

error or warn in .cul for non-function definitions

Legit JS so doesn't/won't cause an error in esm linting.

Also debatable for now (JS should be in sep. files, but force this already? Maybe ok when implementation is user-configurable?).

+the runtime error is confusing:

fix cul_scope_ids_to_resource

I need to create relative paths for implicit imports.

Direct use of cul_scope_ids_to_resource is not up to scratch!

Without fixing, modularity probably only works for flat directory structures!

--no-memo causes error & clarify readme

browser extension and detection logic

Browser tools require interfaces with webpage and detection logic to be specified (at minimum).

#4 should be used for interface to browser extension.

no build-time error for typos in function calls

eg:

bundler/transpiler considerations

ATM using webpack v4.

May move to v5, but I'm interested in considering other options, especially Parcel v2.
Parcel can potentially run in the browser, which opens wider possibilities (e.g. for REPL execution), this is the issue to follow: parcel-bundler/parcel#1253

Is esbuild a better option?

Also got to consider swc versus babel, and might feed into bundler considerations.

compiler license treatment?

consider if license should go into .cul.js files, and how that translates into bundle if a user puts a license there (does webpack/bundle keep it?)

Also can/how to pick out the license programmatically? (for tooling e.g. web extension visibility)

tests failing (until Nov. 29)

Because I use string.replaceAll()

Only works in node >= 15, apparently actions will go to 16 on Nov. 29:

actions/runner-images#4446

debug/useful output: results of loaders

cleanup 2x visitor/loader code a lot

Much needed.

blockers for me:

language feature tags: #14 and #13

and solid tests of language features

viz of model results, model browser

Release some of my old model vizualisations/interactions.

Integrate with #4, potentially

instrumentation and debugging / fondue

I used fondue instrumentation in really early development.
Fondue has a massive runtime overhead, but data generated was useful for understanding the runtime execution and for visuals and is something lost now.

Theseus debugger type interface can also be useful for model debugging.

An --instrumentation flag would probably be a useful thing to consider for the cul-js command.

Obvious challenges: I don't expect fondue to work on ESM code (but can probably be applied after transpilation), and I expect memo to mess it up (so can leave memo off with instrumentation, but that changes the calling pattern, a memo/cache inside fondue/instrumentation logic can get this right though).
It will also be useful to be able to recognize a distinction between calculang and JS code, but this may be some work.

Of course, open to using other instrumentation tools.

bad run time errors when missing a param

e.g. added lag_in, didn't reflect in JS:

project logo

Starting concept:

Requirements:

Simple, colorful, and made using free software tools (Inkscape is brilliant).

bug? `cul-js` doesn't work installed with Yarn on unix: "env: node\r: No such file or directory"

I got this problem via github actions env, where I wanted to run cul-js:

yarnpkg/yarn#5480 (apparently affecting yarn v1, maybe not v2?)

I got a fix in my action by moving to an npm install rather than yarn.

From checking npm tarball published to npm CRLF endings in calculang-js/bin/index.js and not other files.

Or else is the problem with some conversion process inconsistency in lerna publish (or whatever it uses, yarn publish?)

Suggested fix from another project? nodecg/nodecg-cli#67 (but not sure how git-related?)

memoization for modular models

If #11 save depends on cache, then #47 depends on this !

developer/stakeholder tools : web app

To see, develop, validate etc. models and workings behind results, developers and stakeholders should be working with specialised tools. This should be a web app, also accessible from browser extension with ability to detect workings behind numbers as we browse, also accessible from QR codes for IRL use cases e.g. on a letter from your bank.

Preferences may differ depending on developer/user role for instance, but still no reason for this tool to not be common.

Possibly (or optional), in-page results should be dynamic e.g. dynamically changes to inputs specified in tooling.

Model or problem-specific views as well as general modelling views may be a useful feature (e.g. some reasonableness check viz or higher-level approximation specific to a model used to guide developers/users).

A lot of possibilities about what this could turn into. Useful to make a start/POC.