mocktomata / mocktomata

A behavior simulator

Mocktomata

Welcome to mocktomata, the behavior simulation system.

My name is mockto.

I'm the main automaton and your guide around here.

In a nutshell, mocktomata saves the behavior of your code, and simulate it at a later time.

We can isolate your code from the changes and uncertainty of the outside world. And we can connect your code back to it by a flip of a switch.

That means, instead of manually writing mocks, you can write e2e tests, and run them as e2e or as unit tests.

Documentation

To understand more about us, please take a look at https://mocktomata.github.io/website

mocktomata's Issues

capture context

Some functions do not return anything and in turn, write some internal states.
Need to capture these internal states and replay them.
(If it calls some internal function, we are probably out of luck at the moment)

rethink SpyCall implmentation to avoid mixing promise and class logic together

https://github.com/unional/komondor/blob/f47ba7c75cc7003276e6b91e30570088a43d48f9/src/class/spyClass.ts#L13-L18

---

This issue was generated by todo based on a TODO comment in f47ba7c when #112 was merged. cc @unional.

Support isError

For throw case, I'm not adding info to deserialize the payload to an Error because of the difficulty to get it right (no access to custom error).

But should do reasonable work to at least make it pass err instanceof Error.

send records to remote server

Allow the record to be saved on a remote server instead of in local file system.

support UI

Env invoke bug

onEnv()
env.sim()
env()

The second call will have no effect.

Need to track if live call is made.

Support websocket

Dependency change detection

If the dependency supports sem-ver, we can use that to determine if a re-run of system integration test is needed.

On the flip side, if the current package supports sem-ver, we can run system integration tests on the dependent projects to ensure the code is still working.

(maybe sem-ver is not necessary, but may be helpful in reducing workload)

Support websocket server

Steam and WebSocket does not work well with Promises.

Need to figure out how to do it.

One idea is the callback can be called multiple times with chunks of data. And then a terminal string.

First check server side code (e.g. Hapi) on how the data is transmitted.

make support of childProcess generic

Support childProcess is nice but it tightly coupled with NodeJS.

That means other languages and platforms cannot use the spec.

Need to make the support of objects like childProcess generic and create a plugin system to support different platforms.

ensure given() execution uniqueness across processes

Test runner can run tests on multiple workers or processes.

onGiven('normal load', () => {})

config.given('normal load', 'run')

The onGiven() clause should execute in run mode only once across all processes.

Need to create some synchronization between them to ensure this uniqueness.
global-store won't help in this case.

concurrency issue with writing spec record

If concurrency becomes an issue,
use dnode, get-port to start a server to do the writing,
and each Spec and Scenario will send messages to it for reading/writing.

The server will save a temp file to remember the last used port (which will be the current port if server is running),
and the client will use that file to determine which port to connect to.
The server will have an isKomondor() function to make sure we are dealing with the right server.

handle `returnType` and `<type>Id` in the framework

Current each plugin need to handle that themselves, which is not necessary.

Also, it is a global counter, thus the counter is affected by other spec() which would mess up comparison.

add `spec.live()`

spec() will honor what's in given.?().

It would be nice to force live while given.simulate()

A workaround right now is using global spec() instead. i.e.:

const env = await given.simulate(...)
const s = await env.spec(...)

// change to
const env = await given.simulate(...)
const s = await spec(...)

throws when specId and givenId contains invalid characters

Some characters are not valid in some operating systems, e.g. : is not valid on Windows.

Need to detect and fail early on those ids.

configure running step once or each

In the normal situation, onGiven() should be executed only once.

This is based on the assumption that setting up the environment is heavy, and all tests should clean up after themselves.

This is similar to beforeAll() (and afterAll()?).

beforeEach() and afterEach() should not exists in the first place in my opinion because that is just a lazy catch for people saying "in case you forgot to clean up, I'll clean up for you".

You can easily do beforeEach() and afterEach() by:

test('...', () => {
  setup()
  ...
  teardown()
})

It actually makes the code easier to understand.

The real problem comes when there is a mutable global state introduced in one of the tests (i.e. changed the environment and didn't clean up).

Running onGiven() once makes it hard to pinpoint the error.

So, adding a way to change onGiven() to run on every test, and validate the environment correctness using the spec() inside onGiven(), can pinpoint that error.

The way to do it is through config().

One proposal is:

config.given(/some given/, { executeOnEveryTest: true })

env.sim() should not invoke handler

When save is ready

class need to spy on parent methods

e.g. currently this doesn't work:

class Connection { send() { ... } }
class TestConnection extends Connection { ... }

const s = await spec(TestConnection)
const a = new s.subject()
a.send() // not captured.

inner call should not be captures when public call returns a promise

In the following code, the inner() call should not be recorded (and spied).

    class PromiseInner {
      do() {
        return new Promise(a => {
          setImmediate(() => {
            a(this.inner())
          })
        })
      },
      inner() {
        return 'inner'
      }
    }
    const s = await spec(PromiseInner)
    const a = new s.subject()
    await a.do()

support Stream in function input

For example, sending a file to a remote server using node-fetch, a file stream will be passed in the body property.

move npm/TypeScript to sub-folder

To make way for multi-language implementation.

Determine should `spec` be linked inside `environment`

My plan was to provide a linked version and it will switch mode as environment switches mode.

But deferring it as I need more actual usage to determine the behavior, for example:

environment('...', ({ linkedSpec }) => {
  // live environment but simulate spec?
  // live environment and live spec. When environment switch to simulate, should spec switch too, or are there cases it should not switch?
  // live environment and save spec. When environment switch to simulate, what should happen to spec?
})

environment.simulate('...', ({ linkedSpec }) => {
  // simulate environment and simulate spec. When environment switch to live, should spec switch too? or are there cases it should not?
  // simulate environment and save spec. Should this be an error?
  // simulate environment and live spec.  Should this be an error?
  // simulate environment and live spec, should it become simulate automatically?
  //   meaning user only need to change `environment()` to `environment.simulate()`, instead of changing both.
  //   likely doesn't make sense, as the spec record may not be saved.
})

Port to Kotlin

rename asyncOutput

to response?

can change to a map and remove invokedCallback:

{
  asyncOutput: {
    success: [...calls],
    fail: [...calls]
  }
}

Port to python

Support local caching for remote server

Loading spec from a remote server every time is slow.

Should have a local cache store.

repurpose `.verify()`

That will be used on the other side of the boundary.

const dep = spec(InternalDependency)
const sub = spec.verify(Subject)

const target = new sub.subject(new dep.subject(...), ...)

...
sub.satisfy() // will use saved expectation to check against output.

missing one level of id

Currently, we have instanceId and invokeId.
But that doesn't support classes and similar structure.

For class, it really needs 3 ids:

• id: of the stub instance
• instanceId: of the instance of the created stub
• invokeId: of the invoked method / event.

create expectation helpers

The data structure is now flexible, but it could be verbose for people to use.

Create some expectation helpers for each "plugins" so it is easier to create the right expectations.

Improve plugin interface

It is getting a bit messy for my taste.

interface KomondorPluginUtil {
  registerXXX,
  getSpy(...),
  getStub(...),
  getReturnSpy(...),
  getReturnStub(...),
  log...,
}

export function activate(komondor: KomondorPluginUtil) {
  komondor.registerSpy(...)
  komondor.registerStub(...)
  komondor.registerReturnSpy(...)
  komondor.registerReturnStub(...)
}

add s.done()

and update readme to suggest not using s.satisfy()

avoid using camel-case?

Treating simpleCallback failed equals to simpleCallbackFailed could create confusion to user.

support multiple filters

config.spec('live', filter1, filter2, ...)

// or
config.spec({
  'live': [filter1, filter2,...].
  'save': [filter3,...],
  'simulate': [...]
})

Support caching

Always loading from disk is still slow.

Should investigate on using memfs, especially when the code is running in watch mode.

One issue is multi-process / worker runner.
Need to have a way to access the same data across them.

support continuous use of spec

Right now it ends when a call is completed.
Re-using is not possible.

support database

Support template for onGiven()

1. Pass in matched result from regular expression to global handler.
2. Use 'pay {dollar} receives {n} apple to pass value to global handler.

Add BDD support

Use scenario to group spec together.

However, you can use regex in config({ spec: /.../ }) and organize your spec into nicely grouped folders to achieve the same result.

spec(..., { id: '<scenario>/<spec function>/<context>', ... })

// e.g.
spec(..., { id: 'login/auth/success', ... })
spec(..., { id: 'new user/getProfiles/success', ... })
spec(..., { id: 'new user/getProfiles/no rights', ... })

But having the scenario as a standalone concept allows specs to be reused in multiple scenarios.

May adopt BDD syntax:

Feature -> Scenario -> Given

The purpose is making it consumable and definable by Business Analyst and QA.

on() and onAny()

Use fsa-emitter to support that when a new action is added.

Create Expectation Interfaces

Tailored for komondor.

rename asyncError

It is about the connection failure.
Currently only happens on Promise style.

Need to see what should be done on callback style (to catch these errors).

support fs

spy and stub onfs

spec.save() and spec.simulate()

instead of { mode: 'save' | 'replay' }

Add legend to diff

Like jest:

- expected
+ received

...

Remove `getReturnXXX`

just getSpy and getStub should be used.

expectation with context

Currently expectation must be pure,
it cannot reference values from outside because it will be serialized and run in a different context.
The scope can't be preserved.

Maybe it is possible to create a construct to capture the necessary context to make the expectation aware of its surrounding.
Maybe a class instance passing in the context, and analyze the expectation to stripe out value when saved?
Or the function, when deserialized, can be called with f.call(this, ...args).

rename 'closing'

spec.closing is created under the stream concept.

The naming of 'closing' does not apply to other concepts.

Need to figure out a better, more generic name.

support stream

reconsider `environment()` design

The current design of environment() does not fulfill a primary function of it: provide a smooth communication mechanism between engineers and IT.

To provide better support on this end, it needs to:

• compose of multiple smaller configuration. One environment is easy to use, but hard to implement (and unable to communicate all the tiny details about an environment).
• Able to save and add additional details such as descriptions.
• Able to validate the environment meets the need. This may be hard to do inline using context.spec.satisfy() because it is hard to test everything, and multiple environment changes can overlap each other (e.g. can't check users.length when there are multiple environment config that add different users to the system).

Use komondor.json to config givens