tuwilof / fitting

Library add improve test log for RSpec and WebMock, validate its according to API Blueprint and Open API, show the documentation coverage with log.

Fitting

Library add improve test log, validate its according to your API documentation, show the documentation coverage with log.

Test log setting supports RSpec test and WebMock stubbing for Ruby On Rails application, API documentation supports API Blueprint and OpenAPI.

This reduces the costs of support, testers and analysts.

Telegram community for any Fitting related questions

Log

FITTING incoming request {"method":"POST","path":"/public/api/v1/inboxes/tEX5JiZyceiwuKMi1oN9Sf8S/contacts","body":{},"response":{"status":200,"content_type":"application/json","body":{"source_id":"00dbf18d-879e-47cb-ac45-e9aece266eb1","pubsub_token":"ktn6YwPus57JDf4e59eFPom5","id":3291,"name":"shy-surf-401","email":null,"phone_number":null}},"title":"./spec/controllers/public/api/v1/inbox/contacts_controller_spec.rb:9","group":"./spec/controllers/public/api/v1/inbox/contacts_controller_spec.rb","host":"www.example.com"}
FITTING outgoing request {"method":"POST","path":"/v1/organizations/org_id/meeting","body":{},"response":{"status":200,"content_type":"application/json","body":{"success":true,"data":{"meeting":{"id":"meeting_id","roomName":"room_name"}}}},"title":"./spec/controllers/api/v1/accounts/integrations/dyte_controller_spec.rb:50","group":"./spec/controllers/api/v1/accounts/integrations/dyte_controller_spec.rb","host":"api.cluster.dyte.in"}

validation

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF.FFF..FFFFFFFFFF....F.......F...FF.....F...F....F..............................FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..FF.F..FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..FF........FFF...FFFF......FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF........FFFFFFFFFFF..FFFFFF..FFFFFFFFFFFFFFFFF.......FFFFFF.............FFFFFFFFFFFF....F........FFF.F...FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF............FF........FFF......FFFFFFFFFFFFFFFFFFFFFF....FFFFFF......F............FFFF........FFFFFFFFFFFFFF.....FFFFFFFFFFFFFFFFFFFFFFF..FF.....FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF.....FF..........FFFFFFFFFFFFFFFFFF...FFFF...............F.F....FF..FFFFFFFF

  1) Fitting::Doc::NotFound log error:

host: www.example.com
method: POST
path: /public/api/v1/inboxes/{inbox_identifier}/contacts
code: 200

content-type: application/json

json-schema: {
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "id": {
      "type": "integer",
      "description": "Id of the contact"
    },
    "source_id": {
      "type": "string",
      "description": "The session identifier of the contact"
    },
    "name": {
      "type": "string",
      "description": "Name of the contact"
    },
    "email": {
      "type": "string",
      "description": "Email of the contact"
    },
    "pubsub_token": {
      "type": "string",
      "description": "The token to be used to connect to chatwoot websocket"
    }
  }
}

body: {
  "source_id": "c9e8c31f-06df-49b4-8fb9-4466457ae65b",
  "pubsub_token": "Zgc7DEvaj5TkgZ1a4C7AvJXo",
  "id": 3293,
  "name": "restless-snowflake-670",
  "email": null,
  "phone_number": null
}

error [
  "The property '#/email' of type null did not match the following type: string in schema e56b7e65-d70c-5f7a-a96c-982df5f8f2f7"
]

...

804 examples, 565 failure, 0 pending

Coverage: 65.51%

and cover

Installation

Add this line to your application's Gemfile:

gem 'fitting'

After that execute:

$ bundle

Or install the gem by yourself:

$ gem install fitting

Usage

Log

Firstly, improve test.log.

To your spec_helper.rb:

require 'fitting'

Fitting.logger

Delete all files log/*.log and run rspec

You get more information about incoming and outgoing request in log/fitting*.log.

FITTING incoming request {"method":"POST","path":"/public/api/v1/inboxes/tEX5JiZyceiwuKMi1oN9Sf8S/contacts","body":{},"response":{"status":200,"content_type":"application/json","body":{"source_id":"00dbf18d-879e-47cb-ac45-e9aece266eb1","pubsub_token":"ktn6YwPus57JDf4e59eFPom5","id":3291,"name":"shy-surf-401","email":null,"phone_number":null}},"title":"./spec/controllers/public/api/v1/inbox/contacts_controller_spec.rb:9","group":"./spec/controllers/public/api/v1/inbox/contacts_controller_spec.rb","host":"www.example.com"}
FITTING outgoing request {"method":"POST","path":"/v1/organizations/org_id/meeting","body":{},"response":{"status":200,"content_type":"application/json","body":{"success":true,"data":{"meeting":{"id":"meeting_id","roomName":"room_name"}}}},"title":"./spec/controllers/api/v1/accounts/integrations/dyte_controller_spec.rb:50","group":"./spec/controllers/api/v1/accounts/integrations/dyte_controller_spec.rb","host":"api.cluster.dyte.in"}

Validation

Secondly, validate the log to the documentation.

Add this to your .fitting.yml:

APIs:
  - host: www.example.com
    type: openapi2
    path: swagger/swagger.json

Run

bundle e rake fitting:validate

Console output

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF.FFF..FFFFFFFFFF....F.......F...FF.....F...F....F..............................FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..FF.F..FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..FF........FFF...FFFF......FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF........FFFFFFFFFFF..FFFFFF..FFFFFFFFFFFFFFFFF.......FFFFFF.............FFFFFFFFFFFF....F........FFF.F...FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF............FF........FFF......FFFFFFFFFFFFFFFFFFFFFF....FFFFFF......F............FFFF........FFFFFFFFFFFFFF.....FFFFFFFFFFFFFFFFFFFFFFF..FF.....FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF.....FF..........FFFFFFFFFFFFFFFFFF...FFFF...............F.F....FF..FFFFFFFF

  1) Fitting::Doc::NotFound log error:

host: www.example.com
method: POST
path: /public/api/v1/inboxes/{inbox_identifier}/contacts
code: 200

content-type: application/json

json-schema: {
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "id": {
      "type": "integer",
      "description": "Id of the contact"
    },
    "source_id": {
      "type": "string",
      "description": "The session identifier of the contact"
    },
    "name": {
      "type": "string",
      "description": "Name of the contact"
    },
    "email": {
      "type": "string",
      "description": "Email of the contact"
    },
    "pubsub_token": {
      "type": "string",
      "description": "The token to be used to connect to chatwoot websocket"
    }
  }
}

body: {
  "source_id": "c9e8c31f-06df-49b4-8fb9-4466457ae65b",
  "pubsub_token": "Zgc7DEvaj5TkgZ1a4C7AvJXo",
  "id": 3293,
  "name": "restless-snowflake-670",
  "email": null,
  "phone_number": null
}

error [
  "The property '#/email' of type null did not match the following type: string in schema e56b7e65-d70c-5f7a-a96c-982df5f8f2f7"
]

...

804 examples, 565 failure, 0 pending

Coverage: 65.51%

Coverage

And task will create HTML (coverage/fitting.html) reports.

More information on action coverage

Settings

APIs

type

OpenAPI 2.0

Swagger

APIs:
  - host: www.example.com
    type: openapi2
    path: doc/api.json

OpenAPI 3.0

Also OpenAPI

APIs:
  - host: www.example.com
    type: openapi3
    path: doc/api.json

API Blueprint

First you need to install drafter or crafter. Works after conversion from API Blueprint to API Elements (in YAML file) with Drafter or Crafter.

That is, I mean that you first need to do this

drafter doc.apib -o doc.yaml

or

node_modules/.bin/crafter doc.apib > doc.yaml

and then

APIs:
  - host: www.example.com
    type: drafter
    path: doc/api.yaml

or

APIs:
  - host: www.example.com
    type: crafter
    path: doc/api.yaml

Tomograph

To use additional features of the pre-converted tomograph

example

bundle exec tomograph -d crafter --exclude-description doc/api.yml doc/api.json

and then

APIs:
  - host: www.example.com
    type: tomogram
    path: doc/api.json

prefix

Setting the prefix name is optional. For example, you can do this:

APIs:
  - host: www.example.com
    prefix: /api/v3
    type: openapi2
    path: swagger/swagger.json

SkipValidation

host

It is not necessary to immediately describe each host in detail, you can only specify its name and skip it until you are ready to documented it

SkipValidation:
  - host: api.cluster.dyte.in

prefix

If you want to skip a specific prefix in the host

SkipValidation:
  - host: api.cluster.dyte.in
    prefix: /admin/api

method and path

If you want to skip a specific request in the host

SkipValidation:
  - host: api.cluster.dyte.in
    method: GET
    path: /api/v1/cars

NoCov

It is not necessary to immediately test each doc in detail, you can only specify its name and skip it until you are ready to test it

host

NoCov:
  - host: sso.test

method

NoCov:
  - host: sso.test
    method: GET

path

NoCov:
  - host: sso.test
    method: GET
    path: /users/{userId}

code

NoCov:
  - host: sso.test
    method: GET
    path: /users/{userId}
    code: 200

content-type

NoCov:
  - host: sso.test
    method: GET
    path: /users/{userId}
    code: 200
    content-type: application/json

combination

NoCov:
  - host: sso.test
    method: GET
    path: /users/{userId}
    code: 200
    content-type: application/json
    combination: oneOf.0

combination_next

NoCov:
  - host: sso.test
    method: GET
    path: /users/{userId}
    code: 200
    content-type: application/json
    combination: oneOf.0
    combination_next: oneOf.0.required.users

Debug

If you find bug, you can debug it or create task in this github project with new file coverage/fitting.debug.yml

Debug:
  - host: www.example.com
    method: GET
    path: /api/v3/users
    code: 200
    content-type: application/json

Community

Join us on Telegram.

Contributing

Bug reports and pull requests are welcome on GitHub at github.com/tuwilof/fitting. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Sponsored by FunBox

fitting's Issues

delete legacy mode

need delete match_schema and

fitting/lib/fitting/configuration/yaml.rb
fitting/lib/fitting/configuration/legacy.rb
fitting/lib/fitting/configuration.rb

show test details 🎭

need show more info about test

Pretty-print JSON Schemas in output

[Error] No such file or directory - drafter

When running this script:

rake 'fitting:documentation[s]'

Will produce error:

No such file or directory - drafter
rake aborted!
TypeError: no implicit conversion of nil into String

Tasks: TOP => fitting:documentation
(See full trace by running task with --trace)

upgrade node-forge

CVE-2020-7720

high severity
Vulnerable versions: < 0.10.0
Patched version: 0.10.0
The package node-forge before 0.10.0 is vulnerable to Prototype Pollution via the util.setPath function. Note: Version 0.10.0 is a breaking change removing the vulnerable functions.

new cover report

without js, only css and ruby rendor
because a lot dependabot alerts for this project

display tests_without_responses details on the main page 🌶

Do not output empty statistics in console report

Fully conforming requests:
<LOTS OF METHODS>

Partially conforming requests:


Non-conforming requests:


API requests with fully implemented responses: 12 (100.0% of 12).
API requests with partially implemented responses: 0 (0.0% of 12).
API requests with no implemented responses: 0 (0.0% of 12).

API responses conforming to the blueprint: 35 (100.0% of 35).
API responses with validation errors or untested: 0 (0.0% of 35).

I think that there's no need to output empty sections and statistics for them.

Add combinations join in new report 🎲

For the new fitting:report task, need add check combination json-schema validation, as in the report rake 'fitting:documentation_responses[s]', 'fitting:documentation_responses[m]' and 'fitting:documentation_responses[l]'

add console output for new report 🎮

fix build

how in esplanade
funbox/esplanade@d5b3233

update bundler

move json-schemas to separate files 📂

JSON-Schemas and their combinations must be placed in separate files so as not to overload the main report file.

fix dependency warning

gem build fitting.gemspec 
WARNING:  open-ended dependency on rake (>= 12.3.3, development) is not recommended
  if rake is semantically versioned, use:
    add_development_dependency 'rake', '~> 12.3', '>= 12.3.3'
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: fitting
  Version: 2.15.0
  File: fitting-2.15.0.gem

show tests without combinations 🎲

Show tests without combinations in action page

fix cover if response without combinations 💯

Now cover is always 100% if the response is without combinations

add test page 🧪

For the new fitting:report task, need add test page, with details on test

Use github actions for CI instead of Travis

Travis is not cool anymore. ;)

Fitting breaks with ArgumentError from output console

I have doc.abip with long name method api, as long as I running bundle e rake fitting:report
This is the full trace running with debug:

ArgumentError: negative argument
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:18:in `*'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:18:in `block (2 levels) in output'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:16:in `each'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:16:in `inject'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:16:in `block in output'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:13:in `each'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:13:in `inject'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/fitting/report/console.rb:13:in `output'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/fitting-2.17.0/lib/tasks/fitting.rake:84:in `block (2 levels) in <top (required)>'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/unilog-2.5.2/lib/unilog/rake_logging.rb:7:in `block in invoke'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/unilog-2.5.2/lib/unilog.rb:92:in `within_context'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/unilog-2.5.2/lib/unilog/rake_logging.rb:4:in `invoke'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/bin/ruby_executable_hooks:22:in `eval'
/Users/antonkolmakov/.rvm/gems/ruby-3.0.1/bin/ruby_executable_hooks:22:in `<main>'
Tasks: TOP => fitting:report

Add action page 📰

For the new fitting:report task, need add action page, with details on action

Not documented error is not included in the fitting/stats

If all methods is not documented in api.yml, they is not included in the stats.
Might be it also reproduce when some part of requests is not documented - I did not check it.

support parallel_tests

Need to create new dir and generate files "tests_#{ENV['TEST_ENV_NUMBER']}.json". In rake comand use all files in new dir.

delete multi_json

multi_json use for pretty json, bot can use default json lib

Fail test when respoonse status not match to any status from tomogram

There is get schemas with current response status in class Fitting::Response, but if status doesn't specify in tomogram it returns empty array and it pass validation. But must return not_documented.

color for bad response

if response without 100% cover, need pink color

Add responses join in new report 🔥

For the new fitting:report task, need add check response code and json-schema validation, as in the report rake 'fitting:documentation_responses[xs]'

make prefix optional

Zsh support

Have trouble with zsh terminal wrapper:

➜  bundle exec rake fitting:tests_responses[xs]
zsh: no matches found: fitting:tests_responses[xs]
➜  bundle exec rake fitting:tests_responses
need key xs

in bash all OK

Is there another way to set the size key?

white list doesn't work properly

I try to use white_list like this:

config.white_list = {
      '/smth' => ['GET']
    }

but in fitting/stats file this resourse placed in black list.

then I use white list like this:

    config.resource_white_list = {
      '/smth' => ['GET /api/v1/smth']
    }

and this works properly, but why I need to specify api prefix in array with methods descriptions?
Might be will be better to omit api prefix in this part too?

thank you for you time!

update tomograph

Output congratulations on 100% validation

Add actions join in new report 🌸

For the new fitting:report task, need add check METHOD and PATH (actions)

add accordion for prefixes 🎶

For new report need add accordion for prefixes. This will make it easier to interact with a large number of actions.

update ruby

Save tests for all prefixes 🏋️

Since we need to work with more than one prefix.
Old tasks still have to work the old fashioned way 😉.

do not check tests without combinations ☄️

fix directory error

you need to automatically create a fitting folder otherwise an error

bundle e rake fitting:report
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - fitting/report.json
/Users/dima/.asdf/installs/ruby/2.7.1/bin/bundle:23:in `load'
/Users/dima/.asdf/installs/ruby/2.7.1/bin/bundle:23:in `<main>'
Tasks: TOP => fitting:report
(See full trace by running task with --trace)

fix vulnerability CVE-2020-7660 🐛

GHSA-hxcc-f52p-wc94

fix cover if combinations without tests 0️⃣

Total response cover becomes zero If combinations without tests. Need fix it

Show more information in new report ❌

Expand the display of information about negative cases in a new report

Add design for checking prefixes 🖼

For the new fitting:report task, to check the prefixes, we need to add an html view

Disable documentation check by variable environment

It would good for development if it skip documentation checking.
For example:

SKIP_FIT=1 bin/rspec

Prefix checking 👀

To work with more than one prefix #35 we need to check them 👀

show tests without responses 🦦

Show tests without responses in action page

Undefined method 'not_coverage'

NoMethodError: undefined method `not_coverage' for #<Fitting::Route:0x007fabc5917c10>
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/fitting-1.6.0/lib/fitting/statistics.rb:12:in `not_coverage?'
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/fitting-1.6.0/lib/fitting.rb:48:in `run_specs'
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:93:in `run'
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:78:in `run'
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:45:in `invoke'
  /Users/vassilevsky/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.4.4/exe/rspec:4:in `<top (required)>'
  /Users/vassilevsky/.rbenv/versions/2.3.3-p0/bin/rspec:22:in `load'
  /Users/vassilevsky/.rbenv/versions/2.3.3-p0/bin/rspec:22:in `<top (required)>'

check test in new console 🧳

Add method for tests without actions Ⓜ️

Need add method name in list tests without actions

support swagger and openapi

Let's bump to tomograph '3.0.1'

check unnecessary parameters

strict
JSON::Validator.fully_validate(json_schema, body.to, strict: true)