bbva / kapow

Kapow! If you can script it, you can HTTP it.

If you can script it, you can HTTP it.

---

What's Kapow!

Say we have a nice cozy shell command that solves our problem. Kapow! lets us easily turn that into an HTTP API.

Let's see this with an example

We want to expose log entries for files not found on our Apache Web Server, as an HTTP API. With Kapow! we just need to write this executable script:

[apache-host]$ cat search-apache-errors
#!/usr/bin/env sh
kapow route add /apache-errors - <<-'EOF'
	cat /var/log/apache2/access.log | grep 'File does not exist' | kapow set /response/body
EOF
[apache-host]$ chmod +x search-apache-errors

and then, run it using Kapow!

[apache-host]$ kapow server search-apache-errors

finally, we can read from the just-defined endpoint:

[another-host]$ curl http://apache-host:8080/apache-errors
[Fri Feb 01 22:07:57.154391 2019] [core:info] [pid 7:tid 140284200093440] [client 172.17.0.1:50756] AH00128: File does not exist: /usr/var/www/mysite/favicon.ico
[Fri Feb 01 22:07:57.808291 2019] [core:info] [pid 8:tid 140284216878848] [client 172.17.0.1:50758] AH00128: File does not exist: /usr/var/www/mysite/favicon.ico
[Fri Feb 01 22:07:57.878149 2019] [core:info] [pid 8:tid 140284208486144] [client 172.17.0.1:50758] AH00128: File does not exist: /usr/var/www/mysite/favicon.ico
...

Why Kapow! shines in these cases

• We can share information without having to grant SSH access to anybody.
• We can share information easily over HTTP.
• We can effectively limit what gets executed.

Documentation

You can find the complete documentation and examples here.

Security

Please consider the following Security Concerns before using Kapow!

If you are not 100% sure about what you are doing we recommend not using Kapow!

Authors

Kapow! is being developed by BBVA-Labs Security team members.

Kapow! is Open Source Software and available under the Apache 2 license.

Contributions

Contributions are of course welcome. See CONTRIBUTING or skim existing tickets to see where you could help out.

---

• Did you spot the UUoC? Congrats! We are hoping to win this year :)

kapow's Issues

POST /routes should return info of newly created route

As per the spec¹, the call should return a JSON body with the info
of the newly created route. Should we consider this in the mocked server
and in the test?

¹: https://github.com/BBVA/kapow/tree/master/spec#insert-a-route

kapow/internal/client/route_add_test.go

Line 24 in ee00a7b

// TODO: As per the spec¹, the call should return a JSON body with the info

Fix control API delete operation

Headers, ADD or OVERWRITE?

At this moment setResponseHeader adds values to the header with every call (a PUT). Should we use POST for this and PUT to overwrite?

Status code accessing non-existent form

What should the return code be when there is no form in the target request and access to a field is requested?

Explicit request response life-cycle

Partially implemented in branch feature/respond.

Add a mechanism to avoid responding when the shell finishes.

Header key/value encoding validation

GO's implementation doesn't make any header key/value encoding validation. We need to add this validation.

Add to spec: Cookie keys are CaseSensitive

Add a note in the specification syaing that cookie keys are case sensitive.

Implement missing reason phrase in control and data API

Manage args for url_pattern and command_file (exactly two arguments)

kapow/internal/cmd/route.go

Line 31 in be6a2ef

// TODO: Manage args for url_pattern and command_file (2 exact args)

Define route specification

We need to write the route specification to make clear what attibutes are mandatory and the attributes domain (if any is needed)

cmd/route: read command from parameter, file or stdin

kapow/internal/cmd/route.go

Line 43 in be6a2ef

// TODO: Read command from parameter, file or stdin

Status response resource doesn't allow to set reason phrase value

Currently the data API only allows to set status code, there is no way to set a reason phrase. As said by specification the reason phrase is an important element in order to provide valuable information to human beings so Kapow! users should be allowed to set this value.

Default values for the Route entity must be set on the create/insert handler as the server has all the information needed to do so

kapow/internal/cmd/route.go

Line 49 in be6a2ef

// TODO: Add default values for flags and remove path flag

Allow any HTTP status code

GO's current implemetation doesn't allow to set an arbitrary response status trough WriteHeader. We need a way by which circumvent this.

Find the best tech stack for Kapow!

The code is a PoC, a dirty one. No test. No desing. Just a prof that is possible to do Kapow!.

The best tech stack for Kapow! is out there, and we'll find it.

Ensure consistent handling of binding and consumer params

The --url flag and KAPOW_URL env var are no longer sufficient because the control server which they pointed to has been split in two: control and data.

Thus, we need to honour this split. Let's be specific:

kapow route should honour the KAPOW_CONTROL_URL env var and the --control-url flag, with a default value of http://localhost:8081.

kapow server should honour the --control-bind flag, with a default value of localhost:8081.

kapow server should honour the --bind flag, or its alias --user-bind, with a default value of localhost:8080.

kapow server should honour the --data-bind flag, with a default value of localhost:8082.

kapow get|set should honour the KAPOW_DATA_URL env var and the
--data-url flag, with a default value of http://localhost:8082.

For completion, although this hasn't changed, kapow get|set should honour the KAPOW_HANDLER_ID env var and the --handler-id flag. No default value.

Note that flags take precedence over env vars.

*.pow not working without final break line

When you try lo load more than one .pow file, using *.pow, if you don't put "\n" at ending of each file, kapow doesn't works.

Get original endpoint creation info... in an endpoint

Kapow! have no way to recover the original creation info of a given endpoint. This will be a good enhancement because if this endpoint gives you the data with the same format you can feed another Kapow! instance with the info of other Kapow!

Avoid negative number on endpoint index

Ask yourself, what iptables would do?
Maybe avoid a negative index throwing an error?

Validate url_pattern format in control API append operation

Implement GET /routes/{routeID} operation in control API

Propagate shell exit status in Kapow! server

When the shell started by Kapow! server ends with error this error code is not returned from Kapow! so if exited with error the error code is missed.

Kapow! server reject connections after launch long time service

Using this .pow file:

bash kapow route add -X POST /tools/network/nmap -c 'nmap -v -Pn -n -sS -sS -A "$(request /form/ip)" | response /stream'

After launch Kapow! this above rules and launch 3 executions and cancel them (with a CTRL+C), the Kapow! service reject connections until one nmap execution finishes.

It appears that, when CTRL+C was break the connection with Kapow! serve, it doesn't stop the nmap background process.

Screenshot of curl calling the service:

Screenshot of the Kapow! console:

Add /route branch to resources

Add a new /route branch to the resources tree so the external process can query not only the current request data, but the data of the route its being processed.

As suggested by @hhurtado.

Simpler error reporting

Too specific errors are too complex rigth now. Instead 400 and several 422 throw only a 422 malformed entity.

Data handler success feature

Implement in the E2E suite the success feature on data spec.

Unexpected dummy return

As per the spec, this call should return the actual route inserted, with the final values for all its fields.

kapow/internal/server/user/state.go

Line 33 in f5d7521

return model.Route{Index: l - 1}

Co-authored-by: César Gallego [email protected]

Check for gock.IsDone() in all the tests using gock

Example: https://github.com/BBVA/kapow/blob/a865ade26a12f2271fa5796b45fd7f13b684d198/internal/client/route_remove_test.go

Default values managed in control package

The default values for the Route entity must be set on the create/insert handler as the server has all the information needed to do so

Delete should not have Content-Type

kapow/internal/client/route_remove.go

Line 9 in a865ade

return http.Delete(url, "application/json", nil, nil)

• Change spec
• Change code

server/user/state.go: refactor with `defer` if applicable

kapow/internal/server/user/state.go

Line 54 in 034a5ab

// TODO: Refactor with `defer` if applicable

Generate routeID from control API

As discussed we let the "client" in this case the control API layer to generate the route ID to allow in the future users to give a custom routeID name.

Maybe the job of the user state layer is only to validate data on append to avoid collisions or malformed inputs.

Related to #39

Implement E2E test suite

Implement a test suite to check any Kapow! implementation compliance against the specification. The test suite must contain at least:

• HTTP Control API
• HTTP Data API
• Kapow! Framework

This test suite must be independent of the implementation and cannot be coupled with the current or future implementations under any circumstances.

Change 204 to 404 when No Item Found

kapow/internal/client/set.go

Line 9 in a865ade

// TODO: Review spec: Data API > Error responses > 204 Resource Item Not Found should not be 2xx

• Change spec
• Change tests
• Change poc
• Change code

Cookies, ADD or OVERWRITE?

At this moment setResponseCookie adds values to the cookie with every call (a PUT). Should we use POST for this and PUT to overwrite?

Cookie key/value encoding validation

GO's implementation doesn't make any cookie key/value encoding validation. We need to add this validation.

Implement insert operation in control API

Use metasyntactic variables

This is disturbing 😅; please use standard metasyntactic variables (foo, bar, baz...) or descriptive text content (examplebody, exampleheader, ...) as example data for tests.

Support authentication mechanisms

make on poc failing on fresh install

On an empty env gherking-lint is not found. Maybe need pipenv install --dev and added dependency.

Encryption support

Write an interactive tutorial

Continue the works in branch tutorial

Handle execution of .pow files gracefully.

*.pow files can't be executed directly.

They should be executed either running kapow server foo.pow or within an interactive kapow shell. Otherwise, it won't work, and it fails with a call stack trace, like this:

$ ./nmap.pow 
Traceback (most recent call last):
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/bin/kapow", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/pancho/sandbox/bbva/kapow/poc/bin/kapow", line 632, in <module>
    kapow()
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/pancho/sandbox/bbva/kapow/poc/bin/kapow", line 573, in route_add
    "command": source})
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/models.py", line 313, in prepare
    self.prepare_url(url, params)
  File "/home/pancho/.local/share/virtualenvs/poc-DayB6nJe/lib/python3.7/site-packages/requests/models.py", line 387, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'None/routes': No schema supplied. Perhaps you meant http://None/routes?

We should handle this case more gracefully, and return an explicit error message when it happens.

NOTES:

One way to solve this would be:

1. Adding the shebang line to all *.pow files:
   #!/usr/bin/env -S kapow server.
2. Modify the server code so that it won't spawn a second server if the environment variable KAPOW_URL is already defined, which happens when running within an interactive kapow shell.

This could be hard to implement, so we could settle for less:

1. Detect the condition described above and exit with a descriptive error message.
2. If the #!/usr/bin/env -S kapow server is in place and you try to run it within an interactive kapow shell, exit notifying the user that the .pow file shoud be run like this: bash -c foo.pow.

Kapow! debug operation

In order to simplify the Kapow! endpoint creation process a debug operation flag will be great. In this operation mode Kapow! will respond you with full stacktrace of every error.
This debug operation must be avoid on production environments.

Static binary distribution

Currently Kapow! is a PoC and the interpreter was written in Python.

The idea for a next improved version is: Distribute Kapow! as a static binary without any library dependency to improve the portability

Batch Kapow remote feed

As Kapow admin I want to replicate a running kapow from a newly fresh kapow, even if i have no original script files.

Multivalued params

We need to define how are we going to return the values associated to a param (/request/params/{name})

Parse forms in request

The GO's implementation doesn't allow to decode form encoded data sent in a request with an arbitrary method and/or content type other than "application/x-www-form-urlencoded". This is needed for Kapow! semantic so it MUST be changed in the future.

Implement a ParseForm function that doesn't care about method nor Content-Type.

Interrupted access to a file

What should the behaviour be when an error occurs accesing the content of a file?

Create setup.py file for the PoC