shuffle / openapi-apps

Swagger/ OpenAPI specifications for security products and services

Shuffle Automation

Shuffle is an automation platform for and by the community, focusing on accessibility for anyone to automate. Security operations is complex, but it doesn't have to be.

Key Features — Community & Support — Documentation — Getting Started — Development

Follow us on Twitter at @shuffleio.

Try it

• Self-hosted: Check out the installation guide
• Cloud: Register at https://shuffler.io/register and get cooking (missing a lot of features)

Please consider sponsoring the project if you want to see more rapid development.

Support

• Discord
• Twitter
• Email
• Open issue
• Shuffler.io

Blogposts

• 1. Introducing Shuffle
• 2. Getting started with Shuffle
• 3. Integrating Shuffle with Virustotal and TheHive
• 4. Real-time executions with TheHive, Cortex and MISP

Documentation

Documentation can be found on https://shuffler.io/docs and is written here: https://github.com/shuffle/shuffle-docs.

Setting up a local development environment

Please follow the steps mentioned here!

Related repositories

• OpenAPI apps: https://github.com/shuffle/security-openapis
• Documentation: https://github.com/shuffle/shuffle-docs
• Workflows: https://github.com/shuffle/shuffle-workflows
• Python apps: https://github.com/shuffle/shuffle-apps

Features

• Simple, feature rich workflow editor
• App creator using OpenAPI
• Premade apps for your security tools
• Organization and sub-organization control
• Hybrid resource sharing with shuffler.io (optional)

Website

https://shuffler.io

Contributing

We want to make the world of cybersecurity more accessible and need all the help we can get. Send an email to support@shuffler and we'll make sure to give you any training you may need.

These are the main areas to contribute in:

• Frontend (ReactJS)
• Backend (Golang)
• App Creation (Python & GUI w/OpenAPI)
• Documentation (Markdown)
• Workflow creation (GUI & Conceptualizing)
• Content Creation (Blogs, videos etc)

Contributing guidelines are outlined here.

Contributors

Shuffle

App magicians

License

All modular information related to Shuffle will be under MIT (anyone can use it for whatever purpose), with Shuffle itself using AGPLv3.

Workflows: MIT Documentation: MIT Shuffle backend: AGPLv3 Apps, specification and App SDK: MIT

Architecture

Repository overview

Below is the folder structure with a short explanation

├── README.md				# What you're reading right now
├── backend					# Contains backend related code.
│   ├── go-app 			# The backend golang webserver
│   └── app_sdk			# The SDK used for apps
├── frontend				# Contains frontend code. ReactJS, Material UI and cytoscape
├── functions				# Has execution and extension resources, such as the Wazuh integration
│   ├── onprem				# Code for onprem solutions
│   │   ├── Orborus 	# Distributes execution locations
│   │   ├── Worker		# Runs a workflow
└ docker-compose.yml 	# Used for deployments

Get in touch, send a mail to [email protected] or poke me on twitter @frikkylikeme

openapi-apps's Issues

Aqua

Fix and Create Google Chat app (communication)

Fix import and merge errors for OpenAPI specs

Example that has problems: https://connect.withsecure.com/getting-started

Sample problems:

• Body & Response not parsed
• Some actions without names
• Names and descriptions missing
• App authentication is weird

E.g. for this the URL Below: Fix remote reference loading in real-time

https://connect.withsecure.com/specs/security-events/openapi.yaml

Abuseipdb builtin app endpoint path is wrong for network block check

The endpoint should be as below for check-block but the abuseipdb app comes with Shuffle inside has the path error.

openapi-apps/abuseipdb.yaml

Line 22 in 73c7607

/check-block/json:

In the Shuffle the app calls check_block so it gives error.

It is downloaded from cloud i think so it should be corrected from there.

Custom app bug

When I create a custom app and add it to a workflow, if I modify it after (to add an action, for example), when I use the app in the same workflow, in never comes back with results and runs forever. I'll try to reproduce and show a printscreen

AWS WAF

Add OpenAPI apps from

https://github.com/serviceitsecurity/openapis

RE: They contacted us to release them :)

Add Carbon Black

• Cloud
• Response
• Defense

Microsoft Modern Workplace M365

Redmine

Trellix

IRIS: Fix alert structures with underscores

UI bug

When I create a custom app and then delete it, I see this on the right in the Apps panel:

how to use openapi and shuffle?

Hi @frikky ,
I'm pretty new to openapi development workflow, so bear with me.
I'm pretty confused.
I understand what openapi does, but I just don't understand what is the suggested workflow with using it within shuffle. I more or less understood how to use openapi normally and how to use it to make custom shuffle apps. What I don't understand is how (if possible at all) to use openapi specifications on their own.
Maybe this issue could come up useful for other superbeginners like myself 😅

Normal openapi workflow:

• the user develops and deploys the api server by himself

1. an api can be described with openapi specification, which is awesome
2. server code and docs can be generated automatically, which is awesome
3. user still needs to implement the functionality of the endpoints
4. user still needs to package the project in a deployable format (for example with a docker flask image)

Normal shuffle app workflow:

• the user develops the api server but deployment is taken care by shuffle (?)

1. write openapi specification
2. (?) generate an app.py from it. How? is there a specific generator to use? I'm using

docker run --rm 
-v $PWD:/local openapitools/openapi-generator-cli generate \
-i /local/openapi/api.yml \
-g python-flask -o /local/codegen

3. (?) alter the app.py to use from walkoff_app_sdk.app_base import AppBase and to look like your example
4. package the app.py in a Dockerfile based on frikky/shuffle docker image as by example

Openapi shuffle app workflow

• the user writes openapi specification (like those in this github repository) and then automagically everything works (???)

1. write openapi specification
2. (?) put it where?
3. (?) shuffle loads it and deploys an app somehow?

---

As I wrote this, I started to think that:

• Openapi shuffle app workflow does not exist
• the openapi declarations of this repository simply describe api running on a remotely and independently deployed server (for example by following the first workflow)

My questions, if the above is correct, is then:

• how should I build my app? why should I deploy it as a shuffle app when I can just build it as a regular flask app? what is the advantage? ease of deployment I suppose? Will a regular flask app work with shuffle?
• If I don't develop a shuffle app and go with regular Flask app, where should I place the openapi declaration in my shuffle deployment?

Auto label pls

OS ticket

Issue with the sentinelone open api authentication

When trying to run an action in the S1 API (https://github.com/Shuffle/openapi-apps/blob/master/sentinelone.yaml), I get the "Authentication Failed" error.
I believe it is because the header needs to be in this format:
Authorization: Token TheToken

Source (line 43): https://github.com/collectivehealth/sentinelone-sdk/blob/master/sentinelone.py

App fails after adding a new endpoint

Hi

I have a custom OpenAPI app which was working just fine. This morning I have added an additional action to it. The version got changed from 1.0.0 to 1.1.0. When I active the new version in a workflow, the app fails. Also it would be cool if we could just choose the version of an OpenAPI app in the GUI like with the custom apps.

{"success":false,"reason":"An exception occurred while running this function. See exception for more details and contact support if this persists ([email protected])","exception":"'Vectra3596be88ff136d1a90ca7e6db7e7c30e' object has no attribute 'prepare_response'"}

Even when I deleted the new action, it still fails. Do you have any idea what could be the cause here?

Also I am wondering regarding the version system of these apps. I would expect, that the version is changing with every change I make. But this seems not to be the case here. Changed it multiple times today, still on version 1.1.0. Can I somehow force a version change?

Currently running version 1.1.0 of Shuffle, maybe the version is coming from that?

Regards tbi88

App documentation holding onto the documentation from a previously viewed app

When searching for an app then selecting it, then viewing the documentation.

If an app was selected before it, and the documentation was viewed, you will see the documentation from a previously viewed app.

For example, if you search for and select the Sentinelone app then look at the documentation, then do the same and open the Huntress app, you will see the Sentinelone documentation instead of it changing to the Huntress docs :( .

Add Carbon Black

• Cloud
• Response
• Defense

Bug adding a node to an existing workflow

Starting from the same workflow as in the bug #40 I am trying to add a node "router" between node A and node B to see if that would fix. I delete the link between the 2 and add a Shuffle tool As soon as I select "Router", I get a blank windows with this error:

TypeError: Cannot read properties of null (reading 'find')
    at bundle.js:2:1723545
    at sl (bundle.js:2:2644966)
    at ks (bundle.js:2:2663780)
    at t.unstable_runWithPriority (bundle.js:2:2682022)
    at qa (bundle.js:2:2606849)
    at Cs (bundle.js:2:2663539)
    at ls (bundle.js:2:2655315)
    at bundle.js:2:2607140
    at t.unstable_runWithPriority (bundle.js:2:2682022)
    at qa (bundle.js:2:2606849)
Uncaught TypeError: Cannot read properties of null (reading 'find')
    at bundle.js:2:1723545
    at sl (bundle.js:2:2644966)
    at ks (bundle.js:2:2663780)
    at t.unstable_runWithPriority (bundle.js:2:2682022)
    at qa (bundle.js:2:2606849)
    at Cs (bundle.js:2:2663539)
    at ls (bundle.js:2:2655315)
    at bundle.js:2:2607140
    at t.unstable_runWithPriority (bundle.js:2:2682022)
    at qa (bundle.js:2:2606849)

I don't know if it's link, but just before changing to "router", just after adding the shuffle tool, I have a message in the console saying:

Returning because node is not valid:  {id: (...)}

MISP OpenAPI spec validation error

The follow error is appearing in the shuffle-backend container

2022/08/30 11:41:26 [WARNING] Yaml error (1): yaml: line 4522: found unknown escape character
2022/08/30 11:41:26 [WARNING] Yaml error (2): yaml: line 4522: found unknown escape character
2022/08/30 11:41:26 [WARNING] Validation error for misp.yaml: yaml: line 4522: found unknown escape character

Create OpenCTI app

RE: Shuffle/python-apps#1

Escape https://escape.tech/

MaxPatrol Siem

Github OpenAPI

Github has published its own openapi specification located here: https://github.com/github/rest-api-description/blob/main/descriptions-next/api.github.com/api.github.com.yaml, which adds additional capabilities. The yaml file is located here: https://github.com/Shuffle/openapi-apps/blob/master/github_notifications.yaml.

There's probably a way to reference that file dynamically.

Add QRadar

Apparently, a lot of people use QRadar 👯

malwarebazaar

Misformatted JIRA API v2 APP Headers Section

Describe the bug
When using/uploading JIRA API v2 app, the default JSON Headers are formatted incorrectly

To Reproduce
Steps to reproduce the behavior:

1. Download and install the JIRA v2 API for reproductions
   ...

Expected behavior
Expected to run as normal, screen shot included simple fix.

** Debug logs (NOT APPLICABLE FOR CLOUD)**
v2](https://user-images.githubusercontent.com/32825945/177338228-38c4f710-33a4-40c5-84d1-809c933e4929.png)

Fix Openapi spec for F-Secure

This has been exported, but can't be reimported for some reason. Will need to change the openapi parser in Go or change validation slightly.

IceWarp

Bug in a workflow with 2 path to the same app

I have the following workflow with 2 path possible to the same node:
Option 1: if $exec.fields contains computerName, do node A, then do node B
Option 2: if $exec.fields does not contains computerName, do node B.

However, when testing with a case where the Option 1 should be followed, the node B is never executed because the condition for the Option 2 is not fulfilled (that's the reason given when looking at the output of the skipped node B).

Add icons to all apps

Some apps are missing an icon. Fix it.

AWS SES

Hello,

The AWS SES app is missing the possibility to use the "reply to" field of an email

Parse HTML code

When uploading open API JSON or YAML file which contain HTML code, Shuffle is unable to parse it.

malwarebytes

Greenbone

Connect with Varonis

https://www.varonis.com/apps?hsLang=en

Create Microsoft Intune app for assets

Add Patrowl

Multiple requests.. :)

Nagios

label pls

Add hashdd and dehashed.com

https://twitter.com/Frikkylikeme/status/1271099680240267264

runZero (previously Rumble Network Discovery)

Adding support for runZero. runZero is a visibility tool that is used for asset discovery in the network. It could be leverage for incident response in the case that someone wants to do device discovery in the network. It is a good network asset management tool.

https://app.swaggerhub.com/apiproxy/schema/file/apis/runZero/runZero/3.1.0?format=yaml

Add bitdefender app based on API

https://www.bitdefender.com/business/support/en/77211-125280-getting-started.html

Seems easy enough to build in the app creator

Connect with Ivanti Heat

https://help.ivanti.com/ht/help/en_US/ISM/2020/admin/Content/Configure/API/RestAPI-Introduction.htm

Workflow - conditions error

Workflow not take care about conditions.
It looks like from some reason from time to time (I don't know why), the workflow have to end even the condition should stop flow earlier.

frontend:nightly
backend:nightly
orborus:nightly
worker:latest

The same situation is on subflow. Subflow getting no data but still executing flow.

Auto label 2

Index search fails in Elasticsearch SIEM app

Hi,

Using the json downloaded from https://shuffler.io/apps/971706758e274c2e4083f2621fb5a6f7, i try to create an application for Elasticsearch SIEM in Shuffle.

For the starters, my use case is to perform an index search and based in the result, register a case in TheHive.
However, while executing the Elasticsearch node in shuffle, the POST index search is not completing and the corresponding container exits with syntax error.

CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS                      PORTS                                                                            NAMES
9bfa1bdd8857   frikky/shuffle:elasticsearch_agg_1.0.0                "/bin/sh -c 'python …"   18 seconds ago   Exited (1) 16 seconds ago                                                                                    Elasticsearch_Agg_1-0-0_48dd0a59-15fe-49b3-aec8-0b1cf90b77b2_61aad87a-d982-4ef8-9d1b-5597346bc677

root@vmsoar:/home/user/SOAR/Elasticsearch-SIEM# docker logs 9bfa1bdd8857
  File "/app/app.py", line 1171
    %!s(MISSING).run()
    ^
SyntaxError: invalid syntax  

I guess there is something wrong with the JSON and i am unable to figure that out.
Could you please help me fix the issue ?

Ansible