Coder Social home page Coder Social logo

visokio / omniscope-custom-blocks Goto Github PK

View Code? Open in Web Editor NEW
5.0 8.0 3.0 6.2 MB

Public repository for custom blocks for Omniscope

Home Page: https://visokio.com/

R 31.55% Python 68.31% Shell 0.15%
business-intelligence data-science dataanalytics datapreparation python rstats

omniscope-custom-blocks's Introduction

Omniscope Custom Blocks · Refresh indexRefresh readme

Public repository for custom blocks for Omniscope Evo.

How to add a block to this repository

The simple way

  1. Design your custom block in Omniscope Evo 2020.1 or later. The source code should be reasonably documented and potentially contain sections to describe input fields and parameters.
  2. Export as a ZIP file from the block dialog.
  3. Send the file to [email protected] and we will include it for you.

The hard way

  1. Follow points 1-2 from the simple way.
  2. Fork the repository.
  3. Create or use a directory in the forked repository under one of the main sections that specifies the general area of what the block does.
  4. Extract the ZIP file into this directory.
  5. Consider adding a README.md for convenience, and a thumbnail.png.
  6. Run the python scripts create_index.py and create_readme.py located in the root of the repository.
  7. Create a pull request.

Table of blocks (Omniscope 2020+)

  1. Code & AI
    1. Execute Command
    2. AI Chat - OpenAI GPT
    3. AI Chat - Local LLM
    4. AI Chat - Claude
  2. Outputs
    1. BigQuery
      1. Google BigQuery Export
    2. Github
      1. GitHub
    3. PDF
      1. Web Image-PDF output
      2. Report tab to PDF
      3. Append PDF files
      4. Multi-tenant Report to PDF
    4. PowerPoint
      1. Report to PowerPoint
    5. Slack
      1. Slack Bot
  3. Analytics
    1. Clustering
      1. Gaussian Mixture Model
      2. DBScan
      3. KMeans
    2. Network Analysis
      1. Attribute Analysis
      2. TSNE
    3. Prediction
      1. Support Vector Machine
      2. K-Nearest-Neighbours
    4. Validation
      1. Model Validation
    5. Website
      1. Website Analysis
    6. Data Profiler
    7. Survival Analysis
  4. Connectors
    1. Azure
      1. Azure Data Lake Storage Gen2 Blob
    2. Flightstats
      1. Flightstats Airlines
      2. Flightstats Airports
      3. Flightstats Flights
    3. Overpass
      1. Overpass Street Coordinates
    4. Slack
      1. Slack API WebClient
    5. Weather
      1. OpenWeatherMap
    6. Google BigQuery Custom SQL
    7. Google BigQuery Import Table
    8. Trello
    9. HubSpot
    10. XPT Reader
    11. Jira
    12. Dune
    13. Etherscan
    14. Flipside
    15. Yahoo Finance
  5. Inputs
    1. Databases
      1. MongoDB
    2. R
      1. R Data Reader
      2. Rds Batch Append
    3. Sharepoint Online Downloader
    4. SFTP Downloader
    5. PDF Reader
  6. Preparation
    1. ForEach
      1. ForEach multi stage
      2. Project Parameters Batch Setting
    2. Geo
      1. Shapefile
      2. Gridsquare
    3. Interfaces
      1. Kedro
    4. JSON
      1. JSON Normalise
    5. Join
      1. Interval Join
      2. Inequality Join
      3. Fuzzy Terms Join
    6. Partition
      1. Partition
    7. Pivot
      1. Melt De-pivot
    8. Standardisation
      1. Standardise
    9. Workflow
      1. For Each (Separate Workflows)
    10. Unescape HTML
    11. Field Renamer
    12. Unstack Records
    13. URL Encode
    14. Markdown to HTML
    15. Split Address
    16. Add row ID field

Block Overview

Execute Command

Execute a system command.

Link to Github page

AI Chat - OpenAI GPT

Executes a one-off prompt to OpenAI GPT and returns the generated text result

Link to Github page

AI Chat - Local LLM

Executes a one-off prompt to a local LLM and returns the generated text result

Link to Github page

AI Chat - Claude

Executes a one-off prompt to Anthropic Claude and returns the generated text result

Link to Github page

Google BigQuery Export

Allows to write data to a Google BigQuery table. The table can be created/replaced, or records can be appended to an existing table

Link to Github page

GitHub

Reads from and writes data to GitHub

Link to Github page

Web Image-PDF output

Grabs screenshots of webpages, optionally producing a PDF document.

Link to Github page

Report tab to PDF

Prints Report tabs to PDF files for each record of the input data.

Link to Github page

Append PDF files

Append multiple PDF files combining them into one PDF file.

Link to Github page

Multi-tenant Report to PDF

Prints Report tabs to PDF files for each record of the input data.

Link to Github page

Report to PowerPoint

Export a Report to a PowerPoint pptx file

Link to Github page

Slack Bot

Posts messages on a channel.

Link to Github page

Gaussian Mixture Model

Performs GMM clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.

Link to Github page

DBScan

Performs DBScan clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.

Link to Github page

KMeans

Performs KMeans clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.

Link to Github page

Attribute Analysis

Given a dataset in which each record represents an edge between two nodes of a network, and each node has an associated categorical attribute, the block analyses connections between attributes, based on connections between associated nodes. The result of the analysis is a list of records in which each record specifies a connection from one attribute to another. The connection contains a probability field, which gives an answer to the question that if a node has the specified categorical attribute, how probable it is that it has a connection to another node with the linked categorical attribute.

Link to Github page

TSNE

Given a dataset in which each record represents an edge between two nodes of a network, the block will project all the nodes onto a (e.g. 2)- dimensional plane in such a way that nodes which share many connections are close together, and nodes that do not share many connections are far apart.

Link to Github page

Support Vector Machine

Predicts classes of new data from old data by drawing a boundary between two classes whereas the margin around the bondary is made as large as possible to avoid touching the points.

Link to Github page

K-Nearest-Neighbours

Performs k-nearest-neighbour prediction on the data. The prediction for a new point depends on the k-nearest-neighbours around the point. The majority class is used as the prediction.

Link to Github page

Model Validation

Computes a confusion matrix as well as model validation statistics

Link to Github page

Website Analysis

Extracts the structure and content of a website and its pages.

Link to Github page

Data Profiler

Provides detailed statistics about a dataset

Link to Github page

Survival Analysis

Computes an estimate of a survival curve for truncated and/or censored data using the Kaplan-Meier or Fleming-Harrington method

Link to Github page

Azure Data Lake Storage Gen2 Blob

Storage Gen2 Blob connector to load a CSV or Parquet blob/file in Omniscope.

Link to Github page

Flightstats Airlines

Downloads a list of airlines as provided by flightstats (https://www.flightstats.com). The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.

Link to Github page

Flightstats Airports

Downloads a list of airports as provided by flightstats (https://www.flightstats.com). The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.

Link to Github page

Flightstats Flights

Requests information about flights specified in the input data from flightstats (https://www.flightstats.com). If the flight exists the result will contain live information, otherwise it will not be part of it. The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.

Link to Github page

Overpass Street Coordinates

Finds all matching streets given a street name and requests multiple coordinates along the street using data from Overpass API. It will create a row for each point found that is part of a street that matches the given street name. The resulting rows will include the street name, the street Id and the coordinates of the point. The script needs an input with a field with the street name.

Link to Github page

Slack API WebClient

Allows you to call public Slack endpoints.

Link to Github page

OpenWeatherMap

Retrieves current weather and forecasts from OpenWeatherMap

Link to Github page

Google BigQuery Custom SQL

Executes a SQL query on Google BigQuery and imports the query results

Link to Github page

Google BigQuery Import Table

Allows to import a table from Google BigQuery.

Link to Github page

Trello

Retrieves boards, lists and cards, and allows you to search in Trello.

Link to Github page

HubSpot

Retrieves contacts, companies, deals and lists

Link to Github page

XPT Reader

Reads a SAS Transport xpt file, extracting a dataset.

Link to Github page

Jira

Retrieves projects and issues from Jira

Link to Github page

Dune

Execute queries and retrieve blockchain data from any public query on dune.com, as well as any personal private queries your Dune account has access to

Link to Github page

Etherscan

The Ethereum Blockchain Explorer.

Link to Github page

Flipside

Executes a SQL query on Flipside and retrieves the blockchain data

Link to Github page

Yahoo Finance

Fetches price data for tickers from Yahoo Finance

Link to Github page

MongoDB

A connector for MongoDB

Link to Github page

R Data Reader

Joins regions defined in a shapefile with points defined as latitudes and longitudes, and gives meta information about the content of the shapefile

Link to Github page

Rds Batch Append

Reads multiple rds files either from an upstream block, or a folder, and appends them

Link to Github page

Sharepoint Online Downloader

Download a file from a Sharepoint Online site.

Link to Github page

SFTP Downloader

Download files from a SFTP server folder.

Link to Github page

PDF Reader

Extract text from PDF files.

Link to Github page

ForEach multi stage

The ForEach multi stage block allows to orchestrate the execution of another Omniscope project and running the workflow multiple times, each time with a different set of parameter values. Unlike the ForEach block allows multiple stages of execution, executing/refreshing from source a different set of blocks in each stage.

Link to Github page

Project Parameters Batch Setting

None

Link to Github page

Shapefile

Match regions in shapefile with geographical points having latitude and longitude

Link to Github page

Gridsquare

Converts gridsquare / Maidenhead

Link to Github page

Kedro

Intefaces with kedro workflows

Link to Github page

JSON Normalise

Normalise semi-structured JSON strings into a flat table, appending data record by record.

Link to Github page

Interval Join

Performs a join between values in the first input and intervals in the second input. Rows are joined if the value is contained in an interval.

Link to Github page

Inequality Join

Performs a join between the first (left) and second (right) input. The join can be performed using equality/inequality comparators ==, <=, >=, <, > , which means the result will be a constraint cartesian join including all records that match the inequalities.

Link to Github page

Fuzzy Terms Join

Performs a join between the first (left) and second (right) input. The field on which the join is performed must be text containing multiple terms. The result will contain joined records based on how many terms they share, weighted by inverse document frequency.

Link to Github page

Partition

Partitions the data into chunks of the desired size. There will be a new field called "Partition" which contains a number unique to each partition.

Link to Github page

Melt De-pivot

Keep all selected fixed fields in the output, de-pivot all other fields

Link to Github page

Standardise

Standardises the values in the selected fields so that they are in the range between 0 and 1. I.e. The new value of the highest value in each field is going to be 1, and the lowest value 0. All other values are scaled proportionally.

Link to Github page

For Each (Separate Workflows)

Executes another Omniscope project multiple times, each time with a different set of parameter values.

Link to Github page

Unescape HTML

Convert all named and numeric character references to the corresponding Unicode characters

Link to Github page

Field Renamer

Renames the fields of a data set given a list of current names and new names.

Link to Github page

Unstack Records

Unstack all records by splitting on text fields with stacked values, filling records with empty strings where needed.

Link to Github page

URL Encode

URL encode strings in a field using the UTF-8 encoding scheme

Link to Github page

Markdown to HTML

None

Link to Github page

Split Address

Splits an address field into streetname, streetnumber, and suffix.

Link to Github page

Add row ID field

Adds a Row ID field with a sequential number.

Link to Github page

omniscope-custom-blocks's People

Contributors

actions-user avatar guillemsegura avatar manuelpagliaivisokio avatar michael-visokio avatar michalkowalczykopc avatar nils-drechsel avatar supersteves avatar toniopoggi avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

omniscope-custom-blocks's Issues

Omniscope.api reference

I saw on on "DBScan" script a reference to Omniscope.api what is this and where can I find out any information about it? Can any info be put on the README.md i.e. location of it on your computer for instance

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.