Topic: data-processing Goto Github

👇 Here are 1040 public repositories matching this topic...

allenai / dolma
data-processing,Data and tools for generating and inspecting OLMo pre-training data.
Organization: allenai
Home Page: https://allenai.github.io/dolma/
data-processing large-language-models llm machile-learning nlp

alttch / rapidtables
data-processing,Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Organization: alttch
python python3 text-formatting data-processing library dictionary-data

apache / incubator-wayang
data-processing,Apache Wayang(incubating) is the first cross-platform data processing system.
Organization: apache
Home Page: https://wayang.incubator.apache.org/
cross-platform data-processing apache big-data data-management-platform distributed-system hadoop java jdbc middleware open-source performance scala spark

asyml / forte
data-processing,Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
machine-learning natural-language-processing deep-learning python text-data data-processing information-retrieval natural-language pipeline

asyml / texar
data-processing,Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert

asyml / texar-pytorch
data-processing,Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
machine-learning natural-language-processing pytorch deep-learning text-generation python machine-translation dialog-systems texar bert

benibela / xidel
data-processing,Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
User: benibela
Home Page: http://www.videlibri.de/xidel.html
xquery xml html json xpath cli command-line http web rest

brexhq / substation
data-processing,Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
Organization: brexhq
Home Page: https://substation.readme.io
aws serverless security etl go data-engineering data-processing

bytewax / bytewax
data-processing,Python Stream Processing
Organization: bytewax
Home Page: https://docs.bytewax.io/
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data

chenghaomou / text-dedup
data-processing,All-in-one text de-duplication
User: chenghaomou
text-processing de-duplication nlp data-processing

colasgael / machine-learning-for-solar-energy-prediction
data-processing,Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
User: colasgael
machine-learning neural-network data-processing python matlab tensorflow

constellation-rs / amadeus
data-processing,Harmonious distributed data analysis in Rust.
Organization: constellation-rs
Home Page: https://constellation.rs/amadeus
data-processing data-analysis stream-processing parallel-computing distributed-computing rust

dashbitco / broadway
data-processing,Concurrent and multi-stage data ingestion and data processing with Elixir
Organization: dashbitco
Home Page: https://elixir-broadway.org
elixir genstage data-ingestion data-processing concurrent broadway

flow-php / etl
data-processing,PHP - ETL (Extract Transform Load) data processing library
Organization: flow-php
Home Page: https://flow-php.com
etl data-engineering flow-php data-processing

googlecloudplatform / data-science-on-gcp
data-processing,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

googlecloudplatform / dataflowjavasdk
data-processing,Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Organization: googlecloudplatform
Home Page: http://cloud.google.com/dataflow
google-cloud-dataflow data-science data-analysis data-mining big-data data-processing

helmholtz-analytics / heat
data-processing,Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python
Organization: helmholtz-analytics
Home Page: https://heat.readthedocs.io/
gpu tensors distributed machine-learning mpi numpy python pytorch array-api data-analytics

hstreamdb / hstream
data-processing,HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Organization: hstreamdb
Home Page: https://hstream.io/
data-processing database distributed-database distributed-systems financial-analysis haskell hstreamdb iot iot-database kafka materialized-view real-time realtime-database scale sql stream-processing streaming streaming-data streaming-database

hxz393 / brutalityextractor
data-processing,适用于高性能系统的多进程解压缩软件(A multiprocess decompression software for high-performance system)
User: hxz393
brute-force computational-efficiency data-processing decompression extractor high-performance optimization parallel-computing parallel-processing performance-optimization

infoslack / awesome-kafka
data-processing,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

itechart / convtools-ita
data-processing,convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
Organization: itechart
Home Page: https://convtools.readthedocs.io
python code-generation transformations conversions data-preprocessing data-processing data-preparation functional-programming

johnkerl / miller
data-processing,Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
User: johnkerl
Home Page: https://miller.readthedocs.io
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction

keithorange / patternpy
data-processing,📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
User: keithorange
algorithmic-trading data-processing financial-analysis investment numpy open-source pandas pattern-recognition python stock-market strategies technical-analysis trading wealth-creation ohlcv-analysis

kousun12 / eternal
data-processing,👾~ music, eternal ~ 👾
User: kousun12
Home Page: https://eternal.rob.computer
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing

librecat / catmandu
data-processing,Catmandu - a data processing toolkit
Organization: librecat
Home Page: https://librecat.org
code4lib cli csv data-processing datamining ead json lido mab marc21

lithops-cloud / lithops
data-processing,A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Organization: lithops-cloud
Home Page: http://lithops.cloud
python big-data data-processing multicloud big-data-analytics serverless distributed object-storage parallel kubernetes

markus-wa / cq
data-processing,Clojure Query: A Command-line Data Processor for JSON, YAML, EDN, XML and more
User: markus-wa
cli clojure command-line csv data-processing data-transformation edn hacktoberfest json msgpack transformation xml yaml

matousc89 / padasip
data-processing,Python Adaptive Signal Processing
User: matousc89
adaptive-filtering signal-processing data-processing data-analysis machine-learning

maykulkarni / machine-learning-notebooks
data-processing,Machine Learning notebooks for refreshing concepts.
User: maykulkarni
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models

mech-lang / mech
data-processing,🦾 Main repository for the Mech programming language. Start here!
Organization: mech-lang
Home Page: http://mech-lang.org
ide programming-environment programming-language compiler data-processing reactive-programming live-programming language robotics

microsoft / dialogpt
data-processing,Large-scale pretraining for dialogue
Organization: microsoft
dialogue machine-learning pytorch transformer text-generation dialogpt gpt-2 text-data data-processing

microsoft / godel
data-processing,Large-scale pretrained models for goal-directed dialog
Organization: microsoft
Home Page: http://aka.ms/GODEL
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation

ml6team / fondant
data-processing,Production-ready data processing made easy and shareable
Organization: ml6team
Home Page: https://fondant.ai/en/stable/
data-processing fine-tuning foundation-models machine-learning pipeline python

msamogh / nonechucks
data-processing,Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
User: msamogh
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch

numaproj / numaflow
data-processing,Kubernetes-native platform to run massively parallel data/streaming jobs
Organization: numaproj
Home Page: https://numaflow.numaproj.io
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest

nvidia / dali
data-processing,A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Organization: nvidia
Home Page: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu gpu-tensorflow audio-processing pytorch mxnet paddle

onceupon / bash-oneliner
data-processing,A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
User: onceupon
Home Page: http://onceupon.github.io/Bash-Oneliner/
oneliner-commands bash data-processing linux-administration terminal linux variables grep xargs system xwindow hardware shell shell-oneliner one-liners

pathwaycom / pathway
data-processing,Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Organization: pathwaycom
Home Page: https://pathway.com
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing

polyaxon / haupt
data-processing,Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
Organization: polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib plotly bokeh mlops data-science data-visualization data-processing data-profiling tracking lineage serving

python-bonobo / bonobo
data-processing,Extract Transform Load for Python 3.5+
Organization: python-bonobo
Home Page: https://www.bonobo-project.org/
data-processing extract-transform-load bonobo python3 automation parallelization

pytlab / vaspy
data-processing,Manipulating VASP files with Python.
User: pytlab
Home Page: https://pypi.python.org/pypi/vaspy/
vasp data-processing data-visualization chemistry catalysis-informatics computational-chemistry

remotesensinginfo / rsgislib
data-processing,Remote Sensing and GIS Software Library; python module tools for processing spatial data.
Organization: remotesensinginfo
Home Page: https://www.rsgislib.org
remote sensing gis spatial data earth observation classification machine-learning remote-sensing

scramjetorg / scramjet
data-processing,Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.
Organization: scramjetorg
Home Page: https://www.scramjet.org
data-processing data-space data-stream edge-computing event-stream javascript python raspberry-pi reactive-programming transformations virtual-data-environment

sebkrantz / collapse
data-processing,Advanced and Fast Data Transformation in R
User: sebkrantz
Home Page: https://sebkrantz.github.io/collapse/
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted

senbox-org / snap-engine
data-processing,ESA Earth Observation Toolbox and Java Development Platform
Organization: senbox-org
Home Page: http://step.esa.int
data-processing data-visualization earth-observation eo raster-data remote-sensing linux macos windows

streamnative / pulsar-flink
data-processing,Elastic data processing with Apache Pulsar and Apache Flink
Organization: streamnative
apache-pulsar apache-flink data-processing batch-processing stream-processing flink flink-connector schema-registry schema sql

svenkreiss / pysparkling
data-processing,A pure Python implementation of Apache Spark's RDD and DStream interfaces.
User: svenkreiss
Home Page: https://pysparkling.readthedocs.io
data-science data-processing python apache-spark

tomwright / dasel
data-processing,Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
User: tomwright
Home Page: https://daseldocs.tomwright.me
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools

unionai-oss / pandera
data-processing,A light-weight, flexible, and expressive statistical data testing library
Organization: unionai-oss
Home Page: https://www.union.ai/pandera
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check

yord / pxi
data-processing,🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
User: yord
pxi pixie json csv ssv tsv dsv data-processing deserializer parser