Topic: data-processing Goto Github

👇 Here are 1088 public repositories matching this topic...

airscholar / e2e-data-engineering
data-processing,An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
User: airscholar
Home Page: https://www.youtube.com/watch?v=GqAcTrqKcrY
apache-airflow apache-kafka apache-spark apache-zookeeper big-data cassandra containerization data-engineering data-pipeline data-processing data-storage docker etl-pipeline postgresql real-time-analytics

allenai / dolma
data-processing,Data and tools for generating and inspecting OLMo pre-training data.
Organization: allenai
Home Page: https://allenai.github.io/dolma/
data-processing large-language-models llm machile-learning nlp

alttch / rapidtables
data-processing,Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Organization: alttch
python python3 text-formatting data-processing library dictionary-data

apache / incubator-wayang
data-processing,Apache Wayang(incubating) is the first cross-platform data processing system.
Organization: apache
Home Page: https://wayang.incubator.apache.org/
cross-platform data-processing apache big-data data-management-platform distributed-system hadoop java jdbc middleware open-source performance scala spark

asyml / forte
data-processing,Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
machine-learning natural-language-processing deep-learning python text-data data-processing information-retrieval natural-language pipeline

asyml / texar
data-processing,Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert

asyml / texar-pytorch
data-processing,Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Organization: asyml
Home Page: https://asyml.io
machine-learning natural-language-processing pytorch deep-learning text-generation python machine-translation dialog-systems texar bert

benibela / xidel
data-processing,Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
User: benibela
Home Page: http://www.videlibri.de/xidel.html
xquery xml html json xpath cli command-line http web rest

bytewax / bytewax
data-processing,Python Stream Processing
Organization: bytewax
Home Page: https://docs.bytewax.io/
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data

chenghaomou / text-dedup
data-processing,All-in-one text de-duplication
User: chenghaomou
text-processing de-duplication nlp data-processing

colasgael / machine-learning-for-solar-energy-prediction
data-processing,Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
User: colasgael
machine-learning neural-network data-processing python matlab tensorflow

constellation-rs / amadeus
data-processing,Harmonious distributed data analysis in Rust.
Organization: constellation-rs
Home Page: https://constellation.rs/amadeus
data-processing data-analysis stream-processing parallel-computing distributed-computing rust

dashbitco / broadway
data-processing,Concurrent and multi-stage data ingestion and data processing with Elixir
Organization: dashbitco
Home Page: https://elixir-broadway.org
elixir genstage data-ingestion data-processing concurrent broadway

flow-php / etl
data-processing,PHP - ETL (Extract Transform Load) data processing library
Organization: flow-php
Home Page: https://flow-php.com
etl data-engineering flow-php data-processing

googlecloudplatform / data-science-on-gcp
data-processing,Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Organization: googlecloudplatform
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering

googlecloudplatform / dataflowjavasdk
data-processing,Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Organization: googlecloudplatform
Home Page: http://cloud.google.com/dataflow
google-cloud-dataflow data-science data-analysis data-mining big-data data-processing

helmholtz-analytics / heat
data-processing,Distributed tensors and Machine Learning framework with GPU and MPI acceleration in Python
Organization: helmholtz-analytics
Home Page: https://heat.readthedocs.io/
gpu tensors distributed machine-learning mpi numpy python pytorch array-api data-analytics

hstreamdb / hstream
data-processing,HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Organization: hstreamdb
Home Page: https://hstream.io/
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems

hxz393 / brutalityextractor
data-processing,适用于高性能系统的多进程解压缩软件(A multiprocess decompression software for high-performance system)
User: hxz393
brute-force computational-efficiency data-processing decompression extractor high-performance optimization parallel-computing parallel-processing performance-optimization

infoslack / awesome-kafka
data-processing,A list about Apache Kafka
User: infoslack
apache-kafka apache-spark data-pipeline data-processing infrastructure kafka kafka-streams stream-processing streaming-data

itechart / convtools-ita
data-processing,convtools is a python library to declaratively define conversions for processing collections, doing complex aggregations and joins.
Organization: itechart
Home Page: https://convtools.readthedocs.io
python code-generation transformations conversions data-preprocessing data-processing data-preparation functional-programming

johnkerl / miller
data-processing,Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
User: johnkerl
Home Page: https://miller.readthedocs.io
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction data-regression statistics statistical-analysis devops devops-tools tabular-data command-line command-line-tools unix-toolkit miller

keithorange / patternpy
data-processing,📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
User: keithorange
algorithmic-trading data-processing financial-analysis investment numpy open-source pandas pattern-recognition python stock-market

kousun12 / eternal
data-processing,👾~ music, eternal ~ 👾
User: kousun12
Home Page: https://eternal.rob.computer
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing

librecat / catmandu
data-processing,Catmandu - a data processing toolkit
Organization: librecat
Home Page: https://librecat.org
cli code4lib csv data-processing data-wrangling datamining ead json lido mab marc21 mods oai-pmh parser perl rdf xml xsd yaml z39-50

lithops-cloud / lithops
data-processing,A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Organization: lithops-cloud
Home Page: http://lithops.cloud
python big-data data-processing multicloud big-data-analytics serverless distributed object-storage parallel kubernetes multiprocessing cloud-computing serverless-computing serverless-functions

markus-wa / cq
data-processing,Clojure Query: A Command-line Data Processor for JSON, YAML, EDN, XML and more
User: markus-wa
clojure yaml json edn csv command-line cli msgpack hacktoberfest xml

matousc89 / padasip
data-processing,Python Adaptive Signal Processing
User: matousc89
adaptive-filtering signal-processing data-processing data-analysis machine-learning

maykulkarni / machine-learning-notebooks
data-processing,Machine Learning notebooks for refreshing concepts.
User: maykulkarni
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models

mech-lang / mech
data-processing,🦾 Main repository for the Mech programming language. Start here!
Organization: mech-lang
Home Page: http://mech-lang.org
ide programming-environment programming-language compiler data-processing reactive-programming live-programming language robotics

microsoft / dialogpt
data-processing,Large-scale pretraining for dialogue
Organization: microsoft
dialogue machine-learning pytorch transformer text-generation dialogpt gpt-2 text-data data-processing

microsoft / godel
data-processing,Large-scale pretrained models for goal-directed dialog
Organization: microsoft
Home Page: http://aka.ms/GODEL
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation

ml6team / fondant
data-processing,Production-ready data processing made easy and shareable
Organization: ml6team
Home Page: https://fondant.ai/en/stable/
data-processing fine-tuning foundation-models machine-learning pipeline python

msamogh / nonechucks
data-processing,Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
User: msamogh
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch

numaproj / numaflow
data-processing,Kubernetes-native platform to run massively parallel data/streaming jobs
Organization: numaproj
Home Page: https://numaflow.numaproj.io
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest

nvidia / dali
data-processing,A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
Organization: nvidia
Home Page: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu gpu-tensorflow audio-processing pytorch mxnet paddle

onceupon / bash-oneliner
data-processing,A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
User: onceupon
Home Page: http://onceupon.github.io/Bash-Oneliner/
oneliner-commands bash data-processing linux-administration terminal linux variables grep xargs system

pathwaycom / pathway
data-processing,Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Organization: pathwaycom
Home Page: https://pathway.com
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing

polyaxon / haupt
data-processing,Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
Organization: polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib

python-bonobo / bonobo
data-processing,Extract Transform Load for Python 3.5+
Organization: python-bonobo
Home Page: https://www.bonobo-project.org/
data-processing extract-transform-load bonobo python3 automation parallelization

pytlab / vaspy
data-processing,Manipulating VASP files with Python.
User: pytlab
Home Page: https://pypi.python.org/pypi/vaspy/
vasp data-processing data-visualization chemistry catalysis-informatics computational-chemistry

remotesensinginfo / rsgislib
data-processing,Remote Sensing and GIS Software Library; python module tools for processing spatial data.
Organization: remotesensinginfo
Home Page: https://www.rsgislib.org
remote sensing gis spatial data earth observation classification machine-learning remote-sensing earth-observation rsgislib python data-analysis data-processing data-science numpy spatial-analysis spatial-data spatial-data-analysis

scramjetorg / scramjet
data-processing,Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.
Organization: scramjetorg
Home Page: https://www.scramjet.org
data-stream event-stream transformations reactive-programming javascript data-processing edge-computing python raspberry-pi data-space

sebkrantz / collapse
data-processing,Advanced and Fast Data Transformation in R
User: sebkrantz
Home Page: https://sebkrantz.github.io/collapse/
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted

senbox-org / snap-engine
data-processing,ESA Earth Observation Toolbox and Java Development Platform
Organization: senbox-org
Home Page: http://step.esa.int
data-processing data-visualization earth-observation eo raster-data remote-sensing linux macos windows

streamnative / pulsar-flink
data-processing,Elastic data processing with Apache Pulsar and Apache Flink
Organization: streamnative
apache-pulsar apache-flink data-processing batch-processing stream-processing flink flink-connector schema-registry schema sql

svenkreiss / pysparkling
data-processing,A pure Python implementation of Apache Spark's RDD and DStream interfaces.
User: svenkreiss
Home Page: https://pysparkling.readthedocs.io
apache-spark data-processing data-science python

tomwright / dasel
data-processing,Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
User: tomwright
Home Page: https://daseldocs.tomwright.me
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools

unionai-oss / pandera
data-processing,A light-weight, flexible, and expressive statistical data testing library
Organization: unionai-oss
Home Page: https://www.union.ai/pandera
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check testing-tools assertions data-assertions data-verification dataframe-schema hypothesis-testing pandas-validator data-processing

yord / pxi
data-processing,🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
User: yord
pxi pixie json csv ssv tsv dsv data-processing deserializer parser