Topic: pyspark Goto Github

👇 Here are 3482 public repositories matching this topic...

aipredict / ai-deployment
pyspark,关注AI模型上线、模型部署
User: aipredict
pmml onnx scikit-learn xgboost lightgbm pyspark spark-ml keras tensorflow pytorch

alexioannides / pyspark-example-project
pyspark,Implementing best practices for PySpark ETL jobs and applications.
User: alexioannides
data-engineering data-science etl etl-job etl-pipeline pyspark python spark

ankurchavda / sparklearning
pyspark,A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.
User: ankurchavda
spark big-data pyspark

apache / incubator-graphar
pyspark,An open source, standard data file format for graph data storage and retrieval.
Organization: apache
Home Page: https://graphar.apache.org/
big-data graph graph-storage data-orchestration etl graph-analysis pyspark spark

apache / linkis
pyspark,Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Organization: apache
Home Page: https://linkis.apache.org/
sql spark hive pyspark livy linkis engine storage resource-manager application-manager

awesome-spark / awesome-spark
pyspark,A curated list of awesome Apache Spark packages and resources.
Organization: awesome-spark
apache-spark pyspark awesome sparkr

awesome-spark / spark-gotchas
pyspark,Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Organization: awesome-spark
apache-spark book guide pyspark

azure / azure-cosmosdb-spark
pyspark,Apache Spark Connector for Azure Cosmos DB
Organization: azure
azure-cosmos-db cosmos-db spark connector jupyter-notebook databricks azure-databricks lambda-architecture changefeed databricks-notebooks pyspark apache-spark

camdavidsonpilon / tdigest
pyspark,t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
User: camdavidsonpilon
python percentile quantile estimate pyspark distributed-computing mapreduce

capitalone / datacompy
pyspark,Pandas, Polars, and Spark DataFrame comparison for humans and more!
Organization: capitalone
Home Page: https://capitalone.github.io/datacompy/
python pandas spark data data-science compare dataframes numpy pyspark fugue

cartershanklin / pyspark-cheatsheet
pyspark,PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
User: cartershanklin
spark apache-spark pyspark big-data

cluster-apps-on-docker / spark-standalone-cluster-on-docker
pyspark,Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. :zap:
Organization: cluster-apps-on-docker
spark docker python scala pyspark jupyter r sparkr

commoncrawl / cc-pyspark
pyspark,Process Common Crawl data with Python and Spark
Organization: commoncrawl
spark warc-files wet commoncrawl sparksql pyspark wat-files common-crawl

databrickslabs / automl-toolkit
pyspark,Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.
Organization: databrickslabs
spark feature-engineering apache-spark pyspark scala machinelearning ml

databrickslabs / dbldatagen
pyspark,Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
Organization: databrickslabs
Home Page: https://databrickslabs.github.io/dbldatagen
datagen pyspark python data-generation faker spark spark-streaming delta-live-tables deltalake databricks

ekampf / pyspark-boilerplate
pyspark,A boilerplate for writing PySpark Jobs
User: ekampf
python apache-spark pyspark boilerplate

ericxiao251 / spark-syntax
pyspark,This is a repo documenting the best practices in PySpark.
User: ericxiao251
Home Page: https://ericxiao251.github.io/spark-syntax/
pyspark best-practices

g-research / spark-extension
pyspark,A library that provides useful extensions to Apache Spark and PySpark.
Organization: g-research
spark pyspark scala java python gr-oss

h2oai / sparkling-water
pyspark,Sparkling Water provides H2O functionality inside Spark cluster
Organization: h2oai
Home Page: https://docs.h2o.ai/sparkling-water/3.3/latest-stable/doc/index.html
h2o spark machine-learning integration pysparkling rsparkling big-data pyspark scala

harisekhon / devops-python-tools
pyspark,80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
User: harisekhon
Home Page: https://www.linkedin.com/in/HariSekhon
cloudformation python hbase json avro parquet spark pyspark travis-ci elasticsearch

hi-primus / optimus
pyspark,:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Organization: hi-primus
Home Page: https://hi-optimus.com
spark pyspark data-wrangling bigdata big-data-cleaning data-science data-cleansing data-cleaner data-transformation machine-learning data-profiling data-extraction data-exploration data-analysis data-preparation cudf dask dask-cudf data-cleaning

huseinzol05 / gather-deployment
pyspark,Gathers Python deployment, infrastructure and practices.
User: huseinzol05
tensorflow airflow docker python docker-compose kafka pyflink pyspark

ibis-project / ibis
pyspark,the portable Python dataframe library
Organization: ibis-project
Home Page: https://ibis-project.org
python impala pandas database clickhouse postgresql sqlite mysql datafusion sql

jadianes / spark-py-notebooks
pyspark,Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
User: jadianes
Home Page: http://jadianes.github.io/spark-py-notebooks
big-data bigdata data-analysis data-science ipython ipython-notebook machine-learning mllib notebook pyspark python spark

jelmerk / hnswlib
pyspark,Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
User: jelmerk
k-nearest-neighbors knn-search java algorithm spark scala pyspark

johnsnowlabs / spark-nlp
pyspark,State of the Art Natural Language Processing
Organization: johnsnowlabs
Home Page: https://sparknlp.org/
nlp natural-language-processing spark pyspark named-entity-recognition sentiment-analysis lemmatizer spell-checker entity-extraction part-of-speech-tagger

josephmachado / efficient_data_processing_spark
pyspark,Code for "Efficient Data Processing in Spark" Course
User: josephmachado
Home Page: https://josephmachado.podia.com/efficient-data-processing-in-spark
apache-spark data-engineering data-pipeline minio pyspark pyspark-notebook

jupyter-incubator / sparkmagic
pyspark,Jupyter magics and kernels for working with remote Spark clusters
Organization: jupyter-incubator
spark kernel cluster livy magic sql-query pandas-dataframe jupyter pyspark kerberos

kevinschaich / pyspark-cheatsheet
pyspark,🐍 Quick reference guide to common patterns & functions in PySpark.
User: kevinschaich
Home Page: https://spark.apache.org/docs/latest/api/python/pyspark.sql.html
pyspark pyspark-tutorial cheatsheet cheat cheatsheets reference references documentation docs data-science

kuwala-io / kuwala
pyspark,Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Organization: kuwala-io
Home Page: https://kuwala.io
data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt

logicalclocks / hopsworks
pyspark,Hopsworks - Data-Intensive AI platform with a Feature Store
Organization: logicalclocks
Home Page: https://hopsworks.ai
feature-store aws azure data-science feature-engineering feature-management gcp governance kserve machine-learning

lyhue1991 / eat_pyspark_in_10_days
pyspark,pyspark🍒🥭 is delicious，just eat it!😋😋
User: lyhue1991
pyspark spark

mahmoudparsian / data-algorithms-book
pyspark, MapReduce, Spark, Java, and Scala for Data Algorithms Book
User: mahmoudparsian
Home Page: http://mapreduce4hackers.com
hadoop-mapreduce java distributed-computing scala mapreduce data-algorithms python machine-learning pyspark distributed-algorithms

mahmoudparsian / data-algorithms-with-spark
pyspark,O'Reilly Book: [Data Algorithms with Spark] by Mahmoud Parsian
User: mahmoudparsian
spark pyspark data algorithms transformations partitioning-algorithms machine-learning design-patterns data-algorithms data-abstractions

mahmoudparsian / pyspark-tutorial
pyspark,PySpark-Tutorial provides basic algorithms using PySpark
User: mahmoudparsian
Home Page: http://mapreduce4hackers.com
big-data big-data-analytics data-algorithms pyspark spark spark-dataframes spark-rdd

microsoft / synapseml
pyspark,Simple and Distributed Machine Learning
Organization: microsoft
Home Page: http://aka.ms/spark
spark pyspark azure scala microsoft ml machine-learning databricks cognitive-services lightgbm

morphl-ai / morphl-community-edition
pyspark,MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization
Organization: morphl-ai
Home Page: https://morphl.io
artificial-intelligence machine-learning user-experience conversion-rate-optimization front-end-development data-driven-design product-development morphl-platform pyspark cassandra

mrpowers / chispa
pyspark,PySpark test helper methods with beautiful error messages
User: mrpowers
Home Page: https://mrpowers.github.io/chispa/
pyspark testing

mrpowers / mack
pyspark,Delta Lake helper methods in PySpark
User: mrpowers
Home Page: https://mrpowers.github.io/mack/
deltalake pyspark

mrpowers / quinn
pyspark,pyspark methods to enhance developer productivity 📣 👯 🎉
User: mrpowers
Home Page: https://mrpowers.github.io/quinn/
pyspark apache-spark

nike-inc / koheesio
pyspark,Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Organization: nike-inc
Home Page: https://engineering.nike.com/koheesio/
data-engineering delta-lake pydantic pyspark python

paypal / gimel
pyspark,Big Data Processing Framework - Unified Data API or SQL on Any Storage
Organization: paypal
Home Page: http://gimel.io
spark spark-streaming big-data gimel paypal data-api kafka cassandra hbase aerospike

ptyadana / sql-data-analysis-and-visualization-projects
pyspark,SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
User: ptyadana
sql mysql mysql-notes exercises mysqlworkbench mysql-database data-analysis postgresql postgres sqlite

quintoandar / butterfree
pyspark,A tool for building feature stores.
Organization: quintoandar
python package data-engineering etl-framework etl feature-store data-science pyspark

runawayhorse001 / learningapachespark
pyspark,LearningApacheSpark
User: runawayhorse001
Home Page: https://runawayhorse001.github.io/LearningApacheSpark/
html latex pyspark python spark tutorial

tirthajyoti / spark-with-python
pyspark,Fundamentals of Spark with Python (using PySpark), code examples
User: tirthajyoti
pyspark spark apache-spark dataframe mlib machine-learning big-data database map-reduce python hdfs analytics hadoop distributed-computing parallel-computing sql apache

titicaca / spark-iforest
pyspark,Isolation Forest on Spark
User: titicaca
iforest-model isolation-forest spark-ml iforest spark anomaly-detection pyspark-mllib pyspark

uber / petastorm
pyspark,Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Organization: uber
tensorflow pytorch deep-learning machine-learning sysml pyspark pyarrow parquet parquet-files

webankfintech / scriptis
pyspark,Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
Organization: webankfintech
hue zeppelin spark hive sql pyspark scala ide hql hive-table

xd-deng / spark-practice
pyspark,Apache Spark (PySpark) Practice on Real Data
User: xd-deng
spark pyspark