Microsoft Machine Learning for Apache Spark

Linkis helps easily connect to various back-end computation/storage engines(Spark, Python, TiDB...), exposes various interfaces(REST, JDBC, Java ...), with multi-tenancy, high performance, and resource control.

State of the Art Natural Language Processing

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

A curated list of awesome Apache Spark packages and resources.

🚚 Agile Data Science Workflows made easy with Pyspark

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Jupyter magics and kernels for working with remote Spark clusters

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

Example project implementing best practices for PySpark ETL jobs and applications.

This is a repo documenting the best practices in PySpark.

80+ DevOps & Data CLI Tools - AWS, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, Ambari, Blueprints, CloudFormation, Elasticsearch, Solr, Pig, IPython - Python / Jython Tools

A boilerplate for writing PySpark Jobs

Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks

t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark

MorphL Community Edition uses big data and machine learning to predict user behaviors in digital products and services with the end goal of increasing KPIs (click-through rates, conversion rates, etc.) through personalization

Big Data Processing Framework - Unified Data API or SQL on Any Storage

Apache Spark (PySpark) Practice on Real Data

Apache Spark Connector for Azure Cosmos DB

pyspark methods to enhance developer productivity 📣 👯 🎉

Updated repository

Isolation Forest on Spark

HandySpark - bringing pandas-like capabilities to Spark dataframes

LearningApacheSpark

BERT, AWS RDS, AWS Forecast, EMR Spark Cluster, Hive, Serverless, Google Assistant + Raspberry Pi, Infrared, Google Cloud Platform Natural Language, Anomaly detection, Tensorflow, Mathematics

Process Common Crawl data with Python and Spark

Fundamentals of Spark with Python (using PySpark), code examples

Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

Relation Extraction using Deep learning(CNN)