A modern data workflow platform
-
Updated
Jul 30, 2020 - Python
#
data-engineering
Here are 468 public repositories matching this topic...
d3
yaml
gist
chart
json
data-science
data
csv
big-data
dashboard
data-driven
data-visualization
data-engineering
business-intelligence
github-gist
d3js
just-dashboard
-
Updated
Jul 29, 2020 - JavaScript
A list of useful resources to learn Data Engineering from scratch
-
Updated
Jul 29, 2020
search
nlp
data-science
machine-learning
reinforcement-learning
production
data-engineering
recsys
recommendation
papers
applied-data-science
applied-machine-learning
-
Updated
Jul 30, 2020
Quilt is a versioned data portal for S3
-
Updated
Jul 30, 2020 - Jupyter Notebook
Pandas on AWS
mysql
python
emr
aws
data-science
lambda
aws-lambda
athena
etl
pandas
data-engineering
redshift
apache-parquet
amazon-athena
apache-arrow
aws-glue
glue-catalog
amazon-sagemaker-notebook
-
Updated
Jul 30, 2020 - Python
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-science
machine-learning
data-visualization
data-engineering
cloud-computing
data-analysis
data-processing
data-pipeline
-
Updated
Jun 30, 2020 - Jupyter Notebook
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
python
airflow
spark
apache-spark
scheduler
s3
data-engineering
data-lake
warehouse
redshift
data-migration
livy
etl-framework
apache-airflow
emr-cluster
etl-pipeline
etl-job
data-engineering-pipeline
airflow-dag
goodreads-data-pipeline
-
Updated
Mar 9, 2020 - Python
Clean APIs for data cleaning. Python implementation of R package Janitor
-
Updated
Jul 30, 2020 - Python
Example project implementing best practices for PySpark ETL jobs and applications.
-
Updated
Jul 9, 2020 - Python
blog
learning
computer-science
education
aws
engineering
unix
data-science
algorithm
research
university
course-materials
math
mathematics
data-engineering
waterloo
educational
software-engineering
-
Updated
Jul 30, 2020 - CSS
Cascading is a feature rich API for defining and executing complex and fault tolerant data processing workflows on various cluster computing platforms. Please see https://github.com/cwensel/cascading for access to all WIP branches.
-
Updated
Nov 29, 2018 - Java
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
infrastructure
aws
postgres
data
airflow
cloudformation
cassandra
cluster
aws-s3
aws-sdk
data-warehouse
data-engineering
data-lake
aws-ec2
postgresql-database
data-modeling
cassandra-database
etl-pipeline
data-engineering-pipeline
airflow-operators
-
Updated
Mar 5, 2020 - Python
elasticsearch
data-engineering
locality-sensitive-hashing
elasticsearch-plugin
knn
similarity-search
knn-algorithm
-
Updated
Apr 20, 2020 - Python
Open Metadata and Governance
-
Updated
Jul 30, 2020 - Java
Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift
-
Updated
Jul 30, 2020 - TypeScript
A daily digest of the articles or videos I've found interesting, that I want to share with you.
javascript
java
kubernetes
microservices
scala
kafka
akka
spark
reactjs
ddd
architecture
technology
watch
bigdata
data-engineering
category-theory
googlecloudplatform
-
Updated
Jul 29, 2020
An Awesome List of Open-Source Data Engineering Projects
-
Updated
May 11, 2020
A package to easily open an instance of a Google spreadsheet and interact with worksheets through Pandas DataFrames.
python
data-science
data
google
google-sheets
pandas
data-engineering
gspread
data-analytics
dataframes
google-spreadsheets
sheets
-
Updated
Jun 18, 2020 - Python
An automatic ML model optimization tool.
data-science
machine-learning
deep-learning
data-engineering
neural-networks
hyperparameter-optimization
hyperparameter-tuning
hpo
automl
automated-machine-learning
-
Updated
Jul 28, 2020 - Python
The Accelerator is a tool for fast and reproducible processing of large amounts of data.
data-science
data-mining
big-data
data-engineering
high-performance-computing
reproducibility
parallel-processing
-
Updated
Jun 24, 2020 - Python
Study materials for the Google Cloud Professional Data Engineering Exam
-
Updated
Mar 25, 2019
Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive
-
Updated
Jun 30, 2020 - Scala
Projects done in the Data Engineering Nanodegree by Udacity.com
aws
postgres
cassandra
udacity-course
data-engineering
data-modeling
udacity-nanodegree
dimensional-model
-
Updated
Aug 7, 2019 - Jupyter Notebook
Interactive computing for complex data processing, modeling and analysis in Python 3
-
Updated
Feb 24, 2020 - Python
Tool to build production-ready pipelines for experimentation with Kedro and MLflow
-
Updated
Jun 22, 2020 - Python
Ansible playbook to deploy distributed technologies
aws
ansible
devops
kafka
ansible-playbooks
zookeeper
infrastructure-management
data-engineering
ec2-instance
-
Updated
Nov 20, 2017 - Python
The DataHelix generator allows you to quickly create data, based on a JSON profile that defines fields and the relationships between them, for the purpose of testing and validation
-
Updated
Jul 30, 2020 - Java
Documentation for data enthusiasts
documentation
data-science
machine-learning
cloud
big-data
deep-learning
knowledge
wiki
best-practices
ebook
collaboration
data-engineering
moocs
-
Updated
Jul 17, 2020 - JavaScript
Improve this page
Add a description, image, and links to the data-engineering topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-engineering topic, visit your repo's landing page and select "manage topics."