Build software better, together

nerevu / riko

1.5k

A Python stream processing engine modeled after Yahoo! Pipes

stream-processing etl parallelism rss data asynchronous library cli functional-programming featured

Python Updated Feb 5, 2019 3 issues need help

thbar / kiba

1.2k

Data processing & ETL framework for Ruby

etl etl-ruby ruby data rubydatascience

Ruby Updated Apr 9, 2019

mara / data-integration

1.2k

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

etl data-integration python postgresql pipeline data

Python Updated Apr 15, 2019

compose / transporter

1k

Sync data between persistence engines, like ETL only not stodgy

go etl mongodb elasticsearch rethinkdb postgresql rabbitmq

Go Updated Feb 1, 2019 12 issues need help

blockchain-etl / ethereum-etl

533

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, t…

ethereum csv export exporter erc20 erc20-tokens etl transaction sql aws gcp bigquery google-cloud erc721

Python Updated Apr 20, 2019 2 issues need help

koopjs / koop

429

🔮 Transform, query, and download geospatial data on the web.

gis server nodejs geojson arcgis spatial api etl data-management

Shell Updated Apr 16, 2019

thenaturalist / awesome-business-intelligence

414

Actively curated list of awesome BI tools. PRs welcome!

business-intelligence etl data-visualization data-analysis data-science database sql awesome-list

Updated Apr 18, 2019

react-csv / react-csv

359

React components to build CSV files on the fly basing on Array/literal object of data

csv-document reactjs etl excel reporting

JavaScript Updated Apr 2, 2019 4 issues need help

seanharr11 / etlalchemy

346

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

etl-framework etl python database migrations sqlalchemy

Python Updated Jan 8, 2019

singer-io / getting-started

315

This repository is a getting started guide to Singer.

etl etl-framework python singer data-analysis

Makefile Updated Apr 11, 2019

Cascading / cascading

295

Cascading is a feature rich API for defining and executing complex and fault tolerant data processing workflows on va…

cascading hadoop flink tez scalding machine-learning data-engineering etl

Forked from cwensel/cascading Java Updated Nov 29, 2018

2ndQuadrant / pglogical

290

Logical Replication extension for PostgreSQL 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slo…

postgresql replication logical-decoding database-replication subscription publish-subscribe data-transformation data-transport etl cdc zero-downtime

Updated Mar 14, 2019 1 issue needs help

dotnetcore / SmartCode

243

SmartCode = IDataSource -> IBuildTask -> IOutput => Build Everything!!!

code-generator etl dotnet-core dotnet dotnetcore smartcode

C# Updated Apr 18, 2019

deeplearning4j / DataVec

242

ETL Library for Machine Learning - data pipelines, data munging and wrangling

etl spark machine-learning transformations svmlight hadoop-ecosystem writables schema pipeline formatter datapipeline data-munging

Java Updated May 21, 2018

datacleaner / DataCleaner

237

The premier open source Data Quality solution

data dataquality database desktop datacleaner mdm etl data-analysis data-science profiling

Java Updated Apr 18, 2019

ropensci / elastic

204

R client for the Elasticsearch HTTP API

elasticsearch http database database-wrapper json rstats r r-package data-science etl

R Updated Apr 11, 2019

datamade / data-making-guidelines

202

📘 Making Data, the DataMade Way

etl makefile datamade principles

HTML Updated Mar 27, 2019

appbaseio / abc

202

Power of appbase.io via CLI, with nifty imports from your favorite data sources

elasticsearch etl appbase cli

Go Updated Apr 16, 2019

Cinchoo / ChoETL

157

ETL Framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value formatted files)

csv parser writer reader flat xml json keyvalue etl cinchoo-etl etl-framework csharp dotnet

C# Updated Apr 19, 2019

jike-engineering / mongo-es

148

A MongoDB to Elasticsearch connector

mongodb elasticsearch connector etl

TypeScript Updated Feb 19, 2019

ICIJ / extract

145

A cross-platform command line tool for parallelised content extraction and analysis.

tika etl index ediscovery solr

Java Updated Apr 19, 2019

Nextdoor / bender

142

Bender - Serverless ETL Framework

aws-lambda etl-framework aws-kinesis aws-firehose aws-s3 etl java

Java Updated Jul 2, 2018

Swirrl / grafter

141

Linked Data & RDF Manufacturing Tools in Clojure

clojure etl linked-data rdf semantic-web data grafter

Clojure Updated Apr 15, 2019

airflow-plugins / Example-Airflow-DAGs

137

Example DAGs using hooks and operators from Airflow Plugins

apache-airflow dag etl airflow-plugins selenium google-analytics facebook-ads sftp marketo salesforce hubspot marketo-sdk imap airflow mongodb mailgun singer dbt

Python Updated Jul 24, 2018

JumpMind / metl

137

Metl is a simple, web-based integration platform that allows for several different styles of data integration includi…

integration etl messaging java web vaadin

Java Updated Apr 19, 2019

AlexIoannides / pyspark-example-project

137

Example project implementing best practices for PySpark ETL jobs and applications.

pyspark etl-job python data-engineering spark data-science etl etl-pipeline

Python Updated Feb 19, 2019

ceumicrodata / mETL

132

mito ETL tool

python data-integration etl etl-framework pipeline

Python Updated Jan 20, 2017

BitwiseInc / Hydrograph

121

A visual ETL development and debugging tool for big data

etl etl-framework big-data cascading apache-spark

Java Updated Feb 25, 2019

51zero / eel-sdk

121

Big Data Toolkit for the JVM

parquet orc hive kudu kafka big-data scala etl hadoop

Scala Updated Feb 11, 2019 8 issues need help

uber / storagetapper

109

StorageTapper is a scalable realtime MySQL change data streaming and transformation service

mysql kafka avro cdc etl

Go Updated Jan 7, 2019

etl

Repositories 664