DataOps

Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.

The open standard for data logging

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors

✨ Open-source tool for data-centric NLP. Argilla helps domain experts and data teams to build better NLP datasets in less time.

Cloud Native DataOps & AIOps Platform | 云原生数智运维平台

Open-source data observability for analytics engineers.

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

Alternative to Parquet. 100x faster for random access, includes a vector index and automatic versioning. Apache Arrow and DuckDB compatible.

Your CLI for ELT+. It's open source, flexible, and scales to your needs. Confidently move, transform, and test your data using tools you know with a data engineering workflow you’ll love.

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Support agile DataOps Based on DataX and Flink-CDC, Chunjun with Web-UI

A list of tools for annotating data, managing annotations, etc.

Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.

🔮 Visibility Across Space and Time – The network telemetry engine for data-driven security investigations.

DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.

Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

📙 Awesome Data Catalogs and Observability Platforms.

Build, run and manage your data pipelines with Python or SQL on any cloud

A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way 🌰