Download as PDF, PPTX
Uploaded byWeaveworks
Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service
This document describes Cortex, a multi-tenant horizontally scalable Prometheus as a service. It retrieves metrics from applications using Prometheus scrapers, distributes the metrics across ingesters using consistent hashing, stores metrics in DynamoDB with indexes and chunks in S3, and provides a Prometheus compatible query API. The goal is to build a proof of concept quickly to monitor tens of thousands of users sending tens of millions of samples per second in a cost effective and scalable way, reusing Prometheus where possible. There is still work to be done on features like recording rules, alerting, reliability, performance, and code cleanup before it is production ready.
Weave Cortex: Multi-tenant, horizontally scalable Prometheus as a Service
- 1. Multi-tenant, horizontally scalable Prometheusas a Service @tom_wilkie November 2016 weavecortex https://github.com/weaveworks/cortex
- 5. Prometheus and Kubernetes:A Perfect Match Prometheus and Kubernetes: Deploying https://www.weave.works/blog/ Prometheus and Kubernetes: Monitoring Your Applications Prometheus and Kubernetes: Monitoring Your Infrastructure
- 6.
- 7. requirements: 1. API compatiblewith Prometheus 2. easy to operate and manage 3. tens of thousands of users, tens of millions samples/s 4. cost effective to run 5. reuse as much of Prometheus as possible … so we can sell it
- 8. Aim: build proofof concept as quickly as possible 16/06 started design doc 26/07 launch jobs 25/08 give talk at PromCon! … make it robust … 09/11 talk at KubeCon http://goo.gl/prdUYV
- 9. Retriever scraping your jobs Your DC WeaveCloud Frontend, Authenticator Distributor Ingester Distributor… IngesterIngester DynamoDB S3
- 10. Is a vanillaOSS Prometheus. Does service discovery, scraping and relabelling. Configured to send samples to Weave Cloud: remote_write: url: https://cloud.weave.works/api/prom/push basic_auth: password: <redacted> PRs up stream for generic write path: #1930 #1957 #1987 Retriever
- 11.
- 12. • Uses consistenthashing to assign timeseries to Ingesters • Input to hash is (user ID, metric name) • Tokens stored in Consul • Also currently handles queries Distributor http://goo.gl/U9u1U2
- 13.
- 14. • Heavily modifiedMemorySeriesStorage • Use same chunk format as Prometheus • Keeps everything in memory (for up to an hour) • Also stores in memory inverted index for queries • Flushes chunks to S3 and indexes them in DynamoDB Ingester
- 15.
- 16. External inverted indexmaintained in DynamoDB, chunks stored in S3 Item in DynamoDB looks like: { hash key: “{user ID}:{metric name}:{hour}”, range key: “{label name}:{label value}:{chunk ID}”, metric: ..., from, through: ..., ID: ..., } DynamoDB S3
- 17.
- 18.
- 21. Why not justrun my own Prometheus?
- 22. Lots left todo… Features: • Recording rules • Alerting & Alertmanager Reliability: • Replication between ingesters, commit log etc • Ingestor lifecycle • Separate query service? Performance: • Query parallelisation • Background chunk coalescing Code: • Code cleanup • Upstream appropriate changes
- 23. Questions? Sign up athttps://cloud.weave.works/ $ kubectl -n kube-system apply -f ‘https://cloud.weave.works/k8s/cortex.yaml?t=...' https://github.com/weaveworks/cortex Try It Out!
- 24. We’re hiring! London BerlinSanFrancisco