Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
rxtx
rxtx is a queue based data collector > data transmitter. Useful for online/offline data collection, back pressure buffering or general queuing. rxtx uses bbolt maintained by CoreOs, a single file database for storing messages before they can be sent.
rtbeat was developed to consume rxtx POST data and publish as events into elasticsearch, logstash, kafka, redis or directly to log files.
Test on MacOs
Install with brew
brew tap txn2/homebrew-tap
brew install rxtxHelp
rxtx -hTest Docker Container
Help
docker run --rm -it txn2/rxtx -hon arm 6/7 based device:
docker run --rm -it txn2/rxtx:arm32v6-1.2.0 -hTest Source
Help
go run ./rxtx.go -h
Usage of rxtx:
-batch int
Batch size. (default 5000)
-ingest string
Ingest server. (default "http://localhost:8081/in")
-interval int
Seconds between intervals. (default 30)
-maxq int
Max number of message in queue. (default 2000000)
-name string
Service name. (default "rxtx")
-path string
Directory to store database. (default "./")
-port string
Server port. (default "8080")
Start server on 8080
go run ./rxtx.go Add message to queue
The rxtx services accepts HTTP POST data to an API endpoint in the following form /rx/PRODUCER/KEY/LABEL/.../. One label is required, however as many labels as necessary may be added, separated by a forward slash.
curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/dataAdd message to queue every second
while true; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; sleep 1; doneAdd 1000 messages to the queue.
time for i in {1..1000}; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; doneExample Batch
{
"uuid":"d7642975-4241-4f43-b704-b67621b184b8",
"size":4,
"messages":[
{
"seq":"2019072100000000038",
"time":"2019-07-21T02:44:49.560087Z",
"uuid":"a93cfe42-79c5-4a32-9dc0-6b77b68d0926",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"6955"
}
},
{
"seq":"2019072100000000039",
"time":"2019-07-21T02:44:50.6214579Z",
"uuid":"66042da0-b0c1-447c-ab0b-66c887e8b56a",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"8923"
}
},
{
"seq":"2019072100000000040",
"time":"2019-07-21T02:44:51.622978Z",
"uuid":"0ec3f6d7-905f-42bc-896e-515bf8f2a06d",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"28324"
}
},
{
"seq":"2019072100000000046",
"time":"2019-07-21T02:44:57.8990519Z",
"uuid":"bfa2d06d-a8f4-4125-93bc-d08a070079d7",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"31460"
}
}
]
}Profile
go build ./rxtx.go && time ./rxtx --path=./data/ --cpuprofile=rxtxcpu.prof --memprofile=rxtxmem.profBrowser-based profile viewer:
go tool pprof -http=:8081 rxtxcpu.profBuilding and Releasing
rxtx uses GORELEASER to build binaries and Docker containers.
Test Release Steps
Install GORELEASER with brew (MacOS):
brew install goreleaser/tap/goreleaserBuild without releasing:
goreleaser --skip-publish --rm-dist --skip-validateRelease Steps
- Commit latest changes
- Tag a version
git tag -a v2.0 -m "Version 2.0" - Push tag
git push origin v2.0 - Run:
GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist
