Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Releasing a new version
enterprise-script-service
The enterprise script service (aka ESS) is a thin Ruby API layer that spawns a process, the enterprise_script_engine, to execute an untrusted Ruby script.
The enterprise_script_engine executable ingests the input from stdin as a msgpack encoded payload; then spawns an mruby-engine; uses seccomp to sandbox itself; feeds library, input and finally the Ruby scripts into the engine; returns the output as a msgpack encoded payload to stdout and finally exits.
Data format
Input
The input is expected to be a msgpack MAP with three keys (Symbol): library, sources, input:
-
library: a msgpackBINset of MRuby instructions that will be fed directly to themruby-engine -
input: a msgpack formated payload for thesourcesto digest -
sources: a msgpackARRAYofARRAYwith two elements each (tuples):path,source; the actual code to be executed by the mruby-engine
Output
The output is msgpack encoded as well; it is streamed to the consuming end though. Streamed items can be of different types.
Each element streamed is in the format of an ARRAY of two elements, where the first is a Symbol describing the element type:
-
measurement: a msgpackARRAYof two elements: aSymboldescribing the measurement, and anINT64with the value in µs. -
output: a msgpackMAPwith two entries (keys are symbols):-
extractedwith whatever the script put in@output, msgpack encoded; and -
stdoutwith aSTRINGcontaining whatever the script printed to "stdout".
-
-
stat: aMAPkeyed with symbols mapping to theirINT64values
Errors
When the ESS fails to serve a request, it communicates the error back to the caller by returning a non-zero status code.
It can also report data about the error, in certain cases, over the pipe. In does so in returning a tuple, as an ARRAY with the type being the symbol error and the payload being a MAP. The content of the map will vary, but it always will have a __type symbol key that defines the other keys.
Build
Run ./bin/rake to build the project. This effectively runs the spec target, which builds all libraries, the ESS and native tests; then runs all tests (native and Ruby).
To rebuild the entire project (which is useful when switching from one OS to another), use ./bin/rake mrproper.
Using it
The sample script bin/sandbox reads Ruby input from a file or stdin, executes it, and displays the results.
You can invoke ESS from your own Ruby code as follows:
result = EnterpriseScriptService.run(
input: {result: [26803196617, 0.475]}, # (1)
sources: [
["stdout", "@stdout_buffer = 'hello'"],
["foo", "@output = @input[:result]"], # (2)
],
instructions: nil, # (3)
timeout: 10.0, # (4)
instruction_quota: 100000, # (5)
instruction_quota_start: 1, # (6)
memory_quota: 8 << 20 # (7)
)
expect(result.success?).to be(true)
expect(result.output).to eq([26803196617, 0.475])
expect(result.stdout).to eq("hello")-
invokes the ESS, with a map as the
input(available as@inputin the sources) -
two "scripts" to be executed, one sets the
@stdout_bufferto a value, the second returns the value associated with the key:resultof the map passed in in <1> -
some raw instructions that will be fed directly into MRuby; defaults to nil
-
a 10 second time quota to spawn, init, inject, eval and finally output the result back; defaults to 1 second
-
a 100k instruction limit that that the engine will execute; defaults to 100k
-
starts counting the instructions at index 1 of the
sourcesarray -
creates an 8 megabyte memory pool in which the script will run
Where are things?
C++ sources
Consists of our code base, plus seccomp and msgpack libraries, as well as the mruby stuff. All in ext/enterprise_script_service
Note: lib seccomp is omitted on Darwin.
Ruby layer
Ruby code is in lib/
Tests
-
googletest tests are in
tests/, which also includes the Google Test library. -
RSpec tests are in
spec/
Other useful things
-
There is a
CMakeLists.txtthat’s mainly there for CLion support; we don’t use cmake to build any of this. -
You can use vagrant to bootstrap a VM to test under Linux while on Darwin; this is useful when testing
seccomp.
Vagrant
$ vagrant up
$ vagrant ssh
vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ bundle install
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ git submodule init
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ git submodule update
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ bin/rake