About
Statement of Purpose
Irys · Huvc4M…Amr8 ↗
Rendered from protocol.json
Experiment protocol
AgentRules
Autonomy
- NoAskHumanToContinue: Yes
- ExperimentTimeoutNotes: If snappy_benchmark exceeds the 60s training budget, kill it and mark the trial as discard. Compilation is excluded from the budget but bounded by the 600s hard timeout.
- LogRedirectExample: ./build/snappybenchmark --benchmarkfilter=^BMUFlat/0/1$ --benchmarkmintime=2s --benchmarkformat=json > .autoresearch/results/run.log 2>&1
- SimplicityCriterion: Yes
Environment
AssetPrep
- testdata/ is vendored in-tree; no external download is required.
- thirdparty/benchmark and thirdparty/googletest are git submodules and must be initialized once during setup.
Constraints
- NetworkPolicy: offline
- NoNewDependencies: Yes
OsHints
- linux-x86_64
- darwin-arm64
PackageManagers
- cmake
- make
SetupCommands
- git submodule update --init --recursive
- cmake -S . -B build -DCMAKEBUILDTYPE=Release -DSNAPPYBUILDBENCHMARKS=ON -DSNAPPYBUILDTESTS=ON
- cmake --build build --config Release -j
Execution
- Command: ./build/snappybenchmark --benchmarkfilter=^BMUFlat/0/1$ --benchmarkmintime=2s --benchmarkformat=json
- Cwd: .
Determinism
- Notes: The benchmark replays a fixed corpus from testdata/. Google Benchmark times each invocation independently; multiple runs typically vary <2% on a quiet host.
- SeedPolicy: optional
- HardTimeoutSeconds: 600
StopCondition
- ExcludeCompilationFromBudget: Yes
- Notes: One run of snappybenchmark filtered to BMUFlat/0 is one experiment trial. CMake build is treated as setup and is not counted toward the budget.
- TrainingSecondsBudget: 60
- Type: wall_clock
ImmutableHarness
Paths
- snappy_unittest.cc
- snappy_benchmark.cc
- snappytesttool.cc
- snappytestdata.cc
- snappytestdata.h
- snappycompressfuzzer.cc
- snappyuncompressfuzzer.cc
- testdata/
- format_description.txt
- framing_format.txt
- third_party/benchmark/
- third_party/googletest/
- CMakeLists.txt
- cmake/
- .github/workflows/
- Rationale: Benchmark driver, unit tests, fuzzers, golden corpus, bitstream format spec, build configuration, and CI define the truth for both performance and correctness. Editing them would either invalidate the measurement, change the comparison surface, or break wire-format stability promised by the README.
Measurement
BaselinePolicy
- BaselineNotes: Baseline is established on the operator's host with the as-shipped snappy.cc at commit 27ab5f7 using testdata/ from the same checkout.
- EstablishOnHardware: Yes
- SameDataSnapshot: Yes
PrimaryMetric
- Direction: maximize
Extract
- ExampleStdout:
{
"context": {"date": "2026-05-12T12:00:00+00:00", "library_version": "v1.8.3"},
"benchmarks": [
{
"name": "BM_UFlat/0",
"iterations": 50000,
"real_time": 12345.0,
"cpu_time": 12345.0,
"time_unit": "ns",
"bytes_per_second": 524288000.0
}
]
}- Kind: regex
- Notes: snappybenchmark filters by regex; ^BMUFlat/0/1$ pins the run to a single benchmark instance (file index 0 = html, level 1). The regex captures the single bytespersecond value emitted by --benchmark_format=json.
- Pattern: "bytespersecond"\s:\s([0-9eE+\-.]+)
- Name: bytespersecondBMUFlat01_html
SecondaryMetrics
- - Direction: maximize
Extract
- Kind: regex
- Notes: Captured by a separate invocation with --benchmarkfilter=BMZFlat/0 --benchmark_format=json; tracks compression-side regressions.
- Pattern: "bytespersecond"\s:\s([0-9eE+\-.]+)
- Name: bytespersecondBMZFlat_0
- - Direction: maximize
Extract
- Kind: custom
- Notes: Run ./build/snappy_unittest; emit 1 on exit 0, 0 otherwise. Acts as a correctness gate — a trial with this at 0 must be discarded regardless of throughput.
- Name: snappyunittestpass
Meta
- Archetype: benchmark_opt
- CreatedAt: 2026-05-12T00:00:00Z
- Eligibility: eligible
- ProtocolBundleId: google-snappy-benchmark-opt-v1
- PurposeStatement: Maximize Snappy decompression throughput (bytes/sec) on BMUFlat/0 from the bundled snappybenchmark suite without breaking the Snappy bitstream format, unit tests, or fuzzers.
Repo
- CloneUrl: https://github.com/google/snappy
- DefaultBranch: main
- Name: snappy
- Owner: google
- UpdatedAt: 2026-05-12T00:00:00Z
MutableSurface
AllowedGlobs
- snappy.cc
- snappy.h
- snappy-internal.h
- snappy-sinksource.cc
- snappy-sinksource.h
- snappy-stubs-internal.cc
- snappy-stubs-internal.h
- snappy-c.cc
- snappy-c.h
AllowedKinds
- code_edit
ForbiddenGlobs
- snappy_unittest.cc
- snappytesttool.cc
- snappytestdata.cc
- snappytestdata.h
- snappy_benchmark.cc
- snappycompressfuzzer.cc
- snappyuncompressfuzzer.cc
- testdata/**
- format_description.txt
- framing_format.txt
- third_party/**
- .github/**
- CMakeLists.txt
- cmake/**
- BUILD.bazel
- MODULE.bazel
- WORKSPACE
- snappy-stubs-public.h.in
ProtocolVersion: 1.0
Provenance
- BaselineArtifactPath: .autoresearch/results/baseline_run.log
GitWorkflow
- BranchPattern: autoresearch/trial-*
- CommitScope: snappy
- StagingExample: git add snappy.cc snappy-internal.h snappy-sinksource.cc
ResultsLog
Columns
- trial_id
- commit
- metric_name
- metric_value
- wall_seconds
- status
- Format: jsonl
- Path: .autoresearch/results/trials.jsonl
Safety
- CrashStatus: crash
- OomPolicy: fail
SchemaKind: protocol
protocol.json (raw)
{
"schemaKind": "protocol",
"protocolVersion": "1.0",
"meta": {
"archetype": "benchmark_opt",
"eligibility": "eligible",
"repo": {
"owner": "google",
"name": "snappy",
"defaultBranch": "main",
"cloneUrl": "https://github.com/google/snappy"
},
"purposeStatement": "Maximize Snappy decompression throughput (bytes/sec) on BM_UFlat/0 from the bundled snappy_benchmark suite without breaking the Snappy bitstream format, unit tests, or fuzzers.",
"createdAt": "2026-05-12T00:00:00Z",
"updatedAt": "2026-05-12T00:00:00Z",
"protocolBundleId": "google-snappy-benchmark-opt-v1"
},
"environment": {
"osHints": [
"linux-x86_64",
"darwin-arm64"
],
"packageManagers": [
"cmake",
"make"
],
"setupCommands": [
"git submodule update --init --recursive",
"cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSNAPPY_BUILD_BENCHMARKS=ON -DSNAPPY_BUILD_TESTS=ON",
"cmake --build build --config Release -j"
],
"assetPrep": [
"testdata/ is vendored in-tree; no external download is required.",
"third_party/benchmark and third_party/googletest are git submodules and must be initialized once during setup."
],
"constraints": {
"noNewDependencies": true,
"networkPolicy": "offline"
}
},
"mutableSurface": {
"allowedGlobs": [
"snappy.cc",
"snappy.h",
"snappy-internal.h",
"snappy-sinksource.cc",
"snappy-sinksource.h",
"snappy-stubs-internal.cc",
"snappy-stubs-internal.h",
"snappy-c.cc",
"snappy-c.h"
],
"forbiddenGlobs": [
"snappy_unittest.cc",
"snappy_test_tool.cc",
"snappy_test_data.cc",
"snappy_test_data.h",
"snappy_benchmark.cc",
"snappy_compress_fuzzer.cc",
"snappy_uncompress_fuzzer.cc",
"testdata/**",
"format_description.txt",
"framing_format.txt",
"third_party/**",
".github/**",
"CMakeLists.txt",
"cmake/**",
"BUILD.bazel",
"MODULE.bazel",
"WORKSPACE",
"snappy-stubs-public.h.in"
],
"allowedKinds": [
"code_edit"
]
},
"immutableHarness": {
"paths": [
"snappy_unittest.cc",
"snappy_benchmark.cc",
"snappy_test_tool.cc",
"snappy_test_data.cc",
"snappy_test_data.h",
"snappy_compress_fuzzer.cc",
"snappy_uncompress_fuzzer.cc",
"testdata/",
"format_description.txt",
"framing_format.txt",
"third_party/benchmark/",
"third_party/googletest/",
"CMakeLists.txt",
"cmake/",
".github/workflows/"
],
"rationale": "Benchmark driver, unit tests, fuzzers, golden corpus, bitstream format spec, build configuration, and CI define the truth for both performance and correctness. Editing them would either invalidate the measurement, change the comparison surface, or break wire-format stability promised by the README."
},
"execution": {
"command": "./build/snappy_benchmark --benchmark_filter=^BM_UFlat/0/1$ --benchmark_min_time=2s --benchmark_format=json",
"cwd": ".",
"stopCondition": {
"type": "wall_clock",
"trainingSecondsBudget": 60,
"excludeCompilationFromBudget": true,
"notes": "One run of snappy_benchmark filtered to BM_UFlat/0 is one experiment trial. CMake build is treated as setup and is not counted toward the budget."
},
"hardTimeoutSeconds": 600,
"determinism": {
"seedPolicy": "optional",
"notes": "The benchmark replays a fixed corpus from testdata/. Google Benchmark times each invocation independently; multiple runs typically vary <2% on a quiet host."
}
},
"measurement": {
"primaryMetric": {
"name": "bytes_per_second_BM_UFlat_0_1_html",
"direction": "maximize",
"extract": {
"kind": "regex",
"pattern": "\"bytes_per_second\"\\s*:\\s*([0-9eE+\\-.]+)",
"exampleStdout": "{\n \"context\": {\"date\": \"2026-05-12T12:00:00+00:00\", \"library_version\": \"v1.8.3\"},\n \"benchmarks\": [\n {\n \"name\": \"BM_UFlat/0\",\n \"iterations\": 50000,\n \"real_time\": 12345.0,\n \"cpu_time\": 12345.0,\n \"time_unit\": \"ns\",\n \"bytes_per_second\": 524288000.0\n }\n ]\n}",
"notes": "snappy_benchmark filters by regex; ^BM_UFlat/0/1$ pins the run to a single benchmark instance (file index 0 = html, level 1). The regex captures the single bytes_per_second value emitted by --benchmark_format=json."
}
},
"secondaryMetrics": [
{
"name": "bytes_per_second_BM_ZFlat_0",
"direction": "maximize",
"extract": {
"kind": "regex",
"pattern": "\"bytes_per_second\"\\s*:\\s*([0-9eE+\\-.]+)",
"notes": "Captured by a separate invocation with --benchmark_filter=BM_ZFlat/0 --benchmark_format=json; tracks compression-side regressions."
}
},
{
"name": "snappy_unittest_pass",
"direction": "maximize",
"extract": {
"kind": "custom",
"notes": "Run ./build/snappy_unittest; emit 1 on exit 0, 0 otherwise. Acts as a correctness gate — a trial with this at 0 must be discarded regardless of throughput."
}
}
],
"baselinePolicy": {
"establishOnHardware": true,
"sameDataSnapshot": true,
"baselineNotes": "Baseline is established on the operator's host with the as-shipped snappy.cc at commit 27ab5f7 using testdata/ from the same checkout."
}
},
"provenance": {
"resultsLog": {
"format": "jsonl",
"path": ".autoresearch/results/trials.jsonl",
"columns": [
"trial_id",
"commit",
"metric_name",
"metric_value",
"wall_seconds",
"status"
]
},
"gitWorkflow": {
"branchPattern": "autoresearch/trial-*",
"commitScope": "snappy",
"stagingExample": "git add snappy.cc snappy-internal.h snappy-sinksource.cc"
},
"baselineArtifactPath": ".autoresearch/results/baseline_run.log"
},
"safety": {
"oomPolicy": "fail",
"crashStatus": "crash"
},
"agentRules": {
"simplicityCriterion": true,
"autonomy": {
"noAskHumanToContinue": true
},
"experimentTimeoutNotes": "If snappy_benchmark exceeds the 60s training budget, kill it and mark the trial as discard. Compilation is excluded from the budget but bounded by the 600s hard timeout.",
"logRedirectExample": "./build/snappy_benchmark --benchmark_filter=^BM_UFlat/0/1$ --benchmark_min_time=2s --benchmark_format=json > .autoresearch/results/run.log 2>&1"
}
}