summary_*.jsonThe output is the product
A load test that only prints to a terminal tells you how things went while you were watching. Every SMTPBench run writes this instead.
{
"run_uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"config": {
"threads": 5, "messages": 100, "rate": null,
"tls_mode": "starttls", "auth": false, "port": 587, "offline": false
},
"totals": { "sent": 495, "failed": 5, "retried": 2, "success_rate": 99.0 },
"latency_ms": { "p50": 120, "p95": 350, "p99": 510, "max": 820 },
"per_mx": { "mx1.example.com": { "sent": 495, "failed": 5 } }
}Three things to know before reading one: sent plus failed is messages attempted, while retried counts retry attempts and is not part of that sum. latency_ms covers successful sends only and is null when nothing succeeded. And config.auth is a boolean — the summary records that authentication happened, never who authenticated.
Because the percentiles land in a file, a mail path can be gated in CI like any other service:
# fail the build if p95 exceeds 2s
smtpbench [email protected] lb_host=smtp.internal port=587 \
threads=5 messages=20 rate=5
python -c "import json,glob,os,sys; \
f=max(glob.glob('logs/summary_*.json'), key=os.path.getmtime); \
d=json.load(open(f)); \
sys.exit(1 if (d['latency_ms'] or {}).get('p95',0) > 2000 else 0)"