Factorial plan
A JSONL row defines one controlled workload.
Security mode, broker count, message size, acknowledgement mode and throughput target are explicit, creating a reproducible experiment identity.
Back to selected work
Case study 03 · Performance research
KafkaSecurity
A dissertation-scale experiment built to measure the performance cost of Kafka security with repeatable evidence.
- Role
- Researcher & engineer
- Stack
- Kafka, AWS EC2, Terraform, Bash, Python
- Type
- MEng dissertation artefact
- Scale
- 5,184 matched producer runs
01 / Problem
Security has a cost. Measure it.
Teams often discuss the performance overhead of TLS and mutual TLS in broad terms. My dissertation needed a controlled way to compare that overhead across Kafka security modes and workload configurations.
The challenge was as much operational as statistical: long-running experiments had to survive interruptions, preserve raw evidence and remain comparable across three- and five-broker environments.
Research question
Under controlled workloads, how much throughput and latency does transport security add to Apache Kafka?
02 / Architecture
Infrastructure and experiment as one system.
Terraform provisions EC2 brokers and a benchmark client. Deployment scripts render plaintext, TLS or mTLS configurations. JSONL plans drive resumable producer and consumer campaigns, while telemetry and raw logs flow into analysis-ready exports.
01Terraformprovision
→
02EC2 clusterdeploy
→
03JSONL plansexecute
→
04Result packsanalyse
03 / Walkthrough
Follow one row from plan to evidence.
Terraform / Bash
The matching cluster mode is deployed.
Provisioning and deployment scripts configure the Kafka nodes, certificates, listeners and benchmark client before readiness is checked.
Resumable runner
Checkpoints make long campaigns recoverable.
Started, completed and failure records preserve progress. A failed attempt can be investigated and rerun without discarding completed work.
Python analysis
Matched runs expose security overhead.
Raw benchmark output and telemetry become structured metrics, paired effects, bootstrap intervals, CSV tables and dissertation-ready plots.
04 / Decisions
Reproducibility was a design constraint.
EC2 over managed Kafka
Direct broker access was necessary to control listener configuration, certificates and security-mode comparability.
Plans over manual commands
Machine-readable factorial rows make the campaign inspectable and prevent parameters from drifting between modes.
Checkpoint every run
Long cloud campaigns fail. Resumable execution protects both time and spend while retaining failure history.
Raw evidence stays public
The repository contains the result evidence, not only the scripts and headline claims, so the analysis can be challenged.
05 / Results
Thousands of runs. Matched comparisons.
5,184valid matched producer runs
72/72five-broker consumer validation rows completed
3plaintext, TLS and mutual TLS modes
- Mean producer throughput penalties measured 10.8% for TLS and 9.6% for mTLS across matched runs.
- The system exports raw logs, structured JSON, CSV, LaTeX tables and SVG plots.
- Paired effects, bootstrap intervals and factor sensitivity keep analysis tied to comparable configurations.
06 / Reflection
The experiment is also a platform.
The most valuable engineering lesson was that a credible benchmark needs operational recovery, traceable configuration and careful claims. Next, I would package the analysis as a browsable results explorer and automate cost-aware environment teardown.
- Publish an interactive filterable result explorer.
- Add automated infrastructure cost reporting per campaign.
- Extend validation across broker versions and alternative instance families.