GitHub Actions / Terraform
Infrastructure begins as reviewed code.
The deployment workflow applies a repeatable AWS environment instead of relying on resources assembled manually in the console.
Back to selected work
Case study 02 · Cloud data pipeline
CommitScope
A cloud pipeline that turns repository history into traceable datasets and engineering dashboards.
- Role
- Cloud engineer & developer
- Stack
- AWS, Terraform, Python, Athena, QuickSight
- Type
- Independent cloud project
- Status
- Deployed and evidenced
01 / Problem
Cloud ability should look like architecture.
A local repository-analysis script can calculate useful metrics, but it does not prove orchestration, infrastructure or data-platform thinking. I wanted every stage—from source repository to dashboard—to be visible and reproducible.
That meant separating raw evidence from transformed outputs, tracking each execution, and making the pipeline queryable rather than ending with an opaque report.
Design question
How can repository analysis become a repeatable cloud data product rather than a one-off script?
02 / Architecture
One traceable path from Git to insight.
GitHub Actions and Terraform define the environment. Step Functions coordinates the run, ECS Fargate performs analysis, and S3 preserves raw, processed and curated layers. Glue, Athena and QuickSight form the query and presentation layer.
03 / Walkthrough
Follow a repository through the pipeline.
Step Functions / ECS
The workload runs in a container with room to grow.
Step Functions creates a visible, retryable orchestration path. ECS Fargate handles analysis workloads that can outgrow a small function runtime.
S3 / Glue
Every execution keeps its identity.
Raw, processed and curated data stay separate. Explicit execution IDs prevent dashboards from silently mixing unrelated runs.
Athena / QuickSight
Metrics become questions engineers can ask.
Athena queries code-health tables and QuickSight presents repository trends, class hotspots and method-level signals.
04 / Decisions
Designed for traceability, not a service checklist.
Fargate over Lambda
Repository analysis can be heavier and longer-running than a small function workload. Containers give the analyser predictable packaging and runtime space.
Orchestration over one script
Step Functions makes stages, retries and failure points visible instead of burying the full workflow inside one process.
Three S3 layers
Raw evidence remains intact while processed and curated outputs can evolve for query and dashboard consumers.
Execution-scoped dashboards
Explicit IDs give each run provenance and keep “latest” reporting from blending data across separate executions.
05 / Evidence
The proof lives in the outputs.
7parser-backed languages plus fallback summaries
3separate S3 data layers
10+code-health and repository metrics
06 / Reflection
What I would build next.
The strongest next step is to make cost and operability as visible as the data flow: add explicit budgets, service-level telemetry and a recruiter-friendly recorded run from repository URL to dashboard.
- Publish an automated cost estimate and teardown policy.
- Add operational dashboards and pipeline-duration alerts.
- Record a concise end-to-end demo using a public repository.