Back
Year
2025
Tech Stack
Python, Dash, Plotly, MQTT, pandas, Requests
Description
An end-to-end real-time data pipeline and dashboard for the
Australian National Electricity Market. Six integrated components handle data
collection, MQTT streaming, error recovery, and visualisation of live power
generation and emissions data from facilities over 100MW, combined with historical
emissions records for longitudinal analysis.
Why This Project Matters
Energy and emissions data can be siloed and delayed, making it
difficult to analyse current conditions alongside historical trends. This project
provides a real-time five-minute pipeline from API data collection through MQTT
streaming and dashboard visualisation, with error recovery incorporated into the
continuous operation workflow.
Technical Highlights
- Implemented 5-minute interval ingestion from the OpenElectricity API
- Built a six-component architecture covering the pipeline, integration, MQTT publisher, continuous runner, dashboard, and historical preprocessor
- Implemented change-aware MQTT publishing to reduce unnecessary network traffic
- Built a continuous runner with automatic error recovery and graceful shutdown
Key Features
- Live geospatial and temporal dashboard using Dash and Plotly
- Real-time MQTT data streaming
- Historical emissions comparison
- Continuous autonomous operation
- Integration of live and historical emissions data
My Role
- 🧩 Worked as the sole developer across all six system components
- 🔌 Built the OpenElectricity API data pipeline for live power-generation and emissions data
- 📡 Built the change-aware MQTT publisher for real-time data streaming
- 🔁 Built the continuous runner with error recovery and graceful shutdown
- 📊 Built the live Dash/Plotly dashboard and integrated the historical emissions data workflow