Distributed Storage
- HDFS
A selection of projects from my GitHub profile, focused on computer vision, machine learning, data analysis and practical web development.
Machine learning-based pose-to-cloth prediction using neural networks, PCA, ReLU and Adam for fast, interactive garment deformation.
A collection of my projects, experiences, and side quests as I continue learning, building, and exploring new ideas.
A health-focused project built with HTML.
Computer vision projects and experiments in Jupyter Notebook.
Data analytics and visualization of Formula 1 race results and championship standings.
A software engineering project for an electronics store inventory management system, with inventory management, reporting, a dashboard, authentication and security.
Distributed systems · pipelines · analytics
Building distributed systems, processing large-scale datasets, and engineering reliable data pipelines.
Pipeline lifecycle
Technology map
Hands-on work
Batch analytics and distributed messaging, with each system's data flow represented separately.
Project 01
An end-to-end batch analytics pipeline that processes ride-hailing trip data across multiple cities.
Airflow orchestrates the ingestion, processing, and output stages.
SeaTunnel handles ingestion and cleaning before the data is stored in HDFS. Hadoop MapReduce processes the stored trip data to produce city-level analytics, including average speed and average fare per kilometre. Apache Airflow coordinates the batch workflow, and the resulting analytics are routed into categorized output datasets.
Project 02
A distributed food-ordering data pipeline across four virtual machines, demonstrating messaging, batch processing, broker replication, and fault tolerance.
Kafka messaging and Spark batch processing are separate paths. Spark reads local JSONL files; it does not consume Kafka.
The Kafka deployment uses KRaft and two brokers with replicated topics. Producers and consumers run across four virtual machines connected through ZeroTier. Separately, Apache Spark batch-processes local JSONL datasets to enrich order records and generate analytics; Spark does not consume Kafka. Broker failure and recovery scenarios demonstrate replication and fault tolerance.