What I've built

Projects

A selection of projects from my GitHub profile, focused on computer vision, machine learning, data analysis and practical web development.

Exploring AI, Computer Vision and Web Development
Contributing to open source projects
Turning curiosity into projects and ideas into reality

Project Gallery

View GitHub Repos

Distributed systems · pipelines · analytics

Data Engineering & Big Data

Building distributed systems, processing large-scale datasets, and engineering reliable data pipelines.

Pipeline lifecycle

  1. Ingestion
  2. Processing
  3. Storage
  4. Analytics

Technology map

Big data toolkit

Distributed Storage

  • HDFS

Data Processing

  • Apache Hadoop
  • MapReduce
  • Apache Spark

Data Streaming

  • Apache Kafka
  • KRaft

Data Ingestion

  • Apache SeaTunnel

Workflow Orchestration

  • Apache Airflow

Programming & Infrastructure

  • Python
  • SQL
  • Linux
  • Docker
  • Virtual Machines

Hands-on work

Featured Projects

Batch analytics and distributed messaging, with each system's data flow represented separately.

Project 01

Rides Analytics Pipeline

An end-to-end batch analytics pipeline that processes ride-hailing trip data across multiple cities.

  • Apache Airflow
  • Apache Hadoop
  • MapReduce
  • Apache SeaTunnel
  • HDFS
  • Python

Key features

  • Ingests and cleans source data with SeaTunnel.
  • Stores datasets in HDFS for distributed processing.
  • Runs city-level analytics with Hadoop MapReduce.
  • Calculates average speed and average fare per kilometre.
  • Uses Airflow to orchestrate the workflow.
  • Routes processed analytics into categorized output datasets.

Architecture

  1. Trip data
  2. SeaTunnel
  3. HDFS
  4. Hadoop MapReduce
  5. City analytics

Airflow orchestrates the ingestion, processing, and output stages.

Technical implementation

SeaTunnel handles ingestion and cleaning before the data is stored in HDFS. Hadoop MapReduce processes the stored trip data to produce city-level analytics, including average speed and average fare per kilometre. Apache Airflow coordinates the batch workflow, and the resulting analytics are routed into categorized output datasets.

Project 02

Distributed Food Ordering System

A distributed food-ordering data pipeline across four virtual machines, demonstrating messaging, batch processing, broker replication, and fault tolerance.

  • Apache Kafka 3.9.2
  • Apache Spark 3.3.4
  • Python 3.10
  • KRaft
  • ZeroTier

Key features

  • Runs a distributed Kafka architecture with KRaft.
  • Uses two brokers with replicated Kafka topics.
  • Distributes producers and consumers across four VMs.
  • Batch-processes local JSONL datasets with Spark.
  • Generates enriched order data and analytics.
  • Demonstrates broker failure, recovery, and replication.

Architecture

  1. Distributed producers
  2. Kafka · KRaft · 2 brokers
  3. Distributed consumers

Separate Spark batch path

  1. Local JSONL files
  2. Spark batch processing
  3. Enriched orders & analytics

Kafka messaging and Spark batch processing are separate paths. Spark reads local JSONL files; it does not consume Kafka.

Technical implementation

The Kafka deployment uses KRaft and two brokers with replicated topics. Producers and consumers run across four virtual machines connected through ZeroTier. Separately, Apache Spark batch-processes local JSONL datasets to enrich order records and generate analytics; Spark does not consume Kafka. Broker failure and recovery scenarios demonstrate replication and fault tolerance.