PostgreSQL → Airflow → GCS → dbt → BigQuery → Metabase. Corridor-weighted synthetic transactions seed the warehouse; dbt models turn raw events into FX corridor margins analysts can query. Loads are idempotent, so a re-run is never a duplicate.
An independent side project running on a single AWS EC2 box: RSS and API monitors poll for market movement, an LLM classifies and drafts, systemd keeps the whole thing alive. Detection to published post lands between 10 seconds and a minute, fully automated end-to-end.
A 24-task Airflow DAG ingests telecom usage data and lands it in BigQuery through idempotent MERGE loads. Task-level retries and dependency ordering mean a partial failure resumes rather than restarts.
with DAG(
dag_id="due_fx_elt",
schedule="15 * * * *",
catchup=False,
default_args={"retries": 3, "retry_delay": timedelta(minutes=5)},
) as dag:
extract = PostgresToGCSOperator(
task_id="extract_transactions",
sql=INCREMENTAL_SQL, # bounded by data_interval_start
bucket=RAW_BUCKET,
export_format="parquet",
)
load = GCSToBigQueryOperator(
task_id="load_staging",
write_disposition="WRITE_TRUNCATE",
destination_project_dataset_table=f"{PROJECT}.staging.transactions",
)
merge = BigQueryInsertJobOperator(
task_id="merge_core",
configuration={"query": {"query": MERGE_SQL, "useLegacySql": False}},
)
transform = BashOperator(task_id="dbt_run", bash_command="dbt build --target prod")
extract >> load >> merge >> transform
I came to data engineering from statistics, which is why I care less about the dashboard and more about whether the number behind it can be trusted at 3am. I build pipelines that are boring on purpose: idempotent, observable, and easy for the next person to read.
I’m in Abuja and work with teams anywhere.