Senior Data Engineer - SMALS
Description
Within the DEP (Data Exchange Platform) programme, SMALS is building an on-premise data platform for the Belgian Ministry of Defence. The platform must meet strict security, governance, and data sovereignty requirements. You will join the Sierra Team alongside a Data Architect, a Solution Architect, and a BI developer. The assignment is full-time with a hybrid working arrangement in Brussels.
You will handle heterogeneous data ingestion, incremental and streaming pipelines, and transformations across the Bronze, Silver, and Gold layers. You will work on data historisation practices, Delta Lake exports, ACID transactions, and time travel. You will also manage orchestration, monitoring, error handling, and automated alerts. Pipeline reliability, idempotence, and traceability are integral parts of the role.
The position also covers governance, lineage, and data exposure for BI use cases. You will push technical and business metadata, maintain end-to-end lineage, and contribute to DUA, SLA, and related governance documentation. You will develop endpoints to expose the Gold layer and collaborate with reporting teams. The environment includes Python, SQL, dbt, DuckDB, Delta Lake, Dagster, and other modern open-source technologies.
Top Reasons to Apply
Strategic defence project
Modern data stack
Security-focused impact
Strong learning opportunities
Experienced data team
Detailed Responsibilities and Skills
Additional Responsibilities
- Implement data historisation
- Develop Delta exports
- Manage Delta tables
- Maintain Delta tables
- Ensure transactional consistency
- Implement orchestration assets
- Maintain orchestration assets
- Monitor pipeline runs
- Manage pipeline errors
- Automate pipeline alerts
- Push technical metadata
- Push business metadata
- Maintain end-to-end lineage
- Draft governance documents
- Develop Gold endpoints
- Collaborate reporting teams
- Feed BI dashboards
General skills
- Rigour and autonomy in managing complex multi-domain pipelines
- Ability to document technical decisions
- Ability to explain technical topics to non-technical stakeholders
- Team spirit
- Curiosity and solution-oriented mindset
- Belgian nationality required
- French: native level
- Dutch: active knowledge
- English: active knowledge
- Awareness of data security in restricted-access environments
Technical skills
- Mastery of Python
- Mastery of SQL
- Confirmed experience with dbt-core: models, snapshots, macros, tests, and profiles
- Good knowledge of DuckDB as an embedded analytical query engine
- Experience with Delta Lake: ACID transactions, time travel, OPTIMIZE/VACUUM
- Knowledge of MSSQL and PostgreSQL, including advanced SQL and CDC
- Experience with a data orchestrator: Dagster, Airflow, or equivalent
- Comfortable with Linux/OpenShift and PowerShell in a Windows development environment
- Familiarity with governance tools: DataHub, OpenMetadata, or equivalent
- Experience in BI development
- Confirmed experience in ETL/ELT data acquisition
- Confirmed data engineering experience
- Confirmed Data Governance experience
- Confirmed Metadata Management experience
- Confirmed experience with Python, Pandas, and Apache Spark from a Data Engineer perspective
- Confirmed SQL level
- Kafka/Debezium experience for real-time data capture
- Knowledge of Spark: Spark SQL, Thrift Server, and Beeline
- Experience with Power BI Report Server (PBIRS) or Apache Superset
- Awareness of data security in restricted-access environments
- Knowledge of Lakehouse concepts, Medallion architecture, DWH, and Datalake
Tools
- dbt-core
- DuckDB
- Delta Lake
- Dagster
- Airflow
- Linux
- OpenShift
- PowerShell
- DataHub
- OpenMetadata
- MSSQL
- PostgreSQL
- Kafka
- Debezium
- Spark
- Spark SQL
- Thrift Server
- Beeline
- Power BI Report Server
- Apache Superset
- Pandas