Data Engineer

We are building a modern Databricks-based lakehouse from the ground up and are now looking for a data engineer to support the ingestion and stabilization phase of the platform. The focus of this role is to help operationalize, debug, and harden data pipelines so they are production-ready, reliable, and well-monitored. You will work in an established engineering setup with infrastructure already provisioned, CI/CD in place, and clear data engineering patterns defined. You will work closely with a senior data engineer who designs the ingestion patterns and prototypes each data source. Your role is to implement, debug, test, and stabilize the pipelines following these established patterns, with ongoing guidance and code reviews. Scope of assignment Build and maintain ingestion pipelines (Bronze layer) from internal and external sources (BigQuery, SharePoint, Dynamics 365, files in cloud storage, APIs) Build standardized Delta tables from raw data (Silver layer) Debug, stabilize, and redeploy existing pipelines following established patterns Implement basic data quality checks and tests (row counts, schema validation, null checks, duplicates) using pytest and SQL Monitor and troubleshoot Databricks pipelines and jobs Assist with operational monitoring and cost/usage dashboards in Databricks Work with Git-based workflows and deploy changes across different environments Assist in migrating existing data transformation notebooks from legacy platforms (such as Classic Databricks and Azure Synapse) into the new Databricks lakehouse. This includes adapting and refactoring the code to follow clearly defined patterns, standards, and best practices, with guidance and support. Technical requirements Hands-on experience with Databricks: Delta tables Notebooks Jobs & Workflows SQL endpoints and Spark clusters Databricks Dashboards Unity Catalog Databricks Asset Bundles Programming languages: PySpark SQL Familiar with Git and pull-request–based workflows Comfortable working with existing infrastructure and patterns Basic understanding of cloud concepts (storage accounts, permissions, service principals) Experience of AI-assisted development tools for increased productivity is a plus. Databricks Associate Data Engineer certification is a plus