Senior Data Engineer

Key Responsibilities

• Design, develop, and optimize data solutions using Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake.
• Develop and maintain data pipelines following the Medallion Architecture (Bronze, Silver, Gold) to deliver scalable and trusted data products.
• Implement ingestion frameworks for structured and semi-structured data from APIs, JSON files, databases, and other enterprise sources.
• Manage and optimize data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and associated Azure services.
• Build, schedule, and support workloads using Databricks Workflows/Jobs, with experience in Delta Live Tables (DLT) or Lakeflow considered a strong advantage.
• Implement schema management practices, including schema enforcement and schema evolution.
• Develop and maintain data quality controls, validation frameworks, reconciliation processes, and automated testing.
• Design and implement data transformation, cleansing, and standardization processes to support business reporting and analytics.
• Establish metadata management, data lineage, and governance capabilities using Unity Catalog and related technologies.
• Develop API-based ingestion and export integrations for upstream and downstream systems.
• Support solutions involving document metadata management and references to PDF and eLabel assets.
• Drive performance tuning, partitioning strategies, and platform optimization to improve scalability and efficiency.
• Implement CI/CD practices for notebooks, code deployments, and automated testing throughout the development lifecycle.
• Develop monitoring, alerting, error-handling, and reprocessing capabilities to ensure operational stability and reliability.
• Support regulated data environments with a strong focus on traceability, auditability, reproducibility, and controlled release management.

Desired Skills and Experience Required Qualifications Strong hands-on experience with Azure Databricks, Python/PySpark, Spark SQL, and Delta Lake. Deep understanding of modern data lakehouse architectures, including the Bronze-Silver-Gold (Medallion) design pattern. Experience developing enterprise-scale ETL/ELT pipelines and data integration solutions. Strong knowledge of Azure storage technologies, particularly ADLS Gen2. Experience implementing data governance, lineage, and access control frameworks. Proven experience with data quality, validation, monitoring, and operational support. Experience working in regulated environments where compliance, auditability, and data traceability are key requirements. Preferred Qualifications Experience with Delta Live Tables (DLT), Lakeflow, and advanced Databricks orchestration capabilities. Familiarity with document-centric data solutions, including PDF and eLabel metadata management. Experience implementing DevOps and CI/CD best practices within data development teams. Experience in pharmaceutical, healthcare, life sciences, or other highly regulated industries.