Data Engineer

Technical skills: Azure Databricks with Unity Catalog • Expertise in Azure Databricks: Extensive experience with Azure Databricks for data engineering, data science, and analytics solutions. • Unity Catalog: Proficiency in using Unity Catalog for comprehensive data governance, access control, and lineage tracking across multiple environments. Advanced Python Skills • Advanced proficiency in Python for data engineering tasks, including designing and implementing ETL/ELT pipelines and developing reusable, modular code for efficient data transformations. • Familiarity with Python libraries like Pandas, NumPy, PySpark, SQL Expertise • Strong SQL skills to design complex queries and optimize database performance. • Experience working with Azure databases like Azure SQL Database, Synapse Analytics, and Data Lake for querying, processing, and managing large datasets. IDMC Data Integration • Mapping design in Cloud Data Integration (CDI) — Source, Target, Expression, Aggregator, Joiner, Lookup, Filter, Router, Sorter, Rank transformations · Mapping Task and Task Flow configuration and scheduling · Parameterization (in/out parameters, parameter files) for reusable, environment-agnostic pipelines · Working with Mapplets and reusable transformations · Runtime environments — Secure Agent setup, agent groups, and troubleshooting agent connectivity Git and CI/CD Expertise • Version Control: Advanced experience with Git for managing code repositories, versioning, and collaborating in multi-developer environments. • CI/CD Pipelines: Skilled in designing and implementing CI/CD pipelines using tools like Azure DevOps, GitHub Actions. Cloud Data Architecture • In-depth knowledge of Azure Cloud architecture, including services like Azure Data Lake and Blob Storage. • Hands-on experience in designing and deploying scalable, cloud-native data pipelines for large datasets. Technical Proficiencies · Languages: Python, SQL, · Tools & Platforms: Azure Databricks, Unity Catalog, Data Lake, Azure SQL Database, Git, Azure DevOps, · Version Control: Git · CI/CD Tools: Azure DevOps · ETL: IDMC Data Integration Experience Summary: • 5+ years of hands-on experience in delivering end-to-end data solutions using Azure technologies. • Expertise in Azure Databricks with Unity Catalog for large-scale data processing and governance. • Advanced skills in Python and SQL for efficient data transformation and analysis. • Strong experience CI/CD pipelines, and working in Git-based environments for efficient deployment. • Driving best practices in code quality, data architecture, and DevOps.