Role:
Data Engineer
Location:
Remote (Candidate should be comfortable following PST Time Zone)
Job Type:
Full Time
Active LinkedIn ID is a must and must match with the resume.
Must Have Skills
Azure Databricks, Apache Spark, Snowflake, Data Governance, Unity Catalog, dBT, Microsoft Purview, Kafka, SQL, Python
Experience And Qualifications
Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
10+ years of experience in data engineering / data architecture, with deep hands‑on expertise in Microsoft Azure.
Expert‑level experience with Azure Databricks and Apache Spark (PySpark, Spark SQL).
Delta Lake, Unity Catalog, Job orchestration and performance tuning.
Strong experience with Snowflake on Azure, including:
Schema and data model design
Performance and cost optimization
RBAC, Streams, and Tasks
DBT (Core or Cloud) expertise:
Project structure and best practices
Tests, exposures, macros
Strong fundamentals in SQL and Python.
Proven experience building ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Synapse / Microsoft Fabric pipelines.
Familiarity with streaming platforms such as Event Hubs and/or Kafka.
Solid understanding of Lakehouse and data warehousing architectures, dimensional modeling, medallion architecture patterns, and data quality frameworks.
Strong knowledge of security and governance, including Microsoft Purview (catalog, lineage), data masking and PII handling, managed identities, and private networking.
Excellent communication and documentation skills, with the ability to create architecture and design documents and present to both technical and business stakeholders.
Consulting / agency experience, with the ability to lead multiple concurrent projects.
Willingness to travel as required.
Nice To Have
Experience with Azure ML, feature stores, or serving ML pipelines from Databricks or Snowflake.
Infrastructure as Code (IaC) using Terraform or Bicep.
Containerization experience with Docker.
Observability and data quality tooling, such as Great Expectations / Delta Expectations, Databricks Quality Flows, Monte Carlo, Datadog.
Preferred Certifications
Databricks Certified Data Engineer Professional or Databricks Architect.
Microsoft Azure: DP-203 – Azure Data Engineer, Azure Solutions Architect Expert, Azure Security Engineer Associate.
Snowflake: SnowPro Core, SnowPro Advanced.
Job Description / Responsibilities
Partner with client stakeholders to translate business objectives into scalable Azure data architectures and delivery roadmaps.
Architect and implement lakehouse and data warehouse solutions using Azure Databricks (Delta Lake, Unity Catalog), Databricks Bundle, Snowflake on Azure, dbt (Core / Cloud), ADLS Gen2, and medallion architecture patterns.
Design and build robust ingestion and transformation pipelines using Azure Data Factory (ADF) and Microsoft Fabric / Synapse pipelines.
Orchestrate ELT workflows using Databricks Jobs, Delta Live Tables, and dbt models.
Establish best practices for performance and cost optimization, including cluster sizing and autoscaling, Photon and SQL Warehouses, Snowflake virtual warehouses, caching, file layout optimization, and Z‑ordering.
Drive observability and monitoring using Azure Monitor, Log Analytics, and Databricks metrics.
Implement robust data governance and security using Azure Authentication, Unity Catalog, RBAC / ABAC, managed identities, private endpoints and VNet injection, and Azure Key Vault–backed secrets.
Lead code and design reviews, setting standards for PySpark, SQL, and dbt, unit and integration testing, and data quality checks (expectations and constraints).
Implement CI/CD pipelines using Azure DevOps or GitHub Actions.
Guide multi‑domain programs (Healthcare, Retail, BFSI, or similar).
Mentor data engineers and ensure high‑quality, production‑ready deliverables.
Evangelize solutions through clear documentation, reference architectures, and executive and stakeholder presentations.
#J-18808-Ljbffr
