Databricks Architecture AI

Key Responsibilities

• Design and implement Databricks architecture on Azure and/or GCP (workspace, clusters, networking, security).
• Lead platform setup, governance, and optimization (Unity Catalog, cost control, access management).
• Build and deploy AI/ML solutions using Databricks (MLflow, Feature Store, Model Serving).
• Develop GenAI / LLM solutions (RAG pipelines, embeddings, vector search).
• Integrate LLMs (Azure OpenAI / Vertex AI / other APIs) within Databricks.
• Deliver scalable Lakehouse architecture solutions.
• Collaborate with engineering and business teams for end-to-end AI use cases.
• Ensure performance tuning, reliability, and security of workloads.

Required Skills

• Strong experience in Databricks setup & architecture
• Hands-on with Databricks AI stack:
• MLflow, AutoML
• Feature Store
• Model Serving
• RAG / vector search / embeddings
• Experience with PySpark, Spark optimization, Delta Lake.
• Strong exposure to cloud platforms:
• Azure Databricks + Azure services
• GCP Databricks + GCP ecosystem (BigQuery, GCS, Vertex AI)
• Experience integrating LLMs / GenAI solutions.
• Knowledge of Unity Catalog, governance, and security.

Preferred Skills

• Experience in GenAI applications on Databricks
• Familiarity with LangChain / vector databases
• Exposure to real-time/streaming pipelines
• CI/CD and DevOps practices