Data Engineer (Azure Databricks & AI) | WealthLab
Overview
We're hiring a Data Engineer to join a greenfield Wealth Management platform, helping build the next generation of AI-enabled products and data solutions from the ground up.
This role is ideal for a hands-on engineer with deep Azure and Databricks expertise, strong data engineering fundamentals, and exposure to modern AI technologies. You'll work closely with Product, Business Analysis, Data, Architecture, and AI teams to deliver scalable data platforms, intelligent workflows, and AI-powered solutions.
Key Responsibilities
- Design, develop, and maintain enterprise-scale data pipelines using Azure Databricks, Python, and PySpark.
- Build and optimise ETL/ELT frameworks, data workflows, and processing solutions.
- Develop AI-ready data foundations to support GenAI, RAG, and intelligent automation initiatives.
- Build, deploy, and support Databricks jobs, workflows, and data products.
- Work with structured and unstructured data across complex enterprise environments.
- Partner with Data Scientists, AI Engineers, Product teams, and Business Analysts to deliver business-focused solutions.
- Support data governance, quality, lineage, and security requirements using tools such as Unity Catalog.
- Contribute to the design and delivery of Databricks AI capabilities, Agents, Models, Experiments, and AI workflows.
Essential Requirements
Data Engineering
- Strong hands-on experience with Azure Databricks.
- Advanced Python and PySpark development skills.
- Experience building and maintaining enterprise ETL/ELT pipelines.
- Strong understanding of data management, data profiling, and data architecture.
- Experience building, scheduling, and supporting data pipelines and jobs.
- Ability to troubleshoot, optimise, and support production data solutions.
AI & Databricks
- Exposure to AI/ML initiatives within Databricks.
- Experience building or supporting RAG (Retrieval-Augmented Generation) pipelines.
- Experience working with Databricks AI capabilities, including Agents, Models, Experiments, Playground, or similar tooling.
- Understanding of how AI solutions work under the hood, not just how to consume them.
Industry Experience
- Banking, Wealth Management, or Financial Services experience highly preferred.
- Experience operating within large enterprise environments.
Nice to Have
- LangChain or LangGraph experience.
- AI Agent or Agentic AI development experience.
- OpenShift, Kubernetes, or containerisation technologies.
- FastAPI, Angular, or web application development experience.
- Experience building AI-enabled workflows and intelligent automation solutions.