Job description
Job Title: Databricks Data Engineer
Location: New York, NY – Hybrid
Role Description: As part of our Data Engineering team, you will be joining a tight knit team of motivated technology professionals where ongoing learning and development are central to our ethos and internal culture. We are seeking a Databricks Data Engineer to design, build, and optimize data pipelines and models on the Databricks Lakehouse Platform. This is a hands-on engineering role where you will develop secure, scalable solutions that deliver high-quality, analytics-ready data to power business insights and AI initiatives.
Key Responsibilities:
• Build and maintain ETL/ELT pipelines to ingest, transform, and deliver data using Databricks, Delta Lake, and PySpark.
• Design and implement efficient, scalable data models within the Lakehouse architecture.
• Write performant Spark SQL and Python code optimized for distributed compute environments.
• Develop and manage Databricks workflows, jobs, and notebooks for end-to-end data processing.
• Implement robust data quality, governance, and monitoring frameworks leveraging tools like Unity Catalog and Delta Live Tables.
• Collaborate with data engineers, analysts, and business stakeholders to deliver clean, curated datasets and analytical models.
• Follow engineering best practices including version control (Git), code reviews, and CI/CD integration.
• Troubleshoot data pipeline issues, performance bottlenecks, and optimize cluster configurations.
• Stay current with Databricks platform enhancements, applying new features to improve data delivery, performance, and reliability.
Key Requirements & Technical Experience:
• 2+ years of experience as a Data Engineer with hands-on expertise in Databricks.
• Strong knowledge of Spark, PySpark, SQL, and Delta Lake.
• Experience designing and orchestrating data pipelines using tools such as Airflow, Azure
Data Factory, dbt, or Databricks Workflows.
• Familiarity with at least one cloud platform (AWS, Azure, or GCP) and associated services
• Proficiency in Python for data engineering tasks and automation.
• Understanding of data governance, security, and access control best practices (Unity
Catalog experience a plus).
• Strong collaboration skills and ability to operate within agile, cross-functional teams