Senior Data Engineer (Databricks)

DS STREAM is expanding its AI Data Engineering Team focused on a mature, high-availability data hub built on Azure Databricks using Unity Catalog, Spark Structured Streaming, and Medallion Architecture. The team consists of 50+ senior data consultants experienced in Data Warehouse and Big Data.

• At least 5 years of commercial experience in Data Engineering with focus on large scale data pipelines
• Proven track record using advanced solutions in Azure cloud and Databricks
• Proficiency in Python and SQL
• Ability to build pipelines using Github or similar CI/CD tools

Nice to have:

• Previous exposure to GCP infrastructure and tools (Airflow, Pub/Sub, BigQuery, VertexAI)
• Knowledge of Spark Structured Streaming
• Ability to build enterprise API integrations with FastAPI or similar
• Experience in leading/mentoring engineers

• Develop programs for processing large volumes of data
• Design and implement new PySpark pipelines in Databricks
• Ingest data from APIs
• Handle Spark Structured Streaming pipelines
• Design and manage objects in Unity Catalog according to Medallion Architecture
• Perform web scraping
• Work using Agile and Scrum methodologies

This is a 100% remote role within Poland.