Data Engineer (Azure)

We support recruitment for a US-based company that is a provider of mission-critical background screening solutions. They work with Fortune 100 clients helping them manage risk and hire the best talent. This role will provide you with an outstanding opportunity to work for an industry-leading company. With over 4500 employees from 30+ different nationalities, you will be working with a diverse bunch of creatives redefining the world of digital background check and verification services across the globe.

• Proficiency in Python/PySpark and SQL.
• Strong experience building robust data pipelines with Databricks Spark.
• Proven track record handling large, complex datasets.
• Expertise in developing reusable data transformation libraries (Python packages).
• Deep knowledge of Databricks Delta optimization (partitioning, Z-ordering, compaction, etc.).
• Hands-on experience with CI/CD pipeline development.
• Skilled in event streaming integration using Kafka, Event Hubs, or Kinesis.
• Solid understanding of fundamental networking concepts.
• Familiarity with Agile/Scrum methodologies.

Nice to have:

• Spark Structured Streaming
• IaC (Terraform/Bicep)
• Containers (Azure Container Apps/K8s)
• Event sourcing
• CDC (Debezium)
• Azure native (Data Factory, Functions, ACI)

• Build reusable, metadata-driven data pipelines.
• Automate and optimize data platform processes.
• Develop integrations with data sources and consumers.
• Extend shared ETL libraries with transformation methods.
• Write unit tests.
• Create monitoring solutions for the Databricks platform.
• Proactively address ETL performance and quality issues.
• Collaborate with infrastructure teams on cloud resources.
• Update data platform wiki and documentation.
• Conduct code reviews to ensure quality.
• Initiate and implement architecture improvements.