Senior Data Engineer

We are looking for a proactive Senior Data Engineer (PySpark) to join an international Agile team delivering large-scale data processing solutions for the financial sector. This role sits within a collaborative engineering environment and offers the opportunity to influence architecture, development standards and delivery quality across a complex data ecosystem.

• 5+ years of experience in software engineering with strong focus on PySpark development.
• Hands-on experience with Apache Spark, Hadoop, Hive, Spark SQL.
• Experience with Airflow or similar scheduling tools.
• Solid understanding of Unix/Linux environments.
• Experience with ETL frameworks, SQL, RESTful services and distributed data processing.
• Familiarity with Git/GitHub, CI/CD tools (Jenkins, Ansible) and JIRA.
• Understanding of data modelling (relational & non-relational).
• Strong debugging skills and ability to clearly communicate findings.

Nice to have:

• Experience with Elasticsearch.
• Java API development.
• Experience with ingestion frameworks.
• Understanding of cloud design patterns.
• Exposure to DevOps practices and Agile methodologies (Scrum, Kanban).

• Design and develop scalable data processing solutions using PySpark or Scala.
• Build and maintain data pipelines leveraging Hadoop ecosystem components (Spark, Hive, YARN, MapReduce).
• Promote engineering best practices through code reviews, mentoring and knowledge sharing.
• Provide production support, troubleshoot issues and ensure system stability.
• Implement tools and processes ensuring performance, availability, accuracy and monitoring.
• Collaborate with Business Analysts to ensure correct interpretation and implementation of requirements.
• Contribute to architecture and system design discussions.
• Participate in Agile ceremonies: sprint planning, reviews, retrospectives and daily stand-ups.