Data Engineer

We are looking for a Data Engineer to join an international project delivered for a large organization in the financial services sector. The team is responsible for building and maintaining scalable data pipelines, supporting analytical and business teams, and ensuring high-quality data for strategic decision-making.

• 5+ years of experience in software design and development, preferably in Pyspark or Scala.
• Hands-on experience building data pipelines using Hadoop ecosystem components (Hive, Spark, Spark SQL).
• Experience with scheduling tools such as Airflow.
• Strong knowledge of Unix/Linux platforms.
• Experience with big data frameworks: Apache Hadoop, Apache Spark, YARN, Hive, Python, ETL frameworks, MapReduce, SQL, RESTful services.
• Familiarity with version control (Git/GitHub), automated deployment tools (Ansible, Jenkins), and requirement management (Jira).
• Strong analytical and debugging skills.

Nice to have:

• Experience with Elasticsearch and Java API development.
• Knowledge of cloud platforms and cloud design patterns.
• Experience with data ingestion processes.
• Exposure to DevOps and Agile methodologies (Scrum, Kanban).

• Collaborate with Engineers, Data Analysts, and Business Analysts to design, develop, and operate data pipelines and systems.
• Implement tools and processes to handle performance, scale, availability, accuracy, and monitoring of data platforms.
• Build and maintain ETL processes, using Hadoop, Spark, Hive, and other big data components.
• Ensure requirements are correctly interpreted and implemented by liaising with BAs.
• Promote development standards through code reviews, mentoring, and knowledge sharing.
• Participate in Agile ceremonies, including sprint planning, reviews, and retrospectives.
• Provide production support, troubleshoot issues, and contribute to improving system reliability.
• Contribute to system architecture and design decisions.