Lead Data Engineer (Spark)

Addepto is a leading AI consulting and data engineering company that builds scalable, ROI-focused AI solutions for large enterprises and startups, including Rolls Royce, Continental, Porsche, ABB, and WGU. The company focuses exclusively on Artificial Intelligence and Big Data, helping organizations unlock the potential of their data through systems designed for measurable business impact and long-term growth. Addepto is part of KMS Technology, a US-based global technology group, combining AI specialization with enterprise-scale delivery capabilities. The company has developed its own product, ContextClue, and contributes open-source solutions to the AI community. It has been recognized by Forbes as one of the top 10 AI consulting companies worldwide.

• 7+ years of proven commercial experience in implementing, developing, or maintaining Big Data systems.
• Strong programming skills in Python or Java/Scala, including writing clean code and OOP design.
• Experience in designing and implementing data governance and data management processes.
• Familiarity with Big Data technologies such as Spark, Cloudera, Kafka, Airflow, NiFi, Docker, Kubernetes, and Iceberg.
• Proven expertise in implementing and deploying solutions in cloud environments, preferably AWS.
• Excellent understanding of dimensional data and data modeling techniques.
• Excellent communication skills and consulting experience with direct client interaction.
• Ability to work independently and take ownership of project deliverables.
• Master’s or Ph.D. in Computer Science, Data Science, Mathematics, Physics, or a related field.
• Fluent English (C1 level) is required.

• Design and develop scalable data management architectures, infrastructure, and platform solutions for streaming and batch processing using Big Data technologies like Apache Spark, Hadoop, and Iceberg.
• Design and implement data management and data governance processes and best practices.
• Contribute to the development of CI/CD and MLOps processes.
• Develop applications to aggregate, process, and analyze data from diverse sources.
• Collaborate with the Data Science team on data analysis and Machine Learning projects, including text/image analysis and predictive model building.
• Develop and organize data transformations using DBT and Apache Airflow.
• Translate business requirements into technical solutions and ensure optimal performance and quality.