Research Engineer (Agentic Behavior – Kotlin AI Value Stream)

JetBrains is a company focused on creating developer tools, with a team dedicated to AI-powered coding agents for Kotlin across multiple platforms including Android, Kotlin Multiplatform, server-side, web, and desktop. The Kotlin AI Value Stream team builds evaluation infrastructure, error analysis tools, and post-training pipelines to improve AI agent behavior on Kotlin developer tasks.

• Hands-on experience building evaluation or analysis pipelines for LLMs or AI coding agents in research or production.
• Strong Python engineering skills (at least three years) with ability to write clean, maintainable code in data-heavy and ML-adjacent codebases.
• Experience with data analysis at scale including querying large datasets, building data pipelines, and statistical analysis.
• Ability to own projects end to end from problem identification to solution deployment.
• Product-aware mindset focused on real developer usage and translating failure modes into evaluation and training.
• Familiarity with Kotlin or strong willingness to develop deep Kotlin expertise.

Nice to have:

• Experience with post-training LLM techniques such as SFT, RLHF, DPO, GRPO.
• Experience with modern deep learning frameworks (PyTorch) and LLM training stacks.
• Experience in AI agent development including tool-using agents and multi-step coding workflows.
• Experience with evaluation frameworks and tools like Inspect AI, Promptfoo, LM-evaluation-harness.
• Experience with experiment tracking and observability tools such as Weights & Biases, MLflow, Langfuse.
• Knowledge of the Kotlin ecosystem including Android, Gradle, KMP, Spring, Ktor.
• Contribution to or maintenance of open-source projects, especially benchmarks or evaluation tools.

Build tools for agentic error analysis:

• Design and implement tooling to systematically capture, classify, and analyse errors that AI coding agents make when generating Kotlin code.
• Build observability pipelines over agentic traces from JetBrains IDEs and other coding agents.

Build evaluation pipelines:

• Design, implement, and maintain evaluation pipelines measuring Kotlin code generation quality across multiple dimensions.
• Build simulation environments for coding agents on realistic Kotlin developer tasks.
• Own evaluation infrastructure including metrics, experiment tracking, automated regression checks, and benchmarking.

Research methods for improving agent and model behavior on Kotlin:

• Experiment with post-training techniques to improve model handling of Kotlin-specific patterns.
• Investigate context engineering approaches and tooling integrations.
• Run experiments to measure impact using A/B comparisons and benchmark suites.
• Collaborate with model providers to translate Kotlin-specific findings into model improvements.

Build public Kotlin benchmarks:

• Design and build open-source benchmarks measuring AI coding agent performance on Kotlin tasks.
• Create task datasets covering various Kotlin usage areas.
• Include mined real-world and synthetic tasks testing Kotlin capabilities.
• Maintain and evolve benchmarks to remain challenging and relevant.

We are an equal opportunity employer welcoming everyone regardless of background, identity, religion, age, accessibility needs, or orientation. Data processing is in accordance with the Recruitment Privacy Policy.