Senior Data Scientist - ML Operations & Analytics ›
Filevine
Data Science, Operations, Software Engineering
Prague, Czechia
Posted on Sep 5, 2025
Responsibilities
- Build ML operations analytics tracking pipeline health, costs, latency, and quality metrics across all services
- Lead LLM evaluation initiatives using frameworks like Braintrust or Langfuse, designing custom metrics, quality monitoring and LLM observability in general
- Collaborate with our data annotation team on prompt engineering and evaluation dataset creation
- Aggregate and operationalize user feedback for model improvement
- Create actionable dashboards and insights for ML teams and leadership
- Help define SLAs, cost optimization and failure prevention strategies for our ML pipelines
Qualifications
- 3+ years in data science, ML operations, or analytics engineering
- Experience with LLM evaluation frameworks and quality metrics design
- Strong Python, SQL, and data visualization skills
- Knowledge of prompt optimization techniques
- Excellent communication skills in English to translate technical metrics into business insights
- Collaborative mindset with ability to work across machine learning teams
- Ability to meet periodically in our Prague office as we value in-person collaboration