Lilia Urmazova

- Collaborating in a cross-functional ML team on cutting-edge R&D initiatives, focusing on scalable AI architecture.

- Leading LLM/Agentic evaluation to assess reasoning capabilities, tool-use proficiency, and output quality.

- Selecting and validating evaluation metrics (e.g., G-Eval, RAGas, BLEU/ROUGE, or custom heuristics) tailored to specific agentic workflows.

- Developing end-to-end evaluation pipelines to automate performance tracking and regression testing for machine learning models.

- Developed and implemented comprehensive testing strategies for AI Vertex search and AI Visual search for merchandisers, including model performance testing, prompt testing, ethics testing, data validation, and bias assessment.
- Covered LLM metrics with Python automation and frameworks for AI testing, using LLM-as-a-Judge approach, and model cost analysis.
- Prepared and managed large-scale datasets in cloud storage for AI Vertex Search, optimizing data quality and accessibility for model training.
- Led a team of 10 QA engineers, ensuring compliance with best testing practices and project deadlines in an Agile environment.
- Collaborated with development teams and data specialists to define testing requirements and ensure end-to-end AI functionality quality.

- Proactively minimized production defects through meticulous testing practices, resulting in a demonstrably positive impact on product quality and customer experience.
- Led multiple projects as a solo QA, both manual and automation on TestComplete and SOAP UI.
- Was selected as the best QA for a challenging project among 100+ QA in the company.