This research explores when and how to efficiently integrate Large Language Models into information retrieval evaluation processes.

• LLMs can reduce the cost and time of creating test collections by assisting with relevance assessments
• The paper identifies optimal scenarios for LLM deployment in evaluation workflows
• Strategic use of LLMs can lead to more robust evaluations with limited research budgets
• Results show promising applications for enhancing research methodology while maintaining quality

For education researchers and professionals, this work provides practical guidance on leveraging AI to improve research efficiency without compromising on evaluation integrity.

LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?