Project Mission
Large Language Models (LLMs) are gradually becoming part of academic and industrial processes due to their inherent capacity to solve a multitude of different problems across different domains. However, evaluating a large LLM portfolio across multiple criteria introduces high computational cost, which then translates into a negative environmental impact, especially in terms of increased carbon emission. A highly relevant question remains open:
From the multitude of LLMs available, how to select the most appropriate LLM to use on a specific supervised machine learning (ML) problem (with or without fine-tuning), without evaluating a large portfolio of LLMs on the labelled dataset related to that ML problem?
The main objective of the AutoLLMSelect project is to develop a robust, explainable, and evolving framework for selecting the most appropriate LLM for use on a previously unseen general-domain or domain-specific dataset, without having to evaluate the LLM on that dataset. The project will develop a proof-of-concept on a selected LLM portfolio, dataset portfolio, and performance metrics. The initial framework will be easily extensible with new LLMs, benchmark datasets, machine learning tasks, and performance metrics.
