How to think about creating a dataset for LLM fine-tuning evaluation

Alex Strick van Linschoten discusses the process of creating a dataset for evaluating LLM fine-tuning, focusing on core evaluations for accuracy, handling out-of-domain data, interpreting gradations of information, addressing spelling variations, and dealing with complex stories in structured data generation tasks. He emphasizes the importance of assessing model performance objectively rather than relying on intuition. By detailing various evaluation criteria such as measuring correct predictions, adapting to new data, handling ambiguous information, and ensuring consistency in output, he aims to enhance the accuracy and reliability of fine-tuned language models. Van Linschoten plans to implement these evaluations to compare different models and improve their performance systematically.