In applied predictive modelling, practitioners usually face many methodological questions which are related to different objectives of the overall development and evaluation pipeline, e.g.:

• “How should I split my data for training, comparison and testing purposes?”
• “Which metric is suitable for assessing the utility of the developed models?”
• “How can I quantify uncertainty of my evaluation results?”

In the talk, different examples of such problems related to model evaluation will be discussed and current developments on the expert system mlguide will be presented. In addition, an outlook on planned features and user research will be given.

While “standard” answers to these questions exist, it is rarely clear if and in what sense they are optimal or even admissible for the prediction task at hand. In the first part of the talk, we will consider three relevant and concrete examples of such questions related to model evaluation. Furthermore, we will illustrate how the derivation of more appropriate (than “standard”) answers is in principle possible based on recent methodological work. However, finding problem-tailored solutions in the scientific literature can also be difficult and time-consuming, in particular for practitioners without a specific data science background. In the second part of the talk, I will therefore introduce the expert system mlguide which is currently under active development at Fraunhofer MEVIS. mlguide aims to provide interactive support to practitioners for solving methodological questions in applied machine learning problems, such as those mentioned before. I will illustrate the overall system architecture, some insights into the guidance engine, and current limitations. To conclude, I will give an outlook on planned features and user research.