Welcome

This course starts with a sentence you have already read: the agent's model reproduces daily sales "with an average error of 9.2%" and therefore "can be used to plan the fourth quarter". Over four blocks you will find out what lies behind that figure and write the evaluation report that management needs before trusting it. Keep the Hornos Lumbre data (daily sales, stores and weather) and your copilot at hand: here it does the calculations and you check them.

What you will find

• Block 1. Validating without shortcuts. A model's exam has to be done with data it has not seen and has to resemble the use it will be given. And the most common trap is leakage: information from the answer that slips in among the variables. First reading of the case: an exam without August.
• Block 2. Measuring: the metric is a decision. Each measure penalises one kind of error, and choosing it means deciding which error matters. Errors in a quantity and errors in a yes or a no, split by segment and with the cost of each in front of you. Second reading of the case: the 9.2% that hides three zones.
• Block 3. Comparing and explaining. Without something to compare it with, an error says nothing. And a model that is going to be used to make decisions has to be explainable and treat everyone equally. Third reading of the case: the rule without a model that got more right.
• Block 4. From the model to the system: evaluating an AI and keeping watch. The bridge of the course: evaluating an assistant or an agent is validating, with an answer that is text and a grader that also makes mistakes. And what comes after the exam: the world changes and the model does not. Last reading of the case: the evaluation report.

How you will work

Each lesson opens with a short introduction and, below it, several pills: short pieces that explain a single idea and end with a case or something to do. Read them in order the first time; afterwards you can go back to the one you need.

Where you see an interactive practice, do it before reading on: it is there for you to decide, calculate or classify, and the explanation comes afterwards. It does not save what you write, so copy whatever you want to keep.

Each block ends with an exam of six questions. All the answers come with a comment, including the wrong ones: failing and reading why is part of the method.

At The Singular Academy you learn by solving real cases. If at the end of a lesson you do not know what you would do differently tomorrow, go back over it.