Welcome

This course begins with a question from the management of a mutual insurance company: the assistant already works, shall we open it to the whole workforce? A demonstration or a brochure's claim is not enough to answer it. In four blocks you will write what getting it right means, build a battery of cases, mark them, read the failures and sign a one-page report. Have your AI assistant to hand and start choosing a task from your work that involves no personal or confidential data: you will build your own test set with it.

What you will find

• Block 1. What getting it right means: the criterion before the score. Before measuring, you have to decide what a good answer is and check that two people judge it the same way. And why a demonstration or a public ranking says nothing about your task.
• Block 2. The test set: an exam made by you. Where the cases come from, how many are needed and how an exam is kept so that it remains one. The score says how much the system fails; the failures, once read, say what to fix.
• Block 3. Who marks: people, rules and models. Three ways of marking and what each costs. A model that marks another is a detector with its false alarms. And how to tell whether a version is genuinely better or has just been lucky.
• Block 4. From the exam to everyday use: tracking, changing and withdrawing trust. An agent is evaluated on the result and on the path. The system you use changes without notice, so the exam is repeated. And someone decides when trust is withdrawn.

How you will work

Each lesson opens with a short introduction and, below it, several pills: short pieces that explain a single idea and end with a case or something to do. Read them in order the first time; afterwards you can go back to the one you need.

Where you see an interactive practice, do it before reading on: it is there for you to decide, calculate or classify, and the explanation comes afterwards. It does not save what you write, so copy whatever you want to keep.

Each block ends with an exam of six questions. All the answers come with a comment, including the wrong ones: failing and reading why is part of the method.

At The Singular Academy you learn by solving real cases. If at the end of a lesson you do not know what you would do differently tomorrow, go back over it.