Free preview · Fundamentals
How a model learns and makes a prediction
10 min
The video plays inside the course after you enrol. Here is what it covers.
Transcript
Cedar Harbor wants to predict which pumps will fail next week. Its analyst uses a table of repairs. One column says whether an engineer replaced the pump. The model looks uncannily accurate. Before approving a purchase, ask when that column becomes available. A replacement recorded after a failure would tell us the answer after the fact. A test that includes it tells us little about how the model will work on Monday morning. Training uses examples to learn a relationship between evidence and an outcome. Here, measurements taken before a failure are evidence. Failure in the following seven days is the outcome. Inference happens later: the fitted model gets a new pump’s measurements and estimates risk. Nothing about making that prediction means the model learns from the request. Someone must decide whether, when and with what data to retrain it. Set aside a later period and reconstruct what staff could have known at each forecast date. Compare the model to the current maintenance schedule. A forecast is useful only if a planner can act on it in time. A missed failure has a different consequence from an unnecessary inspection. Bring both numbers to the funding meeting instead of relying on one accuracy percentage. An executive asks whether the maintenance service uses artificial intelligence or machine learning. The categories can overlap. Artificial intelligence is a broad field. Machine learning is an approach in which a training procedure uses examples to fit a model. A model is the fitted representation used when new inputs arrive. Generative AI includes models that produce new content, such as a draft work note. A deterministic maintenance schedule might simply follow an explicit rule about how long a pump has run. It can be valuable without learning anything. Ask the vendor to name the part that learns from experience, the inputs used when it runs, and the output someone is expected to act on. Let's go back to the pump room. At eight on Monday, Mara has two technicians and three sites to cover. A blue light on the dashboard means a pump is likely to fail within seven days. What can Mara actually do with that information? She can send a technician on Tuesday, while a backup pump is available. She can't send one yesterday. That simple sentence gives the analyst a specification. Every input must have been accessible by Monday at eight, and the prediction must arrive early enough to change Tuesday's route. A model that predicts a breakdown thirty seconds beforehand could be excellent at forecasting and useless to Mara. Take a pencil and work through a hundred historical pump weeks. Ten ended in a breakdown. At the proposed threshold the model flagged twenty weeks, eight of which ended in breakdowns. How many alerts would Mara have investigated without finding the target event? Twelve. How many breakdowns would have arrived without a warning? Two. The analyst says the model caught eight out of ten, which sounds encouraging. The technician says only eight out of twenty callouts were associated with a failure, which sounds expensive. Both are describing the same result. One denominator starts with actual failures. The other starts with alerts. If someone reports eighty percent without naming the denominator, ask them to point at the boxes. Suppose those twenty alerts all arrive in a period with capacity for twelve inspections. Should we switch on the alert for every pump? Make a choice and name the cost you're willing to accept. I would first run the ranked list alongside the current schedule. Ask what the top twelve would have caught, which scheduled visits they would displace, and whether any of the missed failures involve the only working pump at a site. The twenty alert result can't answer the top twelve question. Simply multiplying eight successes by twelve twentieths assumes the ranking carries no information. A planner could also reserve inspection slots for critical equipment and use the score for the remainder. That makes the operational priority explicit. Three months later the finance director spots a puzzle. Most pumps that were inspected never failed. Does that prove the alerts were false? Imagine a technician found a cracked bearing and replaced it. The repair changed what happened next. The failure label now depends partly on the decision taken after the alert. Keep the preinspection measurements, the risk score as originally issued, the work order, the fault observed and the subsequent outcome. Ask a reliability engineer to distinguish evidence of a prevented failure from an entirely healthy pump. A safe prospective evaluation may be possible with comparable eligible routes and established maintenance safeguards. Withholding a mandatory inspection to get a clean comparison would defeat the purpose of the service. There's a second question hiding behind the pump proposal. Which kind of learning fits the evidence? If we have past measurements with reliable subsequent failure labels, we can learn from examples of the outcome we care about. That's supervised learning. Suppose we have vibration traces but maintenance history was lost in a system migration. We could group similar traces or flag unusual patterns without telling the model which pumps failed. That can help an engineer investigate. It can't magically supply the missing relationship between unusual vibration and imminent failure. Finally, a language model could draft a work note from an engineer's observations. Its draft is a description for someone to check. The same pump might appear in all three workflows. What distinguishes them is what the organization knows, what it asks the system to produce, and how it establishes that the result was useful. Return to our hundred pump weeks. Imagine a rival product that always predicts the pump will run normally. Ninety weeks ended without a breakdown, so the rival is right ninety times. It beats our original model's eighty six correct classifications on ordinary accuracy. Would you buy it to schedule inspections? You would receive no warning for any of the ten failures. This simple comparison explains why the sponsor should ask for a result expressed in the action's own units. How many inspections can we schedule? How many impending failures might those visits catch in time? What do we give up when the crew goes to the wrong site? A sensible benchmark might be the engineer's existing inspection order. Compare both methods on the same eligible weeks, count what each would have found within the available visits, and ask whether the difference justifies paying for the forecast. A bakery asks for a prediction of tomorrow's bread sales. It sells out at noon most Saturdays. Think about which label it has actually recorded. On a sellout day, recorded sales are limited by the loaves available. They don't tell the baker how many customers would have bought bread that afternoon. Training on those sales without noticing the stock limit could teach the model to keep ordering too little. Ask for stockout times, comparable days and evidence of unmet demand. As with the pump repair, an action taken by the business changes what the organization subsequently observes. The depot replaces a diesel pump fleet with electric units. The maintenance records are complete, and the old forecast still runs. Which part of the business case now needs new evidence? Sensor readings may mean something different on electric units, and the faults technicians can prevent may also differ. Keep the old results attached to the equipment population that produced them. Ask an engineer to inspect the signals on the replacement fleet and compare predictions with subsequent outcomes before the planner relies on the old threshold. A generated report can describe a prediction without having made the prediction itself. Think back to the example. Which field could leak the answer? Commit to a reason before opening the knowledge check.