Is the problem worth solving?
Clarify the business problem we’re solving, how it maps to an ML problem and its solution, what is the expected effort and what the resulting improvement would actually be worth. If it’s a large engineering effort with marginal improvement, that’s important to know before we start.
Do we need ML at all?
Do we have a simpler baseline? Heuristics, rule sets, or off-the-shelf tools often offer the best value for money.
Do we have the data to support this?
“We have a lot of data” is very different from “The data is high quality, enough to support the specific task, labelled appropriately, and readily available to the right engineers/scientists”.
Can we get better data to support this?
Producing high quality annotations for supervised learning looks expensive and boring, but it often has a higher ROI than choosing an inferior approach simply because it works with the data we already have.
Are we measuring “success” correctly?
We’re optimising for NDCG (a ranking metric), while the business cares about revenue. How are the two connected?
What is the cost of a mistake?
We’re not going to have 100% accuracy. How do we deal with it? Do false positives and false negatives have the same cost? How does that compare with solving this problem manually?
What happens if it doesn’t work?
ML is iterative and experimental. We often don’t know if and what will work. Is the culture ready for experimentation? What if we discover the data isn’t good enough, the signal isn’t there, or the improvement isn’t worth the complexity? Do we have executive sponsorship for this type of uncertainty?
What happens if it works?
Who will use the model output, and how? What decisions will it affect? How does it get to production? What happens when the data changes? How are we monitoring quality in the long term?
These help us make sure we’re building the right thing, and that it gets used.
