Framework for Choosing an AI/ML model(s) (2025) A framework created to guide whether a problem is suitable for a large language model or simpler model. The framework balances UX intent, data complexity, and delivery constraints.
While LLMs are powerful, they can introduce unnecessary complexity, latency, and cost. This framework helps evaluate need with a simple, repeatable framework.
Create a tool to inform technical decisions and build cleaner, more efficient AI-powered tools • Is this problem a good match for a language model? • What kind of LLM architecture (if any) is ideal? • What are the constraints around context, latency, and explainability?
Designed the framework based on OpenAI/HuggingFace guidelines, real-world projects, and PM/UX best practices Prototyped as a lightweight Airtable scoring matrix Defined boundaries around context limits, prompt reliability, and user tolerance for ambiguity
Defined 4 core evaluation axes 1. Input structure - Is input unstructured, nuanced, or ambiguous? 2. Output fidelity - Does output need to be precise, factual or explainable? 3. Latency tolerance - Is speed or real-time feedback critical to UX? 4. Learning Loop - Does the system get better with feedback or personalization?
Used to vet 3+ internal feature ideas across health and productivity domains Scaled back initial use of LLMs since smaller, faster models proved more reliable Prevented unnecessary model overuse and reduced technical overhead Became a foundational tool for scoping technical needs for Chronic AI
Not all new problems need LLMs additionally, UX often benefits from simplicity. It is no different here. PMs must act as AI translators, ensuring user needs and model capabilities stay aligned Good scoping frameworks build cross-functional trust