Fundamentals over tools: how to evaluate and build with generative AI

A team adopts a chat assistant, gets good results for a month, and then the vendor changes the model, the price or the rate limit. Nobody in the room can say whether the alternative is better, worse or merely different, because the team learned a product, not a mechanism. That gap is the subject of this article: what you need to understand about generative AI to evaluate its limits, choose technologies with criteria, avoid depending on a single tool and, eventually, build applications with a model inside them rather than only typing into someone else's.^[9]

1. Fundamentals outlast tools

Generative AI is usually met through an interface: a text box, a completion in the editor, a button that drafts an e-mail. The interface is convenient and also misleading, because it hides the only thing that transfers between products: the mechanism. A person who knows how a model produces text can predict where it will fail, decide what to check, and read a new release for what actually changed. A person who only knows which buttons to press starts over every time the buttons move.

• Evaluate limitations. Knowing that a model samples from a learned distribution explains why it can be fluent and wrong at the same time, and why more confident phrasing is not more evidence.
• Choose with criteria. Understanding the cost structure of inference turns "which model is best?" into "which model is adequate for this load, this budget and this quality bar?".
• Avoid lock-in. When you know what the model does and what your application does, moving to another vendor or to a self-hosted model is an engineering task, not a rewrite.

In an ecosystem where names change every quarter, conceptual knowledge is the asset with the longest shelf life. The rest of this article builds that knowledge in the order an engineer needs it: first the layers, then the mechanism, then the criteria, then the system.^[9,10]

2. Where LLMs sit: generative AI, machine learning and deep learning

The terms are often used interchangeably, and that is the first source of shallow decisions. They are nested. Machine learning is the field of programs that improve from data instead of being written rule by rule. Deep learning is the subset that uses neural networks with many layers to learn representations directly from raw data.^[1] Generative models are models, usually deep, trained to produce new samples that follow the patterns of their training data. Large language models (LLMs) are one family of generative models, built on the Transformer architecture and trained on text at very large scale.^[2,3] A chat assistant is none of these: it is an application built on top of an LLM.

The hierarchy matters because each level answers a different question. When someone uses a chatbot they are interacting with an application: prompts, memory, tools and interface are application decisions. When they choose an LLM they are choosing a model family: size, licence, context window and cost are model decisions. When they discuss training data, loss functions or how meaning is represented in vectors, they are at the level of machine learning and deep learning. Separating the layers makes technical analysis sharper and stops a product feature from being mistaken for a property of the field.

Table 1 — The four levels, the question each answers and the decisions that belong there. A benchmark score is a model-level fact; a good answer in your product is an application-level result.^[6]

Practise: open in the Lab: Exercise 1, Where LLMs sit

3. The mechanism behind generative models

A generative model is a system trained to learn the patterns of a data distribution and to produce new outputs consistent with those patterns. For text the recipe is concrete: the model is trained to predict the next token given the previous ones, over an enormous corpus; at inference, a prompt sets the context and the model samples one token at a time, feeding each choice back as context for the next.^[3] The article on what generative AI is, further along in this series, introduces that factorization in detail; here the interest is in what it implies for the engineer.

Three consequences follow directly from equation (1). First, the model has no notion of truth, only of likelihood under the data it saw: a plausible continuation is preferred to an accurate one when the two differ. Second, everything the model "knows" is frozen in its parameters at training time; what happened afterwards has to be supplied through the prompt. Third, the output depends on a random draw, so the same prompt can produce different answers, and the sampling settings are part of the engineering surface, not a detail.

This is why understanding the mechanism changes how you use the tool. A model that answers well in one context and fails in another is not being capricious: the first context resembled its training data and the second did not, or the prompt left out what the weights could not contain. The response is then a design decision — retrieve the missing context, constrain the output format, lower the temperature, add a validation step — rather than a better-phrased request. The question is never only "what did it answer?" but "where does this capability come from, and what limits come with it?".^[7]

Practise: open in the Lab: Exercise 2, What the model predicts

4. Comparing architectures by engineering criteria

Choosing a model by popularity, or by the impression left by a demo, is the application-level equivalent of picking a database because it is trending. A mature comparison looks at engineering criteria: what the model costs to run at the expected load, how long a request takes, how well it can be specialized, what quality it reaches on your task, and how well it fits the constraints of the case — privacy, licence, deployment, language.^[6] A model that leads a public benchmark can be the wrong choice for production if it is too expensive or too slow for the traffic, and a modest model can be the right one if the task is narrow and the volume is high.

Table 2 — Engineering criteria for comparing generative models. "What to measure" is deliberately about your workload: a public leaderboard measures a different distribution of requests than the one your product will see.^[6,11]

5. Trade-offs: quality, cost, speed and specialization

There is no model that is ideal in every scenario; there are trade-offs. Raising quality usually means a larger model, which raises the cost per request and the time to answer. Chasing speed means a smaller model or shorter outputs, which can lower precision. Specializing a solution to a domain can deliver more value than the most general model available, at the price of keeping the specialized version current. Scaling results make the shape of these curves explicit: capability grows smoothly with parameters, data and compute, and so does the bill.^[4,5]

Two formulas make the cost and speed side concrete. The cost of one request is the number of input tokens times the input price plus the number of output tokens times the output price; output tokens are typically several times more expensive than input, so verbose answers cost more than long prompts. The latency of one request is the time to the first token plus the number of output tokens divided by the generation rate; that second term is why a 500-token answer feels slow even from a fast model.^[11]

The engineering work consists of balancing these variables against the business goal, the budget and the experience the user expects. A useful habit is to write the constraints down before looking at any model: a maximum latency, a maximum cost per request, a minimum quality on the evaluation sample. Candidates that violate a constraint are out, whatever their reputation; the remaining ones are ranked by a weighted score. The script below does exactly that, with the weights as an explicit, reviewable input.

from dataclasses import dataclass

@dataclass
class Candidate:
    name: str
    quality: float        # 0–1, accuracy on YOUR evaluation sample
    cost: float           # currency per request, from equation (2)
    latency_p95: float    # seconds, from equation (3) measured under load
    lock_in: float        # 0–1, how hard it is to replace (licence, formats, hosting)

def rank(candidates, max_cost, max_latency, min_quality, w):
    """Drops every candidate that violates a hard constraint, then ranks the rest
    by a weighted score. w = weights for quality, cost, latency and lock-in."""
    feasible = [c for c in candidates
                if c.cost <= max_cost and c.latency_p95 <= max_latency and c.quality >= min_quality]
    if not feasible:
        raise ValueError("no candidate satisfies the constraints: relax them or change the design")
    # normalize cost and latency against the worst feasible value, so that lower is better
    worst_cost = max(c.cost for c in feasible)
    worst_lat  = max(c.latency_p95 for c in feasible)
    def score(c):
        return (w["quality"] * c.quality
                + w["cost"]    * (1 - c.cost / worst_cost)
                + w["latency"] * (1 - c.latency_p95 / worst_lat)
                + w["lock_in"] * (1 - c.lock_in))
    return sorted(feasible, key=score, reverse=True)

# example: an internal support assistant, 50k requests/day, interactive
weights = {"quality": 0.45, "cost": 0.25, "latency": 0.20, "lock_in": 0.10}
ranked = rank([
    Candidate("large hosted",     quality=0.93, cost=0.012, latency_p95=4.8, lock_in=0.8),
    Candidate("mid open",         quality=0.84, cost=0.004, latency_p95=2.9, lock_in=0.3),
    Candidate("small fine-tuned", quality=0.88, cost=0.001, latency_p95=1.1, lock_in=0.2),
], max_cost=0.010, max_latency=3.0, min_quality=0.80, w=weights)
print([c.name for c in ranked])   # the large hosted model is out: too slow and too expensive for this case

The point of the script is not the arithmetic but the discipline: constraints first, weights explicit, quality measured on your own sample. With those three in place, "we chose model X" becomes a statement someone can audit and revisit when the numbers change.^[8]

Practise: open in the Lab: Exercise 3, Trade-off criteria

6. Building real solutions with AI

Building a real solution means leaving the passive use of a tool and designing an application that solves a concrete problem in someone's day. The model is one component of that application, and rarely the largest one. Around it sit the code that assembles the prompt from the user's request and the relevant context, the retrieval that brings in documents the weights could not contain, the tools the model is allowed to call, the validation that checks the output before it reaches anyone, and the observability that tells you what it costs and how often it fails.^[7,10]

This is where the fundamentals pay off. Selecting the model is one decision among several: evaluating the architecture, understanding the impact on the workflow, deciding what the model may do on its own and what a person must confirm, and aligning the solution with the kind of task the team or the product actually performs. A professional who has these fundamentals stops consuming AI and starts designing systems with AI as a component. That step is what separates experimentation from applied engineering, and it is also what keeps the system alive when a model is swapped: the prompts, the retrieval, the checks and the metrics stay.^[10]

Practise: open in the Lab: Exercise 4, Hard constraints

7. A fast-moving ecosystem: adapt without starting over

The generative AI ecosystem moves quickly: new models, tools and techniques appear continuously, and a leaderboard from six months ago is history. Following the field is therefore not optional; it is part of the technical competence needed to stay relevant. But following it does not mean memorizing product names. It means keeping stable criteria for judging novelty, maturity and usefulness, so that each release can be read quickly: what changed at the model level, what it costs, what it would change in your application, and whether your evaluation sample confirms the claim.^[9,12]

Someone who understands the fundamentals absorbs these changes without restarting from zero with each new technology. The Transformer of 2017 is still the architecture under the models of today; the scaling behaviour described in 2020 still explains why larger models cost more; the pipeline in Figure 2 has not changed. What changes is the position of each product on the trade-off chart, and that is exactly what stable criteria are for.^[2,4]

Practise: open in the Lab: Exercise 5, Fast ecosystem

8. From user to builder

Mastering generative AI means taking an active position in the creation of applications, not only using ready-made interfaces. That mastery combines technical understanding of the mechanism, the ability to read a context — the task, the users, the constraints — and the craft of turning a model into a useful solution. In software teams it translates into more conscious decisions about adoption, integration and practical return: which tasks gain, what the gain costs, and what has to be reviewed. The expected result is not just individual productivity; it is the capacity to build better products and better workflows, with the model as one well-understood part of them. The next article follows what this shift means for the profession itself: where the developer's differential moves when code stops being it, how AI amplifies competence, and what changes for technical leadership and for the products.