Why doesn’t AI give the same answer every time?

An AI tool can provide different answers to the same question. For engineers, that raises a practical concern as these tools become part of everyday work: how do you use them when a calculation, measurement, or decision needs a dependable basis?

Nathan Power leads AutoReview, CoLab’s AI design checker. He previously worked on warehouse robots at Mytra and industrial 3D printers at AON3D, and completed a mechanical design internship at Tesla. In this conversation, he explains why AI responses can vary, where that variation is useful, and what engineers need to verify before acting on a result.

What causes an LLM’s answers to vary?

Nathan Power: The large language models, or LLMs, behind many familiar AI tools are probabilistic, not deterministic. They generate responses by predicting what comes next instead of operating on fixed rules, like a calculator does. 

LLMs process text in pieces called tokens. A token can be a word, part of a word, or punctuation. During training, models learn patterns that help them predict which token is likely to come next in a sequence.

When generating a response, the model assigns probabilities to possible next tokens. If the system samples from those possibilities, it can choose different tokens on different runs. Each chosen token influences the next prediction, so the response can develop differently. That doesn't mean every response must be different, but you shouldn't assume the same question will always produce the same answer.

For engineering applications, answering a query with different wording isn't necessarily a problem. But if two responses can lead to different design decisions, calculations, or checks, you need a way to review the basis for each.

It’s important for engineers using LLMs to understand how they process inputs and generate outputs because it helps clarify the types of problems these tools are suited to tackle, and which they are not. Sometimes open-ended exploration is what you want. Other times, you need a firm answer you can verify.

When is variation useful, and when do engineers need a repeatable result?

NP: Variation is useful when brainstorming or developing new concepts. I probably don't want the AI to give me the same answer every time. I might want ideas for tweaking a design, reducing cost, or using a different material. That gives us possibilities we might not have considered, which we can work through using our engineering judgment.

But there are times when I need a firm answer. A deterministic check applies a defined rule to the inputs and produces a repeatable result: a measured value, an identified violation, or a pass/fail decision. Engineers already use checks like these:

• Interference: Do parts overlap in the assembly?
• Minimum clearance: Is there enough space between components to meet the requirement?
• Hole-to-edge distance: Is the hole far enough from the edge of the part?
• Tolerance stackup: At the worst-case tolerance limits, will the assembly still fit or function?

If the inputs and rule haven't changed, the answer shouldn't change either. A clearance check that passes on one run and fails on the next could lead us to approve parts that won't fit together, or spend time fixing something that already meets the requirement.

One thing we've noticed at CoLab is that an LLM isn't always the best tool to use when reviewing designs. We can extract the geometry, measure it, and run a calculation. If the geometry is sized properly, it's sized properly. There's no need to layer an LLM on top of that.

The same design can involve both kinds of work. You might explore several ways to change a feature, then use a defined check to assess the option you choose.

What does that mean when using AI for a common engineering task, like DFM?

NP: Let's take a machined mounting plate as an example. You've designed the first draft in CAD and want to know whether you can make it with the CNC machine in your shop, or if a supplier can manufacture it. This is where you might ask another engineer for a sanity check before cutting metal or emailing suppliers.

A deterministic hole-to-edge check flags a hole that's too close to the edge of the plate. Drilling it could break through the wall. An LLM can help you explore several possible changes. Sometimes, the fix is easy: move the hole. Other times, the surrounding geometry needs to be reconfigured to accommodate the new hole placement. You can work through those options using your engineering judgment.

Once you've revised the design, you run the same check again: the suggestions can vary, but the pass/fail result comes from the measurements and the defined rule.

This is how CoLab’s AutoReview works. It acts as your first-pass design checker, using prewritten deterministic checks to assess the geometry and an LLM to bring the findings together in the context of the part or assembly.

What evidence should engineers require before acting on the result?

NP: A key characteristic of a great engineer is knowing your assumptions: what you're assuming to get to an answer, and what you're setting aside to come back to later.

Take the mounting plate example above. Before accepting the hole-to-edge check, you need to know:

• Which hole and edge were measured
• Whether the dimensions are correct
• Where the threshold comes from
• Whether the governing requirement or guideline applies to your part and manufacturing process

Those are just some of the inputs and assumptions you need to verify. A repeatable calculation can still give you the wrong answer if it's checking the wrong input or applying the wrong requirement.

Being able to inspect that evidence matters. In CoLab, AutoReview can place findings on the model and cite the uploaded standards behind its feedback so engineers can inspect the flagged geometry alongside the source requirement.

A pass tells you that the feature meets that check. It doesn't establish that every manufacturing requirement has been assessed. You still need to understand what's been checked and what remains unresolved before deciding the part is ready.

The same applies to an AI-generated design suggestion. If an LLM suggests moving the hole, you might assume the new position still lines up with the part the plate mounts to. Passing the hole-to-edge check doesn’t establish that. You still need to verify the alignment before accepting the change.

It's the age-old question of what technology to use for the task. Engineers are already adept at making that choice. AI can inform engineering judgment, but the engineer remains responsible for choosing the tool and deciding whether to act on its output.

So, why doesn’t AI give the same answer every time, and what does that mean for engineering rigor?

NP: LLMs generate responses from probabilities, and sampling can lead them to different answers. Sometimes that open-endedness and variation are great for engineers. At other times, they're a hindrance. Engineers need to choose the right tool for the task. For a defined check, that means a repeatable method with inputs and criteria you can verify.

We don't want engineers blindly accepting whatever comes out of the LLM. We want them to continue using their engineering judgment. As a leader, you should expect your team to know what they're assuming, what they've checked, and what they're setting aside to come back to later.

Getting the same response twice doesn't establish that it's correct. AI can help engineers get to an answer faster, but you still need to be able to explain why the answer is sound.

Subscribe to AI Office Hours for more practical answers to engineering leaders' questions about AI.