Should a Capable Engineering Organization Just Build Its Own AI?

A few months back, an enterprise engineering leader at a large industrial organization recounted to me the time he bamboozled an “advanced AI” part similarity tool by asking it two simple questions.

First, he showed it two bolts: one with modeled threads and one with unmodeled threads. The system treated them as different parts, despite having identical descriptions and properties. Next, he showed it two bolts with identical CAD geometry but different materials. The system treated them as the same part.

For this engineering leader, the fact that this “advanced” AI tool couldn’t reconcile basic geometry or material properties amounted to yet another failure in a string of failed AI pilots. 

This same frustration and skepticism towards AI is common among many engineering organizations that I meet with. In my experience, this isn’t caused by resistance to new technology. Engineers are some of the most technology-savvy people I know. Instead the skepticism is the result of repeated technical failures from vendors who overpromise and underdeliver.

When it’s true that:

1. Engineers inside the organization understand the problem far better than most AI systems, and;
2. Frontier AI models are now customizable enough to build custom solutions to tackle specific use cases;

The natural question becomes: why shouldn't we just build the capability internally?

There are legitimate cases for doing exactly that. But for many engineering organizations, building the AI capability is only the beginning of the conversation. In this article, I’ll unpack what that decision really entails.

Capable orgs can build AI. But should they?

Engineering organizations consist of really smart people building really complicated things. With the capabilities of today’s AI tools, there is no question that many of these teams can build a useful agent or proof of concept.

However, this decision should not be framed as a question of whether the team is smart enough or has access to the right tools. Because the answer to both questions is yes.

‍It is a question of organizational focus.‍

Engineering organizations have their domain of what they build and deliver. Leaders have to decide intentionally whether operating AI should become another domain of expertise, and whether that investment will produce the most capable system and the best value for the organization.

One company I spoke with was grappling between purchasing AI software from a partner and standing up an eight-person team to maintain custom agents across seven separate instances of PLM. The proposal included technical, capable engineers assigned to this team. But the real question concerned the operating scope that would persist after launch.

AI that benefits the enterprise has to:

• Connect the right sources, standards, and guidelines 
• Surface the right data at the right time 
• Bring forward the appropriate capabilities to produce reliable outputs
• Maintain accurate guidance as engineering knowledge and underlying tech changes
• And fit seamlessly into an engineer's day-to-day work 

A successful internal tool does not automatically settle whether the organization should assume permanent responsibility for all of that.

Where building AI internally really does make sense

Sometimes an organization should build an AI capability internally. The strongest case is when the tool does one narrow job in a way that is genuinely unique to the organization.

A good example comes from a team I work with. Its test data was stored in a highly specific, relatively stable format. Engineers built an agent to triage the data and identify the results worth investigating. The inputs and expected output were clear, so the organization knew what it was committing to own. In this case, building internally made sense.

The agent still needed somewhere to live. It remained a specialized component within a broader engineering system that connected its output to the knowledge, decisions, and work around it.

When deciding whether to build AI capability internally, these four questions can help guide the discussion:

1. Does the capability itself encode something unique, or does it simply operate on proprietary data?
2. Can we clearly define its inputs, expected output, and measures of success?
3. Are its requirements stable enough, and is there a permanent owner for its upkeep?
4. Can it connect to the shared system rather than create another information silo?

If those conditions hold, building internally may be the right choice. 

My recommendation is to build the capability that encodes what is unique to your organization, and make a separate decision about the system that connects, governs, maintains, and delivers it.

The decision changes when one bounded capability becomes several, or when the organization starts building the system around those capabilities. What began as a contained tool can quickly compound into a much larger operating commitment. Here are some patterns I’ve seen as that happens.

Different teams can quickly build in different directions

A proper system has to tie AI agents together, connect them to the right data sources, and use the right components at the right time. This is familiar territory for engineers: a collection of locally optimized components does not automatically become a well-performing system.

I've seen organizations give teams access to LLMs and tools for building their own point solutions, dashboards, skills, and agents. One department becomes convinced it is building an innovative tool that will solve its problems. Another department nearby is doing the same thing with a different tool, yet feeling the same way.

The individual work may be technically sound. At the organizational level, the efforts consume parallel engineering bandwidth while information and decisions begin moving in different directions.

Some organizations I work with have responded by creating AI councils that take control of this conversation. They decide where internal investment belongs and which systems need to connect those efforts, focusing attention, resources, and knowledge on one common mission rather than several dispersed ones. That is one operating response, not a universal prescription. Somebody still has to coordinate independently valuable capabilities.

The model is only one layer of the product

Another failure mode I see is a team building toward a really flashy demo that shows the art of the possible, without fully grasping the complexity required to bring it into production. These are full-blown products. Model performance is only one part of the work.

Production introduces access and permissions, licensing, data residency, hosting, model-provider terms, legal and security review, and the user interfaces that connect capabilities. A capable internal team can address those requirements. The question is whether the original decision and business case included them.

At one regulated engineering enterprise we work with, an AI drawing-review capability resonated as a practical first step. When a group wanted to evaluate it using the organization’s own data, it first had to reconcile data-residency and regulated-data requirements, validate where model processing occurred and what retention terms applied, complete a security review, and establish the necessary legal agreement.

This operating layer exists regardless of who built the capability. If you own the whole system, you own this layer too.

The system has to keep pace with two moving targets

Ongoing maintenance is easy to underestimate because an AI system has to keep pace with two moving targets after launch:

1. The organization’s engineering knowledge
2. The underlying AI technology

Keeping the system aligned with both is permanent work.

Standards and manufacturing capabilities are constantly changing. New products generate new lessons about what to do and what to avoid. Even after an organization has made that knowledge usable by AI, somebody has to keep it current and ensure the system continues applying it correctly.

Read more: Is your engineering data ready for AI?‍

An internal tool also starts with the models and infrastructure available at a specific point in time. Keeping it current requires ongoing R&D and a team responsible for the upkeep.

One example from our engineering work shows what that involves. CoLab’s similarity engine allows teams to query their CAD data to find similar historical programs and parts. One team we work with wanted more granular search capabilities. Our machine learning team solved this problem for them, but the technical fix was only the beginning. Rolling out the change meant re-ingesting existing data, carefully reviewing permissions, and implementing the update in stages.

Even for a full-time product team, a seemingly narrow capability change such as refining search functionality can quickly propagate across the surrounding system. Change management is at least half the battle because maintenance continues long after the first version works.

A technically capable AI tool still fails if engineers don’t use it

Building a capable tool is only part of the work. It also has to fit engineers’ day-to-day work well enough to earn sustained use.

Engineers should be able to enter a tool, learn it quickly, and understand how to use it without extensive teaching or coaching. The capability should appear in work they are already doing, bring in the right stakeholders, and reduce the effort required to make a decision. If engineers have to reconstruct the system’s reasoning, repeat the task to verify it, or review the review, the promised efficiencies disappear.

Whether a tool is enjoyable to use may sound like a fluffy measure of utility. It’s not. The build vs. buy decision has to account for the work required to create an intuitive experience alongside a technically correct output. The goal is to delight engineers in a way that drives organic adoption and changes the work for the better. 

A pilot can reveal whether a tool is intuitive for a defined use case. The decision also has to account for the ongoing work of refining that experience across the teams and workflows it will serve.

Should a capable engineering organization just build its own AI?

When an AI demo fails because it treats two bolts made from different materials as the same part, skepticism is the natural response. An engineer familiar with these parts would never build that mistake into a tool. A failure like that makes the instinct to build the capability internally understandable.

Yes, capable organizations can build valuable AI tools. Especially for narrow, clearly defined use cases that are unique to the organization and stable enough to own over time. But domain expertise addresses only one part of the decision. The complexity increases drastically once you understand the scope of the connected, governed, maintained, user-facing system required to deliver those capabilities at enterprise scale.

‍I don’t think that it’s possible to deliver AI at scale across the enterprise in a way that engineers will actually use it by trying to develop this internally. ‍

If you’re asking this question, think hard about your domain of expertise and the core value you offer to the world. Decide which AI capabilities are important enough to own. Then decide separately whether your organization is prepared to operate everything around them.

At CoLab, building AI for engineering organizations is our domain of expertise. With Builder, our upcoming workflow automation engine, we’re creating a way for engineering teams to build workflows on existing infrastructure without taking on the whole system underneath.

See how engineering teams at companies like Schaeffler, Bombardier, and GE Appliances are using CoLab to deploy AI.