Integrating Generative AI into the Enterprise

David Robertson began his career as a programmer for NASA on the International Space Station project before transitioning to lead large-scale technology platform development. A DevOps enthusiast, he has pioneered test automation and infrastructure provisioning, influencing teams globally. You can find his tech musings on X @ShipItSoftware. His struggling career as a musician keeps his focus on architecting software platforms, but he still enjoys playing the guitar whenever he can. Lately, solar and wind systems help entertain his engineer’s mind, and give him chances to collaborate on those things with his kids. Through this article, David emphasizes the rapid evolution of generative AI technology and the importance for organizations to adapt and adopt these technologies to remain competitive. He highlights the need for organizations to understand the different types of generative AI tools available, from those that can be purchased to those that can be built in-house.

Where do we even start with this conversation? LLMs, vector databases, prompt engineering, RAG, and NPUs are all new terms for many of us, and things are changing so fast that it is difficult to keep up. If this is you, please read on, and I will attempt to demystify some of the current generative AI landscape. As technologists—or simply executives trying to stay competitive in today’s market—we must adapt and move forward. That doesn’t make us experts. Rather, it just makes us better than our competitors.

Technology leaders think of generative AI in two broad categories: the things organizations build versus the things organizations buy. In the “buy” category, things enter our workspaces either with a conscious decision to purchase specific AI tools, or they are made available as features to existing tools already available. The former consists of tools like GitHub Copilot or watsonx Code Assistant; tools for knowledge workers that are licensed explicitly as AI assistants. Others are new features added to existing software already in our workspace that may or may not require additional licensing. Most tools in this category are positioned as productivity enhancers: they can summarize our email inbox or create presentations from curated content. Most of these are low risk, and some include customizations to limit any data accidentally exposed to the wrong group. There is a lot of value in them, given the low maintenance and setup. You simply must pay the cost to get the productivity boost and have the right people who can maximize the investment.

  More so than other technology, one should invest in test automation to validate generative AI builds, both on the positive and negative side; double that if you put it in front of customers. It’s not cost-effective to do so manually.  

For things that organizations build, the story gets much more complicated. There is a plethora of tools to use, with more becoming available each month (week??). Sorting through these tools, frameworks, and platforms is not something to be taken lightly, but there are some general selection criteria that can help narrow the list to consider. Like any other technology, in general, the selection should overlap nicely with the current skills possessed by your IT group; fortunately, many of the generative AI tools available today are built with existing languages and platforms, which helps. They are generally “developer-friendly” but still require some nuance. Having a group that includes veteran engineers hosting multi-cloud will make for a longer list, but if yours is a group of junior and senior full-stack developers who only know .Net and Angular deployed to on-premises virtual machines, then you should pick a framework that works for them. Regardless of the tool, much of this originated in the realm of decision science, so you either need someone who understands things like “Cosine Similarity” or someone who is willing to learn the basics; a good tool will help abstract a lot of it.

So, what do we build then? Where do we start creating generative AI tools integrated into the organization? Fortunately, there are some general things to consider, regardless of the problem being solved, but please note these things will change as the technology matures.

Limit or abstain from customer-facing apps

A “customer” in this case means the general public, not company employees. Legislation for consumer AI governance is still catching up to reality, so venture into this space accordingly. For example, the state of Colorado just passed a law requiring AI disclosures, so you don’t want to build something only to have it retracted in a few months if it’s out of compliance.

Use Retrieval Augmented Generation

Basic chat bots have their limitations in the context of an organization but can be greatly enhanced when scoped to your data. Retrieval augmented generation (RAG) can help by increasing answer quality using data that is meaningful to your employees.

Build an Agent with Tools

Did you know that LLMs can make decisions and call external tools (or functions)? Going beyond a chat interface with a human, these tools can be used as intelligent agents to analyze input and determine which tools to call. These can be existing APIs to reuse current investments.

Human in the Loop

This concept keeps people in charge of the final decisions. Use generative AI to make some decisions which are presented to an employee who can act on it. If something is not right, they will spot it.

Test Heavily

More so than other technology, one should invest in test automation to validate generative AI builds, both on the positive and negative side; double that if you put it in front of customers. It’s not cost-effective to do so manually.

In the end, generative AI is just another tool for technologists to use to provide business solutions. If it fits the problem, include it; if not, pick another tool. New capabilities help us solve new problems. We must simply embrace the new and move forward with care.