The process: The fuzzy edges of Generative AI

Executive Summary

My latest article explores the transformative impact and challenges of Generative AI (GenAI) in business processes. I contrast traditional data querying methods, where results are consistent and predictable, with the dynamic and somewhat unpredictable nature of GenAI, which allows users to interact with data using natural language. This shift introduces a ‘fuzzy’ element to project management, requiring new approaches and expectations. Despite its uncertainties, I emphasize the need to adapt our mental models to harness GenAI effectively.

Let’s break it down

There is this thing called Generative AI. If you haven’t heard of it, congratulations! I hope you are well rested from your vacation on a deserted island. GenAI is huge, both in scale of impact and difficulty in figuring out where to start. In my previous article, I discussed how your mental model of a project has to change with these technologies. GenAI is no different in this regard. A GenAI project is more… fuzzy. This underscores the crucial need for mental model adaptation to effectively harness GenAI's potential.

Previous Behavior

Let’s use querying data as an example of something that changes in ways that can be tricky to navigate. Let’s say you want a dashboard of information on sales by region. In the past, you would have someone make a series of data queries and hook them up to your dashboard software du jour. Once you have the data vetted, and all the calculations triple-checked., the dashboard will happily give you constant results for years to come. Also, this dashboard is predictable in that if your source data changes, you can test out your queries and revalidate that everything is as it should be, potentially with automated unit tests that continuously check that things look right.

GenAI Behavior

With GenAI, there are myriad examples where you can direct a GenAI system at your data and simply ask it questions in regular English. The theory is that the days of requiring specialized queries and a scarce resource like a data engineer will be gone. The examples, at first glance, are truly inspiring. You can even upload a picture of a graph and ask questions about the data that can be inferred from the graph's appearance! These types of simple demos have ignited a frenzy in the business world to implement GenAI integrations. But, there is a catch, and it is inherent in the fundamental way that GenAI operates.

Fuzzy wuzzy was a catch

Let’s say you have figured out how to connect GenAI directly to your sales database. First off, nice work! Then you start trying things like “create a graph of my sales for the last quarter where the x-axis is time, and the y-axis is sales. Add a series for every region.”

Here is an example of “Create a graph of retail sales for the last fiscal year by quarter where the x-axis is time and the y-axis is sales.  Add a series for Amazon, Walmart, Costco and Sam's Club”

Boom! Look at that! Instant dashboard! But let’s ask another question to see where it got the data from.

WHAT!?! I have been bamboozled!

Why did this happen? GenAI systems REALLY want to provide an answer and are notorious for making up things to satisfy the query. In fact, one strategy you can use is to tell the GenAI, “It is ok to say you don’t know.” Mark, are you telling me I must worry about my AI’s feelings? Sort of! There have been very interesting studies in prompt engineering where, because of the massive source material that has been created by humans, you have to take into account how we communicate. Words like “brief” or “concise” affect response length differently.

“So I just have to get my prompt correct?”

Correct is where the fuzzy comes in. Not only do results get made up (hallucinations), but results change. On the aforementioned traditional database query, you can repeatedly hit the run button, and the results don’t change unless the data changes. With GenAI, everything can be static (prompt and underlying data), and the results can differ for each prompt run. Many techniques make this problem… well, less of a problem. One-shot versus few-shot queries, for instance. Remember that this is fundamental to how GenAI creates responses. The underlying technology is built upon probability. Anyone who has played roulette in a casino knows that sometimes probability goes your way, and sometimes it does not.

“I am due!” said every person on a losing streak in Las Vegas

There are more parallels to GenAI in the roulette example! Above most roulette tables, a sign shows the last few results. This tactic is genius! This sign induces even people like me who understand probability and math to start thinking, “Man…., the last six numbers have been black…. It has to be red this time!” Now, change the phrase to “Man…, the last ten thousand runs of this query have been perfect…. the next run is going to be fine!”

Say it with me… “Past results do not predict future results…”

Past results do not predict future results! Now, you might be thinking to yourself, “This Mark guy is supposed to be all about emerging technology; the dude is kind of a bummer.” The truth is, in fact, the opposite! All of these technologies are only beginning to reveal how powerful they are. Modifying your strategic thinking is the key to unlocking GenAI for your business. This is not just a new software package you need to deploy. The fundamentals are different, and you approach them using previous thinking at your peril.

Next is a discussion on how costing differs with GenAI using a retail example.

BONUS LEADERSHIP TIP

When you feel the siren song of GenAI luring you into complacency. Think about the following exchange:

“SIMPSON! GET IN HERE!”

“Yes, boss?”

“Why did our GenAI inventory system just send six truckloads of dishwasher soap to one store in Boise?!?!”

“Let me re-run the analysis. Hmm, it looks right to me now. The system says that they needed one pallet of dishwasher soap.”

“Look, Simpson, I just got off the phone with the store manager, and he sent me a photo of six trucks parked outside the store!?! Why did the inventory system do this?”

“I don’t know, but the results look good now, sir.”

P.S. The featured image was found by searching for “fuzzy”. It is a pretty apt comparison. Very strong and yes…fuzzy, but approach with caution.