The Dark Side of GenAI: Understanding the Technology
This is the second part of a multi-part series about the risks and challenges associated with Generative AI technology.
Before we explore the negative impacts of Generative AI technology, it’s important to define our terms. From news headlines to everyday conversations, we all (myself included) have a tendency to use the term “AI” as a shorthand that applies to many different technologies. This can be misleading if we’re not careful.
In this series, I focus on GenAI which is a one type of artificial intelligence technology. More specifically, I will discuss Large Language Models (LLMs) and similar generative tools used for images, videos, and music. If I discuss a different area of artificial intelligence at any point, I will note that explicitly.
While this might seem pedantic, the language we use when we talk about this technology matters. A lot.
When commercials and headlines call every new technology “AI,” they flatten a very broad research area, reducing thousands of different tools and use cases into a single word. When they use a term like “AI” with the general public, this tends to conjure tropes and aesthetics from science fiction. Taken together, “AI” has become a marketing buzzword applied to virtually all new technology. It evokes ideas of futurism and powerful human advancement. It implies we are on the cusp of developing fictional technologies like sentient constructs, making impersonal products feel human (or at least human-like). Many companies love to use this term broadly because they can use it to sell us an imagined ideal rather than the mundane product or service they are actually selling us.
Take, for example, the AI Dog Washing Machine. If a company announced that they developed a washing machine, complete with rotating bristles like a car wash, that you are supposed to put your dog inside, I like to think most of us would have serious questions about the safety of such a product (not to mention concerns about the mindset of the person who thought this was a product the world needs). The company’s current and potential investors would have questions about the financial viability of asking pet owners to stuff their beloved dog into an enclosed box that sprays shampoo and water—because even investors with more money than sense would recognize the bad optics here. But what happens when the company puts “AI” in the name? They’ve evoked the idea of human progress. They imply the possibility of sentience. You aren’t putting your dog inside a Whirlpool. No. You are putting them in the care of an “AI” agent that will curate a spa-like experience. They’ve softened an objectively terrible product into one that is almost tolerable to imagine. It’s marketing magic.
This is a dramatic example—seriously, please don’t put your dog in a tiny car wash machine—but once you start to notice this marketing trend, you’ll see it everywhere. Companies know that when they add “AI” or “AI-powered” to what they are selling, they are benefiting from a set of unconscious associations that can make their product or service seem intelligent and futuristic. The new iPhone isn’t loaded with useless features you don’t need. It’s “AI-powered.” It’s the next inevitable stage of human progress. Buying one is a ticket to the future. And you don’t want to get left behind, do you?
Understanding Artificial Intelligence
The artificial intelligence field is one that feels inscrutable to many, but you don’t have to be an engineer to understand what GenAI is and the industry being built around it.
Generative AI is a machine-learning model that is trained to replicate large amounts of data to produce content in the form of text, images, music, videos, code, and more. GenAI exists under a much bigger umbrella of artificial intelligence technology, and it is only one part of this broad research area.
I find these graphics helpful in visually explaining different types of artificial intelligence technology.
This image from @thatshelbs shows how the popular tool, ChatGPT, fits into the broader field of artificial intelligence. As you climb up the ladder, you can see that ChatGPT is just one type of LLM which is one type of Generative AI technology which is one example of Deep Learning and so on and so forth. Each of these bigger circles is home to other tools that have different functions and can be used to reach different goals.
This image, taken from a Gartner article about When Not to Use GenAI, shows how GenAI (in orange) is just one small part of a wider network of artificial intelligence technology. There are so many different “AI” tools in the world.
Their article notes that GenAI is often not the best tool for the job that a company or a person might be trying to accomplish. They write, “if all you have is a GenAI hammer, everything looks like a GenAI use-case nail.”
But how do you know if GenAI is the right tool for what you are trying to accomplish? First, you have to understand how it works.
How does GenAI work?
Let’s use ChatGPT as an example again since this is a very popular tool many people have heard of or have already started using. Large Language Models, or LLMS for short, are used to create a tool like ChatGPT, and they work by identifying common patterns in written works and replicating those when asked for a response by a user.
LLMs require vast collections of written works to function. This includes everything from books to newspapers to the entirety of what is written on the internet (more on the wide-scale theft used to build these tools later). When a huge collection of written work is loaded into an LLM, that becomes the dataset that it uses to generate new content. In order for this dataset to be useful, words need to be sorted and categorized by meaning and syntax. For example, if the sentence “the cat is orange” were loaded into an LLM, each of those words in that sentence would become its own token. Each token can then have additional meanings assigned to it. This is what allows the technology to differentiate words and determine their meanings. When “cat” is turned into a token, we can add extra information telling the software that a cat is a type of animal. We can be more specific and say that it is a feline. We can also tell the software that “cat” is a noun which helps the technology identify how a word like cat would be used in a sentence. This process is what people mean when they talk about training data or training the model. Portions of the training process are automated, but there is a lot of human effort that goes into categorizing the content and fine-tuning the outputs these tools produce (more on the human labor implications later).
Once an LLM has had all of its training data categorized, it will begin to make predictions based on the patterns in its datasets when prompted to do so. When you ask a tool like ChatGPT a question, the answer that it gives you works just like predictive text on your phone, just with a little more complexity. This means ChatGPT is not creating brand new content—it’s remixing what is in its datasets and deciding what to display by evaluating the probabilities of the words and sentences structures it has seen before. For example, a tool like ChatGPT is more likely to pair “peanut butter” with “jelly” than it is to pair “peanut butter” with “squid.” This is because peanut butter and jelly sandwiches are a staple of many people’s childhoods and the concept of a PBJ gets written down a lot in the materials that were used in its training datasets. There aren’t very many folks who make peanut butter and squid sandwiches and even fewer who would think to write about it. So, ChatGPT has a much lower probability of producing the phrase “peanut butter and squid” than it does “peanut butter and jelly.”
Other forms of Generative AI that create images, videos, or code work on a very similar principle. Large datasets of the type of content a tool is expected to produce—lots of images for image generators, lots of videos for video generators, etc.—are required at the outset, as well as a training process to ensure that the tool will output that content to users in the manner the developer expects.
GenAI Limitations
GenAI is marketed to us as a wonder tool that can do anything. In reality, the only thing GenAI is designed to do, on its own, is produce content when prompted based on probabilities. There are many functions people think GenAI can do that it actually can’t.
GenAI cannot predict real-world outcomes. There are other artificial intelligence technologies that can look at real-world data and make predictions that have high accuracy. For example, there are machine learning tools that are being used in the medical field to analyze large datasets of x-rays or MRI scans to identify early signs of cancer or illness. These are different tools and are not the same as the GenAI technologies that have given rise to products like Chat-GPT, Dall-E, Midjourney, etc. My day job is at an academic medical center, so I see these technologies conflated a lot. It’s really important to remember that GenAI doesn’t provide factual information. It isn’t designed to do so.
GenAI can’t think outside the box. These tools are always going to be limited by the information in their datasets. They don’t create new. They remix what they already have, whether that’s written work, images, videos, or code. These tools often plagiarize their training data too, which we’ll get into later.
GenAI doesn’t know when it’s wrong. GenAI looks at how words and sentences are most likely to be constructed based on what is present in its training data. It is designed to use probability to deliver a likely response to user prompt—not a true response. What these tools deliver are probability and not facts. It would be very unwise to use GenAI as a search engine to find accurate information or as a research tool.
Does anyone else remember that episode of Arthur where Buster learned not to trust everything that he sees on the internet?
We should apply that same caution to GenAI outputs.
When we talk about GenAI delivering probability and not facts, it’s important to remember that the training data used for these tools includes fiction novels and unvetted information from the open internet. As we all know, there are lots of ideas that are posted on the internet that simply aren’t true or that are meant to be humorous and exaggerated. People can recognize that a joke post on Reddit is not meant to be taken seriously. A tool like ChatGPT can’t do that. It’s going to output what occurs most frequently in its training data, even if that’s factually incorrect. Outputs absolutely include misinformation and much more often than you might think. GenAI is not built to evaluate fact from fiction. Outputs are a text roulette and the only way to know if something is true or not is to fact-check everything in the outputs yourself.
I’ll dive deeper into this misinformation issue in a future post, but for now, just know that these GenAI tools are very good at confidently bullshitting you and that’s about it.
There is no robot uprising
I’ve been in rooms many times where people suggest that “AI” is going to change the world, and we’re mere years away from a Jetsons-like future where robots will do everything for us. When they talk about “AI” in this context, it is clear they are talking about ChatGPT or similar GenAI tools. This is by no means an uncommon sentiment. Tech companies aren’t in the business of correcting folks on this point either. They want you to believe these tools are the next inevitable step in human evolution. After all, hype is their business model.
But when you’re out in the world and you hear remarks like this, you’ll know how these tools work, as well as their many limitations. You’ll know not to put stock into arguments that ChatGPT bots are sentient and self-actualized. When we talk about LLMs and similar tools, I much prefer the term used by researchers who have worked in this field for a long time: spicy autocorrect. The technologies that have brought us modern GenAI have existed for many decades, and we have been using simpler versions of them for a long time. So, there is no impending robot uprising to worry about. Just a world with somewhat spicier autocorrect.
Can GenAI be paired with other “AI” tools to create sophisticated software? Certainly. But I still recommend exercising heavy caution when we’re all being sold the idea that we are on the cusp of a science fiction future. I also advise weighing the risks that come with adoption of these technologies against any proposed benefits. In future posts, I will discuss these risks in detail.
We should always consider the costs of using a new technology, especially when those costs are profound, and especially when the companies selling it to us are leaning on science fiction aesthetics to make it seem, if not cool, then at least palatable.