I’m seeing a lot of noise about AI, and I want to make five points, hopefully briefly:
AI is in some ways a useful tool. It’s usefulness is way overhyped, but critics also overreact and misunderstand it.
Much of the use of AI we see today is due to it being pushed onto corporations and individuals. The true use of AI will be seen when the push is done.
Because of the overhyping and the pushing, the demand for AI is way overestimated, so we’re in a datacenter bubble. Any and all efforts to slow down or stop datacenter building are good.
When the bubble pops, a bunch of banks and corporations will be looking for a bailout. We’ve got to be ready to stop them.
Some AI is based on theft of intellectual property and there needs to be a reckoning.
This post is points (1) and (2). I’m not going to write about using AI to generate images or text for personal use or political use. That’s for another post. I’m talking about business uses of AI below:
AI is Useful But Overhyped
Given a good “training dataset”, Large Language Models (LLMs) are generally good at summarizing and “reasoning” about the content they’re fed. in my work, I’ve used AI to write code (the Internet is chock full of good examples and documentation to train AI bots), and to answer complex questions about different software products (again, they’re all well-documented). However, AI models tend to “hallucinate” (i.e., bullshit) when they’re faced with complex questions with limited training material (I’ve seen it). The key point is that “prompting” AI (asking questions) requires some domain knowledge, and evaluating the output of AI also requires more domain knowledge. Cory Doctorow calls this discernment.
The notion that AI is going to wholesale replace white-collar workers is probably overblown. Bernie Sanders, to pick one politician, takes that as a given. In the most recent New Yorker Radio Hour [Spotify link], he essentially accepts that AI is going to replace millions of workers and that legislation is required to address that. I think there will be some AI replacement, but the real legislative issue IMO is the uses of AI to do the nasty things that corporations are already doing. One example is using AI to deny insurance claims.
Being an older software engineer, I have seen wave after wave of hype. (Client-server, SAAS, Agile/Scrum, to name a few). Each of these new technologies have been sold as a silver bullet to credulous managers. They’re all tools, and in some cases useful tools, but none of them are going to solve the fundamental problem: humans have a hard time solving problems with computers. It’s inefficient and expensive. AI is another tool, a very powerful one, but a tool that is trained on human knowledge isn’t going to magically solve a long-standing problem that has a lot to do with bad communication.
AI is being Pushed, Hard
Business groupthink is real. Because the business press is full of AI hype, managers want to say they’re doing AI projects, even if AI isn’t central to the project. First, a good reality check about the true importance and usefulness of AI:
[…] Despite a global mania that has reduced the real, pressing need for digital sovereignty to the imaginary need to create "sovereign AI," no one can really articulate the case for "sovereign AI." If Donald Trump ordered Big Tech to turn off all of your country's chatbots tomorrow, nothing would change. Every one of your country's ministries and corporations would chug on with nary a hitch. Households, too, though perhaps a few of the younger members of those families would have to do their own homework again.
(Contrast this with what would transpire if Trump directed his tech giants to switch off your country's Office 365 access, or to brick your Android and iOS phones, or to killswitch your John Deere tractors. Your country would effectively cease to exist. If "digital sovereignty" means anything, it means doing something about this urgent fact):
The parenthetical point above is the key one: AI could be turned off tomorrow and businesses for the most part would be unaffected.
Similarly, there are a lot of business projects that “use AI” but AI isn’t really central. And then there are the failures.
Unfortunately, we live in a dark timeline. All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half, not only amongst projects we have been asked to participate in2, but even within projects that we have observed in passing while doing totally unrelated work. Even if you grant that AI tooling accelerates specific workloads, the method and scale of the current investments is senseless. Frequently the failure is not related to AI itself, but rather that companies are terminally bad at running software projects effectively, and as I have remarked previously, AI projects are subject to all the failure modes of normal projects plus you can get everything right and then still fail because of the method's novelty. Very few companies are so good at shipping software that they can afford the extra risk profile.
Often enough, though, it’s an actual failure in what LLMs can accomplish. The most common version of this, being rolled out across businesses around the world, is the internally-facing chatbot, or for the more daring company, the customer-facing chatbot. The story is always the same. For the former, I’ve never seen substantial internal uptake from inside a business. Employees don’t use internal chatbots because companies tend to have low-quality documentation and an LLM is not psychic – it can only know things that have been written down and made accessible. For the latter customer-facing applications, I have rarely had a pleasant experience as a consumer, with perhaps the exception of live transcription during medical appointments – hardly something worth pivoting an entire organisation around. In both cases, project leaders are very careful to avoid tracking basic metrics, such as whether the tools are being used at all, or they track metrics that are easily gamed.
That whole blog post is worth reading for a tour of what the author calls “AI psychosis”.
The next post in this series will move on to the datacenter issue.
