Transcript
Speaker 1
This is the Cherryleaf Podcast. Hello and welcome to the Cherryleaf podcast. We’re going to do a different type of podcast today. My name is Ellis Pratt. And on the line from Brighton is my colleague Ginny, Ginny Critcher. Hello.
And the plan is I wrote a script for this podcast and we thought it would be a good idea to have a discussion between us two about the topic and it might be that some questions come up or that there’s some thoughts. We’ll give it a go and see what happens. We’ll go back to looking at AI, technical writing. Where’s the return on investment and is it all hype? And let me go through reading some of it if you want to interrupt at any point, just shout and share your thoughts.
Speaker 2
OK.
Speaker 1
I don’t know if you remember the excitement when ChatGPT burst onto the scene, suddenly it seemed like AI was going to revolutionise everything, including technical writing. Well, roughly 18 months into the world of modern chatbots and we’ve done six podcast episodes on AI since then in that period, and we’re today asking the question, was it all just hype?
And you might have seen in the press and in social media some people saying that AI isn’t delivering the promised return on investment. And yet others are saying it’s completely changed the world. It’s changing the game.
What’s going on? What’s the real story?
We’re going to look at this topic, these claims through the lens of technical writing. What we’ll look at is we’ll examine the recent claims about AI’s lack of return investment, explore how these claims stack up relating to the world of technical writing and look at the potential benefits and pitfalls of artificial intelligence, or the work that technical writers do, and discuss what factors might influence whether AI is worth the investment. So let’s figure out where AI really stands in the world of technical documentation.
There was a report from Goldman Sachs that was around on social media a few weeks ago. I don’t know if you saw it, Ginny, did it come across? It was called. You’re shaking your head.
“Gen AI: Too much spend, too little benefit.”
And it was published in June 2024. It’s quite a long report. I’ll read a couple of bits from it:
“Tech giants and beyond are set to spend over $1tn on AI capex in coming years, with so far little to show for it. So, will this large spend ever pay off?
MIT’s Daron Acemoglu and GS’ Jim Covello are skeptical, with Acemoglu seeing only limited US economic upside from AI over the next decade and Covello arguing that the technology isn’t designed to solve the complex problems that would justify the costs, which might not decline as many expect.
But GS’ Joseph Briggs, Kash Rangan, and Eric Sheridan remain more optimistic about AI’s economic potential and its ability to ultimately generate returns beyond the current “picks and shovels” phase, even if AI’s “killer application” has yet to emerge.
But despite these concerns and constraints, we still see room for the AI theme to run, either because AI starts to deliver on its promise, or because bubbles take a long time to burst.”
We’ll include a link in the show notes to that.
I’ve used the phrase “picks and shovels” before. I should double check, is that something that makes sense to you, Ginny?
Speaker 2
No, not really. I was going to ask you about that. Yeah. Can you elaborate a little bit?
Speaker 1
The phrase “picks and shovels” comes from the gold mining era and the gold rush in the various people went to California and Australia with the idea of finding their fortune in discovering nuggets of golds. And there were people who were selling the picks and the shovels for them to dig out of the ground to find it and it said that the only people that actually made money from the gold rush were the people selling the picks and the shovels are from the people actually doing the prospecting. And so there’s this concept that you don’t necessarily have to make money directly, but if you can provide assistance or tools or things that help people that are trying to find the nuggets of gold you might actually become quite wealthy in that way, and that’s basically the concept of it.
But what I don’t understand is when they use picks and shovels quite what in the AI sense what it sees as that. Maybe it’s the large language models that will power a killer application and no one’s actually built that fantastic application that everyone uses day-to-day yet.
Speaker 2
Yeah. I mean, it’s interesting just from talking to people in various industries, not necessarily technical writing industries. My research has shown that people are using it, just people on the ground. Regular people are using things like ChatGPT, or whatever and it is being used to generate content. But it does seem to be at quite a if I say low level. Does that make sense?
I guess if somebody’s got a report to write, something like that. They will put in the parameters and get a document generated which they can then work with. From the people I spoke to, that seemed to go across various industries. People were just using it to accelerate things getting started. It would generate content quite quickly. And as we’ve said all along, this content then needs to be looked at by human and you have to go through it and check for errors. And you also have to check the veracity of it, but I guess kind of chimes with it. This isn’t a killer app, is it? It’s not actually changing the way we work. I think it’s making things a little bit faster.
Speaker 1
Let’s go through this a bit more, because that comes up a little bit later. and as a company at the moment, the things like our AI course, Using generative AI in technical writing, that is sort of the picks and shovels type offering, it’s giving people the skills to go out and use AI themselves. Somebody else that wrote about this was…
Speaker 2
Absolutely.
Speaker 1
Somebody called Ed Zitron, who has a website called wheresyourhead.at and again, we’ll include a link to it, and he writes in reply to this article from Goldman Sachs was:
“While one might argue that it’ll mean faster generative processes, there really is no barometer for what “better” looks like, and perhaps that’s why ChatGPT, Claude and other LLMs have yet to take a leap beyond being able to generate stuff.
How does GPT – a transformer-based model that generates answers probabilistically
Speaker 2
Probabilistically.
Speaker 1
(as in what the next part of the generation is most likely to be the correct one) based entirely on training data – do anything more than generate paragraphs of occasionally-accurate text?
The reason I’m suddenly bringing up superintelligences — or AGI (artificial general intelligence) — is because throughout every defense of generative AI is a deliberate attempt to get around the problem that generative AI doesn’t really automate many tasks. While it’s good at generating answers or creating things based on a request, there’s no real interaction with the task, or the person giving it the task, or consideration of what the task needs at all — just the abstraction of “thing said” to “output generated.”
Speaker 1
So AI is segmented. You have generative AI, which is chatbots that generate words and then there’s AGI, which is this promise of robots that go around and do things for us. And something the two connected and one being a step to the other and others see them as distinct, and they’ll go on their own paths.
Speaker 2
Things will progress, though, won’t they? It’s like anything. And if you look back to… Well, you could go back to the dark ages of pre-Internet or let’s take mobile phones in the early 90s, we were starting to use our for the brick phones or late 80s and then look at us now 30 years later, we speak through screens and things are coming in leaps and bounds. That’s going to happen with AGI.
Speaker 1
If you go back to what you said about fans, we can predict that this is going to happen, although this happen. But until you actually experience it, you really don’t know what it’s like until you do it. We have this with the crash in 2008 where you have the news reports of banks crashing left, right and centre. You see all this is going to be a problem, but you don’t really know what it’s like until you go to a cash machine and you find that there isn’t any just coming out of it cause your bank’s gone bust or the fear that that it can be hard to really appreciate what the future will be like.
Speaker 2
Yeah, absolutely. I don’t know for older listeners saying I remember a programme in the ’70s called Tomorrow’s World, which was always looking at scientific innovation and how the future might look quite funny when you look back on it at how the future was thought to look. And of course, it looks completely different because, yes, you, you’re essentially predicting something you don’t know. And who knows when the Internet was first a thing. It’s a force for both good and bad. Did we have any idea what a Pandora’s box we were opening? No, we did not have predicted.
Speaker 1
Certainly not with social media.
Speaker 2
You can look back at your history and see that great inventions always have been bad. Generally it’s just very hard to predict how things look. People may look back, at least in 30 years at this podcast and just say how little they knew.
Speaker 1
So let’s look at the other criticisms you might see of which there are seven.
One is high implementation costs without proportional benefits. Development and integration of AI and chatbot systems can be expensive, requiring significant upfront investments for some organisations, costs might outweigh the perceived benefits in the short term, although it’s more specifically about building your own large language model or chatbot. If you’re building something on top of Claude or ChatGPT, the cost is tiny, $20, $30. It can be really.
Second criticism inaccurate or inconsistent outputs, requiring extensive human oversight.
Third maintenance and training and learning systems require continuous maintenance and training to remain effective. This is again about the large language models. The ongoing investment can be a financial burden, particularly for smaller companies. We’ve seen these large language models really being developed by companies like Meta/Facebook, Google, or companies that are backed by other big companies. OpenAI is heavily backed by Microsoft.
Now the fourth, user experience issues. Not all AI and chatbot implementations successful when AI systems fall short. User frustration can go up taking customer satisfaction and return investment along with it.
Speaker 2
I think that’s quite a big one, actually. I’d be interested to hear people who have had a good user experience with chatbots and AI that will be interesting to hear who’s getting it.
Speaker 1
Yeah. Yeah, consumers, a solution looking for a problem there, waiting for that killer app.
Difficulty integrating ROI tools with existing workflows and systems. So you have this chatbot, how do you get it to work in your accounting system? Where does it fit in?
Number six, limited understanding of complex technical concepts by AI system. And certain user preference for human generated content. In technical documentation you see software applications where there will be the option of a chatbot, but I think people are being burnt by trying to support lines for queries or buying tickets for cinemas and for it not being very good.
Speaker 2
Yeah, and that all adds to the user frustration, doesn’t it? Well, higher the user frustrations.
Speaker 1
So next thing that I wrote this analysis in the context of technical writing for examining these return on investment claims in the context of technical writing is important to consider the unique challenges and requirements for the field.
First off, being accuracy and precision. Technical documentation demands a high level of accuracy and it can generate content quickly, but it might struggle with nuanced technical details that might lead to errors, which then have to be corrected by human beings. And as you were saying earlier, often if you’re using a layer to generate content, you get a good first draft you get a useful intern electronic intern, but it needs to be reviewed.
Second thing, consistency across documentation AI tools can help maintain consistency in terminology and style, but they might struggle with complex cross referencing and maintaining coherence across large documentation sets.
Speaker 2
Yeah, I don’t personally have experience with that, but I understand how that would be quite complicated.
Speaker 1
Third adaptation to rapid changes. With fast-paced world of IT products and APIs evolve at breakneck speed. Keeping the other systems up to date with these changes can be a constant and costly battle. AI systems might require frequent retraining to keep up with these changes, which can be resource intensive.
Fourth handling of specialist knowledge. Many technical writing projects involve highly specialised subject matter. Current AI systems might lack the depth of understanding required to generate accurate content in those areas without significant human input.
Speaker 1
The next thing that I wrote are articles responding to addressing the concerns.
So first off, generative AI’s poor handling of specialist knowledge. It’s true, it’s true for large language models. But what we can do is use techniques where you tell the chatbot or large language model AI system to only look at our source content, work with that, only find the answer from that material and that alone. We’ll talk about that a little bit later.
Speaker 2
So it’s quite interesting in the ownership of that content.
Speaker 1
AI’s killer application has yet to emerge again. This is true. We are still waiting for a killer app, particularly within the world of technical writing. I think what will have a big impact is what’s called agentic AI. And that is where you give an AI system a big task and then what the AI system does is it doesn’t just answer your question, it starts to work out what are the steps and many answers that need to be resolved before you get the answer to the big answer. So it breaks down complex jobs into small steps, figures out what needs to be done and then goes off and does it on its own.
Speaker 2
So it does my job, Ellis, is that what you’re saying?
Speaker 1
That’s the bit when you share an AI system that can do something close to that, they start to go pale. But we’re not there yet. And if you asked an agentic AI system to plan your documentation, in theory it might be able to go and research the users, look at the UI screens, create an outline without human intervention. Without you having to break it down into all of those steps. But because of the memory limitations that AI has and the complexity of what we’re asking it to do, it’s not there. But if we can get some of that, that will be a big productivity bonus for technical writers.
Another criticism like of our generative AI is creators for some developers of large language models. Their objective is probably more than just providing a chatbots at the moment. Google dominates the search sector, searching for 90% of searches on Google and they make billions of dollars from selling adverts. If the likes of Microsoft and Facebook can get people to use chatbots to find the answers, instead of going to Google search then that’s good news for Microsoft and Facebook. It makes Google less rich and it makes advertising on LinkedIn, which is owned by Microsoft, advertising on Facebook, which is owned by Meta, more attractive than just thinking about sticking your adverts on Google. So there was another agenda that could be out in play as trying to weaken Google’s dominance for which they’re prepared to spend some money on chatbots without necessarily getting an immediate return or direct return.
Also another argument: A lot of the AI will be hidden. We are using Microsoft Teams for this conversation. At the end of the call, we can get a transcription of this meeting that was set and you can do the same with Word 365. And that transcription is generated using AI. Now within Teams and within Word 365 it’s just a button on the menu. Push that, get a transcription. You don’t even know that AI is working in the background to work out which is the appropriate word for what is being said. That means for a lot of people won’t be aware that AI is being used.
Speaker 1
And there are other people that claim that there are benefits from AI at the moment. One post is very interesting. In the AI field is Ethan Mollick. And his wife is also heavily involved in AI, and he wrote
There was a large survey of 100,000 workers in Denmark six months ago. They found workers see a large productivity potential of ChatGPT in their occupations, estimating it can half working times in 37% of the job tasks of the typical worker.
There is, I think, one of the Gulf states. It’s training all of the workers in that particular country on how to use chatbots so they can be more productive with it and on The Rest is Money podcast Robert Peston was arguing that that type of training should be happening in the UK as well. We should be training my children training and staff on what chatbots can do. And by that way, they will discover how to use them to be more productive.
Speaker 2
Absolutely. Yeah, definitely.
Speaker 1
So partly, it’s whether we see a benefit or whether those benefits are hidden behind the scenes.
OK, so let’s move on to this.
In terms of technical writing, the promise of AI and chatbots for technical writers. Despite these challenges, AI and chatbots offer several potential benefits to technical writers. They offer the potential to automate repetitive tasks, streamline the creation of help content, and provide real-time assistance to users navigating complex applications or APIs. And we’re seeing technical writers have begun incorporating AI and chatbot technologies in their workflow already.
And as you said, Ginny, it’s hard to see the situation. Will we go back to not having chatbots not having AI?
For example, recently MadCap announced that the tool IXIA CMS with the version 7.2 is going to have an AI writing system that’s been part of OxygenXML for a while. And with that you can use it whilst you’re writing. It can help you generate XML, short descriptions or data short descriptions. It can summarise text for you. Correct grammar. You can write code or regular expressions. You can use it to improve content. It can help you enforce style guides that is identifying deviations and suggest corrections. And we can help you ensure a uniform voice. It can help enhance readability across all of the content. And they can also suggest improvements to improve readability, validate DITA or other markup schema. And it can also act as an editor because AI is good at spotting patterns and content, particularly large amounts of content where we as humans struggle to deal with all of that content and spot inconsistencies.
So, what MadCap is claiming is you can use it to rewrite content, rephrase text to simplify complex language, adapt text for different contexts, or enhance writing impact. You can use it to provide feedback as an intelligent editor; it can offer constructive feedback on grammar, style, tone, and structure. So, it can help writers refine their work, making it better. It can also help in terms of search SEO semantics, which is to find, define, and add metadata, ensuring the metadata is relevant and variations of the same tone are identified. This can help users search engines for relevant content. The claim is that it can help with content reuse, buying, and proposing duplicated content for reuse, avoiding the creation of unnecessary topics unlike existing content. From some experiments I’ve done, it can be quite good where you have bits of content that are similarly worded and phrased but not exactly the same, allowing you to spot and pilot those.
Speaker 2
That’s interesting, but did you do much experimentation?
Speaker 1
I took some old policies and procedures from the London Fire Brigade, which were in a PDF, and I asked, I think it was called ChatGPT, to identify any sections that are repeated in this document. There’s nothing that’s exactly the same, but there are sections.
Speaker 2
Yeah.
Speaker 2
Mm-hmm.
Speaker 1
Just similar, talking about it, and they could probably be combined or merged. It’s quite interesting that it was clever enough to respond in that way.
Speaker 2
Yeah.
Speaker 1
Since ChatGPT was released, we’ve seen that generative AI is good at doing routine and repetitive tasks, allowing technical writers to focus on the more complex aspects of documentation. It’s also good at data-driven insights. We know that not many technical writers use data to affect how they work. They don’t necessarily get those related data on how people behave.
Speaker 2
Sorry, can you expand on that?
Speaker 1
If we publish something, we might know how many people read it, but we don’t know if they follow it, whether they follow it correctly, or if they misunderstand and make mistakes. Whether they open a page, find it boring, and then close it, so I don’t know if they’re actually reading it or just opening it. We rarely have statistics on the quality of content in terms of its effectiveness. We can tell if it has errors, we can tell if it’s consistent, but in terms of its usability for users, we’re limited in what we can know.
Speaker 2
Yeah, unless you get user feedback, which is again dependent on the user actually giving the feedback.
Speaker 1
Yeah.
Speaker 1
If you look at shopping sites, you can tell whether somebody clicks on the button and puts something in the shopping cart. You can tell whether they buy that shopping cart, and whether they buy it again in six months. With that type of situation, you’ve got much more data because you can change a button and see if it affects whether people buy more or less, and so on. It’s harder to do that with technical writing.
Speaker 1
Mm-hmm.
Speaker 1
In terms of user experience, chatbots and AI-powered search can help users find relevant information more quickly from a range of sources, enhancing the overall value of the documentation. Chatbots can provide immediate responses to user queries, which is particularly useful for API documentation where developers often want quick, precise answers.
OK, so let’s move on to factors affecting the return on investment for technical writers.
Several factors can affect whether using generative AI will be worth the time, money, and effort. One is that you need to learn and understand where chatbots can be effective in your workflow. The answer is to start small, looking at specific tasks that take your time and seeing if AI can help, like spell-checking, drafting, and routine updates, rather than trying to overhaul an entire documentation process overnight.
Going back to the point we mentioned earlier, for full ROI in generative AI, chatbots need to be trained on high-quality domain-specific data. While the AI models are trained on all the content on the Internet, the most common way to get them to focus and find only the answer from your documentation sets is to use a technique called retrieval-augmented generation (RAG).
This essentially means adapting the AI and giving it a custom knowledge base. Imagine a library; instead of getting the AI system to browse the entire library to find the answer to your question, RAG is like giving the AI a command to look only on a specific shelf of books. With the right system, you feed your old documentation into the system, and when you ask it a question, it’s not searching the larger model or the Internet for the answer—it’s only looking at your content.
This way, you get answers that are relevant to your products or context rather than generic information that might not apply to your situation.
It also means you can look at content that’s up-to-date rather than content that was put into the AI system six months or a year ago. This means if you have good source content to give to the AI system, you’ll get better results. So, the old adage of “garbage in, garbage out” still applies. You need to feed the system good quality content to get good answers out.
Speaker 2
So, you still have to have that content in the first place, don’t you?
Speaker 1
That’s right, which suggests that there will still be a need for people like you and me to do the work to create the content in the first place.
Speaker 2
Yeah, there’s certainly the initial process of putting in that information or having that content.
Speaker 1
At the moment, if the AI can look at a computer screen and work out all the steps to use that computer screen by itself, then potentially it can do it. But it would still probably need help, and at the moment, it’s not really capable of doing that. It can do some of it, but not all of it. The success of ROI and chatbots largely depends on how they’re integrated into existing systems. A well-planned implementation that complements rather than replaces human expertise is more likely to yield a positive return on investment.
Large organisations with more resources for implementing and maintaining a chatbot or an AI system are in a better position to achieve a more positive ROI. But when it comes to highly specialized or complex topics, AI systems might struggle to add value without a human expert’s guidance, again highlighting the “garbage in, garbage out” principle.
Speaker 2
They’ll learn, won’t they? The AI systems learn.
Speaker 1
They will get better, yes, but for very technical and complex content like what we’re involved with, they might still need a guiding hand from a human.
Speaker 1
Can we improve and get better?
Speaker 1
And the final point on that is user acceptance.
The willingness of both technical writers and end users to engage with and use AI-generated content and chatbot interfaces will affect the return on investment. Now, onto the risk factors. There is a risk that using chatbots in technical communication might go wrong. Just throwing up a chatbot to your knowledge base might not be the best idea. You might need to improve the source content beforehand. If it’s poor quality content that the AI is using, it’s going to give users poor quality answers.
Speaker 2
And actually, that’s potentially detrimental, isn’t it? Because user frustration will go through the roof.
Speaker 1
Yes. The loss of human touch is another risk. Technical documentation often benefits from the writer’s ability to anticipate users’ needs and explain complex concepts in relatable terms. AI might struggle to replicate that. AI-generated text can be very wordy, as it’s trained on the literature and tends to emphasise eloquence over succinctness.
Speaker 2
Presumably, you can train it to be more succinct, and with the parameters that you put in or the prompts, you can specify using plain English techniques, etc.
Speaker 1
We cover this in the training course. You can give it guidelines and standards and tell it to write to a particular standard, which will help. This is part of the change: if you know how to use an AI in a sophisticated way, you can get great answers. But if you ask it simple questions or give it simple instructions, you won’t get as good quality results.
Another risk is that technical writers might become overly dependent on AI, leading to a decline in critical thinking and problem-solving skills. You need to be an expert in technical writing to check that what’s created by an AI system makes sense. It tends to be a good intern that you can use rather than a replacement for a technical writer.
Another risk is it might become a graveyard of unmaintained content. Thinking that just throwing up the AI is the job done and never updating the content, allowing it to rot.
The fifth potential risk factor is that technical writers might lack the skills to effectively use and manage these tools, so things like our training course can help with that.
In conclusion, where does this leave us in the AI and technical writing debate?
Firstly, the return on investment isn’t guaranteed. You need to implement chatbots smartly. You need quality data and a clear understanding of where AI can truly add value.
Secondly, it is not a magic bullet, but it’s not just hype either. It is a powerful tool that’s reshaping our field.
The key to success will be balance: blending AI’s efficiency with human expertise and insights. For technical writers, AI isn’t a replacement; it’s an enhancer. It can handle the dull, repetitive writing tasks, giving us time to do more complex, valuable problem-solving tasks focused on user needs. By focusing on areas where it can add value, such as improving content discoverability or routine updates, teams can achieve returns on investment by delivering higher quality, more accurate content.
Thus, the future of technical writing will likely be a collaboration between humans, technical writers, and AI systems, both bringing strengths to the table. As time goes on, the relationship between AI and technical writers is likely to grow. If we’re able to use it and guide it wisely, we’ll need to be adaptable, embracing AI where it helps but not losing sight of the need for the human touch. We can’t just throw AI at something and expect a decent result.
One could argue that AI in technical writing isn’t just about the return on investment; it’s about rethinking how we create, manage, and deliver documentation. That’s the article that responds to the feeling that there’s hype and no return on investment, trying to get some sense of balance.
Speaker 2
I agree with those conclusions. I think it’s currently an enhancer, and that’s the way to look at it. Embrace it; it’s not going anywhere.
Speaker 2
Learn as much as you can about it. Incorporate it where you can. It’s here to stay, and it’s only going to get better.
Speaker 1
Any questions or topics that I didn’t answer from this, do you think?
Speaker 2
I’d be interested to hear from our listeners if they are using AI and, if so, how they are using it, what their thoughts are. Is it helping them? Is it not? Is it a load of hype for them?
Speaker 1
Yes, I hope at the end of this article to suggest that people contact us. There is a problem with technical writers often being the only one in an organisation, lacking the ability to connect and check what’s going on elsewhere. Our newsletter is popular because it disseminates information and shares knowledge, addressing this need.
Speaker 2
Yes, many of us work remotely or are perhaps the only technical author in a team. This hopefully brings people together. If we can share our knowledge and understand where our community is at, that would be really helpful. I’d be really interested to hear from people how they are finding AI. How are they using it? Does it help?
Speaker 1
The best way to do that is by our e-mail info@cherryleaf.com.
I think we can wrap it up there. I think we’ve come to a good conclusion.
Speaker 1
OK.
Speaker 2
It’s a fascinating topic. It really is fascinating. I wish I had a crystal ball to look into the future, 10 years, 20 years, or even five years into the future. Where will we be? What will we be doing? What is our Tomorrow’s World?
Links
Goldman Sachs
Ed Zitron
https://www.wheresyoured.at/pop-culture/
Ethan Mollick:
https://x.com/emollick/status/1815563452640854093
MadCap Software
https://www.madcapsoftware.com/blog/madcap-ixaccms-ai-positron/
Leave a Reply