21/08/2024

It’s Quality Time again, and this time we talk about a very special occasion. Jasmin Nesbigall, Head of MTPE and Terminology Management at oneword, was invited by the European Commission’s Directorate-General for Translation to give a keynote speech on the development of artificial intelligence and terminology. She talks about this in the new episode of Quality Time with Sara Cantaro, Head of Marketing Management. It’s an exciting discussion about the potential applications of AI in terminology work and an insight into the working methods of the European Commission.

Jasmin Nesbigall and Sara Cantaro (Source: oneword)

Sara Cantaro (SC): Jasmin, you recently gave a keynote speech at the European Commission – how did that come about?

Jasmin Nesbigall (JN): That was certainly an exciting story! A member of the terminology department of the European Commission had seen my presentation at the last tcworld conference and then recommended me for this annual event at the Commission. So far, the keynote speeches have always been given by professors from all over Europe. As far as I know, this was the first time that a company from industry was invited.

SC: It’s a really great honour and also recognition of our work and, above all, your work.

JN: Absolutely! It’s always nice to hear that we attract attention with our expertise and are given the opportunity to show it and pass it on.

SC: You had the opportunity to present to the terminology department, a subdivision of the Directorate-General for Translation. How many people attended the event?

JN: There are around 60 people working in the terminology subdivision who attended the presentation. Most of them are trained terminologists or translators who create and maintain the European Commission’s terminology in all 24 official languages.

SC: Your presentation was about the interface between traditional terminology work and AI. You started by saying that at first glance these two areas don’t have much in common. What do you mean by that?

JN: Terminology work requires meticulousness, technical expertise and attention to detail – qualities that AI isn’t exactly known for. Terminology is very detailed and specific, whereas AI is more general and favours frequently occurring words. There’s also the risk that AI will ‘hallucinate’, i.e. generate false information. At first glance, it doesn’t seem like a good match. But in the presentation, I then showed how and where the two can work well together. The title of the presentation fitted well with this idea: Are AI and terminology friends or foes? But, even with friends, you don’t have to have everything in common.

SC: What has our experience been like so far at oneword of using AI in language processes?

JN: In one example, we’re testing whether using AI as an additional language resource – alongside existing technologies such as translation memories and traditional machine translation systems – can improve the process. One advantage of AI is that it’s much more adaptable, and specifications and requirements can be changed more flexibly. Looking at the technology is all the more exciting because AI models weren’t developed specifically for translation, but have learnt this skill ‘on the fly’ through being trained on texts in many languages. For terminology work, we’ve analysed the use of AI in a wide range of processes in the terminology lifecycle, from extracting, cleaning up and enriching terminology to providing and maintaining a terminology database. For the presentation, I focussed on three areas: extraction, systematisation and creating definitions.

SC: What were the results of your experiments?

JN: Let’s start with terminology extraction, meaning the targeted extraction of technical terms from texts. You might assume that AI would be good at this. We compared the result with a manual extraction carried out by two terminologists from our team and with a list of terminology extracted by software. The AI results still have a lot of room for improvement. Even in the best result, the Large Language Model (LLM) only extracted around a quarter of what we extracted in the manual result.

SC: That’s a rather sobering result.

JN: A quarter is, of course, better than nothing. In addition to the actual quantity, you also have to consider other factors, especially quality and time. For example, AI works at an unbeatable speed. We also need to look at the objective: if the aim is to gradually build up a list of terminology that doesn’t have to be comprehensive or if there’s simply no time for manual extraction, then AI can also be very useful for extraction in everyday practice. In our oneword blog post, we went into more detail about the whole experiment and the results.

SC: Was the use in systematisation more successful?

JN: Yes, absolutely. Systematisation means, for example, classifying terms into specific domains. The LLM did this very well. AI suggested sensible domains when it was asked to do so and, in some cases, it went beyond the ideas we had. Sometimes the system categorised incorrectly, but that happens with people too. There are simply grey areas in the systematisation because some terms cannot be clearly assigned to a specific area. The effort involved in human sorting increases as the amount of data increases, which is why using AI enables large efficiency gains, especially when you’re working with a large volume of data. The time saved was most noticeable when suggesting new terms for a generic concept or domain. Generating 50 terms as a word cloud and then simply checking them is much faster than researching those 50 terms yourself.

SC: What about the last area of application – creating definitions?

JN: We tested several prompts in our trials and compared them with a definition that was created manually. We thought that the human result would serve as the gold standard once again, but no method, not even the human one, captured all the aspects necessary for a complete definition. This is a wonderful example to show how AI can provide support and input that you hadn’t thought of yourself. In this case, the best result was achieved by putting all the results together and creating an all-encompassing definition.

SC: So would you recommend using AI in the terminology process? Or are there still too many problems?

JN: Using AI can provide support in all of the aspects we analysed. This became particularly clear during systematisation and when creating definitions. AI can make monotonous tasks much easier and, above all, can provide new ideas. However, it’s always important to check the results. This was particularly evident when it came to citing sources for definitions. None of the sources provided were actually available. During extraction, the result fell far short of the manual result. In some scenarios this may be sufficient, but, in practice, AI can only be a co-pilot and must not take the wheel.

SC: How was the presentation received by the European Commission? What did the experts take away from your presentation?

JN: The feedback was very positive. I think it’s always helpful to gain an insight into real tests and results rather than just discussing theoretical possibilities. I did a small survey at the beginning of the presentation and the majority of participants said that they hadn’t yet tested AI for terminology work. Therefore, the presentation could perhaps provide the impetus for them to try it out in their own everyday working life and take the beneficial aspects forward into their work.

SC: Thank you for the interview, Jasmin! I’m excited to see how the technology and its use in language processes will develop over the coming years.