Dutch is more expensive than English for AI: an analysis

When using AI models, it turns out that communicating in Dutch is significantly more expensive than in English. Research shows that the same 1,004 news sentences in Dutch cost up to 59 percent more tokens with models like Llama from Meta, while OpenAI's ChatGPT shows a 25 percent increase. This difference represents an additional cost for organizations deploying AI in Dutch and is the result of a measurement initiated by the Coalition for Language Equity in AI Research (CLEAR). The researchers call this phenomenon the 'tokenizer tax', a hidden tax for non-English speakers.

The higher costs for the Dutch language can be explained by the way AI models operate. These models process text in the form of tokens, which can be fragments of words, complete words, or punctuation. Most AI models have been primarily trained on English data, allowing them to often recognize English words as a single token. Dutch words, especially longer compounds, are more frequently split into multiple tokens, driving up costs. This is significant when using APIs or subscription services, where the costs per token directly impact companies' budgets.

A comprehensive analysis compared various AI models on their performance in Dutch. While there is variation in costs per model, it is clear that implementing AI in Dutch presents substantial challenges. Parts of the analysis show that a company processing €1,000 worth of English text monthly via Claude would pay approximately €1,500 for the same text in Dutch. This underscores the need for companies to map out these hidden costs when implementing AI applications.

Despite the higher costs, there are ways to address them. For instance, it might help to formulate basic instructions in English and receive the response in Dutch. It is also advisable to compare costs per task rather than solely looking at the price per token. Choosing the most competent model for Dutch text and shortening the input to be delivered are also practical steps that can help companies control costs. For most users, the communication process in Dutch is often still adequate, but business applications require careful consideration.

Read the full article from AI Insider.