Hordes of automated bots run the internet – can we bring them to heel?

Roaming chunks of code, pre-programmed to perform all manner of tasks, shape the web and our experience of it. What are they up to? And as they get smarter, how can we regain control?

By Edd Gent

THE internet is quite popular. The best estimates suggest that by the end of 2019, some 4.1 billion people were online – over half the world’s population. That is hardly news. What’s more surprising, perhaps, is that human users are almost outnumbered by non-human ones. Governed by coded instructions, these bots creep around in the background, largely out of sight, browsing websites, clicking links, downloading content and typing text. What are they up to?

We need to know. Although for the most part we still outsmart these free-roaming bots, they are growing more intelligent and destructive, capable of destabilising everything from financial markets to public debate and even our shared sources of knowledge. Only by understanding their secret, complex ecosystem can we hope to tame them.

If you have been online, it is almost impossible for your experience not to have been shaped by bots. These automated chunks of code, preprogrammed to perform a certain task over and over again, account for as much as 39 per cent of activity on the web, says Matthew Prince, CEO of Cloudflare, a firm that helps companies securely manage web traffic. As much as 1 per cent of that comes from Google’s army of web crawlers, known as spiders, which scuttle around websites plucking out information like text and links that determine where pages appear in search results. Every search engine, from Bing to Baidu, has its own spider armies too.

Other bots help to maintain order on the web. Monitoring bots are digital meerkats on constant alert, periodically checking websites for bugs, poor performance and outages. Moderator bots live inside social …