[Read the full article at https://time.com/archive/6645018/web-crawling-4/](https://time.com/archive/6645018/web-crawling-4/)