<!-- mobian-agent-page publisher="time" canonical="https://time.com/archive/6644893/web-crawling/" -->

[Read the full article at https://time.com/archive/6644893/web-crawling/](https://time.com/archive/6644893/web-crawling/)
