How to crawl the world - do we need Google?

Pre-word

This is a rather tricky question. Or, to put it better - a question worth thinking about: how much do we actually depend on search engines?

In the age of AI we are being charged pretty much for every query or MCP call. Every web search made by an agent costs tokens, API credits, or a subscription tier. And it adds up quickly, because agents search a lot. But does it need to be this way?