Search engine crawlers and indexing programs are basically software programs. These
programs are extraordinarily powerful. They crawl hundreds of billions of web pages, analyze
the content of all these pages, and analyze the way all these pages link to each other. Then
they organize this into a series of databases that can respond to a user search query with a
highly tuned set of results in a few tenths of a second.
This is an amazing accomplishment, but it has its limitations. Software is very mechanical, and
it can understand only portions of most web pages. The search engine crawler analyzes the
raw HTML form of a web page. If you want to see what this looks like, you can do so by using
your browser to view the source.
programs are extraordinarily powerful. They crawl hundreds of billions of web pages, analyze
the content of all these pages, and analyze the way all these pages link to each other. Then
they organize this into a series of databases that can respond to a user search query with a
highly tuned set of results in a few tenths of a second.
This is an amazing accomplishment, but it has its limitations. Software is very mechanical, and
it can understand only portions of most web pages. The search engine crawler analyzes the
raw HTML form of a web page. If you want to see what this looks like, you can do so by using
your browser to view the source.