Search Engines > Web Directories

Whats the largest page size that Googles spider will crawl?

(1/1)

Randy:
Hello folks,

Whats the largest page size that Googles spider will crawl?

Orlova:
Google crawls websites of any size. However, very large sites tend to not be crawled all at once. It's kind of like eating a large meal. You have some now, and save some for later. Google indexes parts of large sites during it's routine spidering process. Then they'll return later on to index the rest.

JustinC:
According to Google's Documentation, files which are larger than 30MB will not be crawled. The crawlers will index up to 2.5MB of an HTML file.
This is it.
I believe that Google will increase the file size in coming few years.

Navigation

[0] Message Index

Go to full version