| Search engine listing delays have come to
| |
| | change weekly. MSNbot has apparently
|
| be called the Google Sandbox
| |
| | found that it likes how
|
| effect are actually true in practice at
| |
| | we changed the page structure to include
|
| each of four top tier
| |
| | a new feature which
|
| search engines in one form or another.
| |
| | links to questions from several other
|
| MSN, it seems has the
| |
| | article pages.Slurp gets strangely
|
| shortest indexing delay at 30 days. This
| |
| | inactive then alternately hyperactive
|
| article is the
| |
| | for periods of time. The Yahoo crawler
|
| second in a series following the spiders
| |
| | will look at 40 pages
|
| through a brand new
| |
| | one day and then 4000 the next, then
|
| web site beginning on May 11, 2005 when
| |
| | simply look at the home
|
| the site was first
| |
| | page for a few days and then jump back
|
| made live on that day under a newly
| |
| | in for 3000 pages the
|
| purchased domain name.First Case Study
| |
| | next day and back to only reviewing
|
| ArticlePreviously we looked at the first
| |
| | robots.txt for two days.
|
| 35 days and detailed the
| |
| | Consistency is not a curse suffered by
|
| crawling behavior of Googlebot, Teoma,
| |
| | Slurp. Yahoo now shows
|
| MSNbot and Slurp as
| |
| | 6 pages in their index, one an errors
|
| they traversed the pages of this new
| |
| | page and another is a
|
| site. We discovered the
| |
| | "index/of" page as we have not posted a
|
| each robot spider displays distinctly
| |
| | home page to several
|
| different behavior in
| |
| | subdomains. But Slurp has crawled easily
|
| crawling frequency and similarly
| |
| | 15,000 pages to date.Lessons learned in
|
| differing indexing patterns.For
| |
| | the first 60 days on a new site follow:1)
|
| reference, there are about 15 to 20 new
| |
| | Google crawls 250 pages on first
|
| pages added to
| |
| | discovery of links to site.
|
| the site daily, which are each linked
| |
| | Then they don't return until they find
|
| from the home page for
| |
| | more links and crawl
|
| a day. Site structure is non-traditional
| |
| | slowly. Google has failed to index new
|
| with no categories
| |
| | domain for 60 days.2) Yahoo looks for
|
| and a linking structure tied to author
| |
| | errors pages and once they find bad links
|
| pages listing their
| |
| | will crawl them ceaselessly until you
|
| articles as well as a "related articles"
| |
| | tell them to stop it.
|
| index varied by
| |
| | Then won't crawl at all for weeks until
|
| linking to relevant pages containing
| |
| | crawling heavily
|
| similar content.So let's review where we
| |
| | one day and lightly the next in random
|
| are with each spider crawling and
| |
| | fashion.3) MSNbot requires robots.txt
|
| look at pages crawled and compare pages
| |
| | files and once they decide they
|
| indexed by engine.The AskJeeves spider,
| |
| | like your site, may crawl too fast,
|
| Teoma has crawled most of the pages on
| |
| | requiring "crawl-delay"
|
| the site, yet indexes no pages 60 days
| |
| | instructions in that robots.txt file.
|
| later at this writing.
| |
| | Implement immediately.4) Bad bots can
|
| This is clearly a site aging delay
| |
| | strain resources and hit too many pages
|
| that's modeled on Google's
| |
| | too
|
| Sandbox behavior. Although the Teoma
| |
| | quickly until you tell them to stay out.
|
| spider from has
| |
| | We banned 3 bots
|
| crawled more pages on this site than any
| |
| | outright after they slammed our servers
|
| other engine over a
| |
| | for a day or two.
|
| 60 day period and appears to be tired of
| |
| | Noted "aipbot" crawled first then
|
| crawling as they've
| |
| | "BecomeBot" came along
|
| not returned since July 13 - their first
| |
| | and then "Pbot" from crawled heavily
|
| break in 60 days.In the first two days,
| |
| | looking
|
| Googlebot gobbled up 250 pages and
| |
| | for image files we don't have. Bad bots,
|
| didn't return until 60 days later, but
| |
| | stay out. Best to
|
| has not indexed even
| |
| | implement robots.txt exclusions for all
|
| a single page in 60 days since they made
| |
| | but top engines if
|
| that initial crawl.
| |
| | their crawlers strain your server
|
| But Googlebot is showing a renewed
| |
| | resources. We considered
|
| interest in crawling the
| |
| | excluding the Chinese search engine
|
| site since this crawling case study
| |
| | named when
|
| article was published
| |
| | they began crawling heavily early on. We
|
| on several high traffic sites. Now
| |
| | don't expect much
|
| Googlebot is looking at a
| |
| | traffic from China, but why exclude one
|
| few pages each day. So far no more than
| |
| | billion people?
|
| about 20 pages at a
| |
| | Especially since Google is rumored to be
|
| decidedly lackluster pace, a true
| |
| | considering a
|
| "Crawl" that will keep it
| |
| | possible purchase of as entry to Chinese
|
| occupied for years if continued that
| |
| | market.The bottom line is that we've
|
| slowly.MSNbot crawled timidly for the
| |
| | discovered all engines seem to
|
| first 45 days, looking over
| |
| | delay indexing of new domain names for
|
| 30 to 50 pages daily, but not until they
| |
| | at least thirty days.
|
| found a robots.txt
| |
| | Google so far has delayed indexing THIS
|
| file, which we'd neglected to post to
| |
| | new domain for 60
|
| the site for a week and
| |
| | days since first crawling it. AskJeeves
|
| then bobbled the ball as we changed site
| |
| | has crawled thousands
|
| structure, then
| |
| | of pages, while indexing none of them.
|
| failed to implement robots.txt in new
| |
| | MSN indexes faster than
|
| subdomains until day
| |
| | all engines but requires robots.txt
|
| 25 - and THEN MSNbot didn't return until
| |
| | file. Yahoo's Slurp crawls
|
| day 30. If little
| |
| | on again off again for 60 days, but
|
| else were discovered about initial
| |
| | indexes only six of total
|
| crawls and indexing, we
| |
| | 15,000 or more pages crawled to date.We
|
| have seen that MSNbot relies heavily on
| |
| | seem to have settled that there is a
|
| that robots.txt file
| |
| | clear indexing delay,
|
| and proper implementation of that file
| |
| | but whether this site specifically is
|
| will speed crawling.MSNbot is now
| |
| | "Sandboxed" and whether
|
| crawling with enthusiasm at anywhere
| |
| | delays apply universally is less clear.
|
| between
| |
| | Many webmasters claim
|
| 200 to 800 pages daily. As a matter of
| |
| | that they have been indexed fully within
|
| fact, we had to use
| |
| | 30 days of first
|
| a "crawl-delay" command in the
| |
| | posting a new domain. We'd love to see
|
| robots.txt file after MSNbot
| |
| | others track spiders
|
| began hitting 6 pages per second last
| |
| | through new sites following launch to
|
| week. The MSN index now
| |
| | document their results
|
| shows 4905 pages 60 days into this
| |
| | publicly so that indexing and crawling
|
| experiment. Cached pages
| |
| | behavior are proven.
|
