Matt Wells 9147d6bb02 fix some diffbot crawls. ...

do not spider pages at the hopcount limit
when 'only spider urls if new' is enabled.
meaning only spider each url once. (unless there is
a temporary error)
fix malformed url bug some more.
added some commented out code for indexing spider replies
(gbss docs) for certain fatal/critical errors, in which
case they are not being indexed.

2015-12-23 13:49:21 -08:00

33 KiB

Raw Permalink Blame History

View Raw