Michael Peter Christen 9ad1d8dde2 complete redesign of crawl queue monitoring: do not look at a ...

ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.

2012-02-02 21:33:42 +01:00

932 B

16x16px

Raw Permalink History