Commit Graph

14 Commits

Author SHA1 Message Date
e9eee473dc Remove diffbot specific code. -> maxcrawl, maxprocess, crawl pattern, crawl done notification, etc... 2016-07-22 13:59:55 +02:00
ab0b9d03ea Standardize header guards 2016-03-08 22:16:02 +01:00
6b6e54a217 Remove "spider proxy test url" which will call a preconfigured url by default
every 10 seconds when proxy is enabled (inside hdtempwrapper)
2016-03-02 20:25:20 +01:00
f6207b75c7 Merged 6e12f96aeab29778b3b6ba64c5fe594e7e6989cd 2015-11-20 12:01:09 +01:00
95e3a760e9 proxy fixes 2015-03-05 11:10:40 -08:00
6fc83566e2 more fixes 2015-02-02 14:06:38 -08:00
c15bd53e52 added support for supplying basic proxy authorization
to spider proxies. username:password@1.2.3.4:80
2015-02-02 13:23:38 -08:00
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
b4ca812ef8 added parm to reset proxy stats in table. erases
all our knowledge/stats for each proxy.
2014-09-30 17:38:59 -07:00
3f2dcda4e1 got new floater/proxy logic compiling. 2014-06-06 15:11:51 -07:00
806cf79b73 spider proxy updates 2014-06-02 13:18:18 -07:00
0b9b77ea46 call buildProxyTable when ips updated 2014-06-02 10:20:29 -07:00
a811462d5f spider proxy stuff compiles now 2014-05-30 15:05:00 -07:00
f341dba0c8 got the general framework for load-balanced/reliabled
floaters in place for the distributed spider network.
need to fill in the blanks now.
2014-05-27 15:21:12 -07:00