|
e9eee473dc
|
Remove diffbot specific code. -> maxcrawl, maxprocess, crawl pattern, crawl done notification, etc...
|
2016-07-22 13:59:55 +02:00 |
|
|
ab0b9d03ea
|
Standardize header guards
|
2016-03-08 22:16:02 +01:00 |
|
|
6b6e54a217
|
Remove "spider proxy test url" which will call a preconfigured url by default
every 10 seconds when proxy is enabled (inside hdtempwrapper)
|
2016-03-02 20:25:20 +01:00 |
|
|
f6207b75c7
|
Merged 6e12f96aeab29778b3b6ba64c5fe594e7e6989cd
|
2015-11-20 12:01:09 +01:00 |
|
|
95e3a760e9
|
proxy fixes
|
2015-03-05 11:10:40 -08:00 |
|
|
6fc83566e2
|
more fixes
|
2015-02-02 14:06:38 -08:00 |
|
|
c15bd53e52
|
added support for supplying basic proxy authorization
to spider proxies. username:password@1.2.3.4:80
|
2015-02-02 13:23:38 -08:00 |
|
|
96b8197ad3
|
now it compiles with -m32
|
2014-11-10 14:45:11 -08:00 |
|
|
b4ca812ef8
|
added parm to reset proxy stats in table. erases
all our knowledge/stats for each proxy.
|
2014-09-30 17:38:59 -07:00 |
|
|
3f2dcda4e1
|
got new floater/proxy logic compiling.
|
2014-06-06 15:11:51 -07:00 |
|
|
806cf79b73
|
spider proxy updates
|
2014-06-02 13:18:18 -07:00 |
|
|
0b9b77ea46
|
call buildProxyTable when ips updated
|
2014-06-02 10:20:29 -07:00 |
|
|
a811462d5f
|
spider proxy stuff compiles now
|
2014-05-30 15:05:00 -07:00 |
|
|
f341dba0c8
|
got the general framework for load-balanced/reliabled
floaters in place for the distributed spider network.
need to fill in the blanks now.
|
2014-05-27 15:21:12 -07:00 |
|