Commit Graph

15 Commits

Author SHA1 Message Date
0aafb70677 Turn while(1) loops into more conventional for(;;) 2016-08-21 17:35:23 +02:00
f9761e1120 constness 2016-08-19 11:49:02 +02:00
4e4e3371e6 Log function will now return void instead of a boolean 2016-08-01 18:12:10 +02:00
9c92e7d058 Fix conversion from string literal to 'char *' for Unicode 2016-05-31 11:26:01 +02:00
da566cc456 More constness in HttpMime 2016-02-16 11:34:51 +01:00
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
e7dd8f7956 replace long long with int64_t 2014-10-30 13:36:39 -06:00
f55d4d1230 merge diffbot-testing 2014-04-09 20:10:30 -07:00
9cb99f7621 Merge branch 'diffbot' into diffbot-testing
Conflicts:
	Spider.cpp
2013-12-16 11:06:11 -08:00
7b768d4b86 Merge branch 'diffbot' into diffbot-testing
Conflicts:
	iana_charset.cpp
	iana_charset.h
2013-12-12 13:01:49 -08:00
16e91375f4 bring in changes from live beta from ~/github.
limit spiders to 50, not 500 to prevent oom.
resume killed merges that had num files shrunk even
if down to one file. show collnum in spider queue.
remove back-to-back whitespace, and make all space
a ' ' for getting the doc checksum for deduping.
2013-12-12 12:58:58 -08:00
82494baa89 move CollectionRec stuff into Collectiondb files
for simplicity.
2013-12-10 15:28:04 -08:00
02bf6ab3cc new crawlbot api. not backwards compatible any more. 2013-09-17 10:25:54 -07:00
ca2a024d04 fixed up thread/spider log msgs.
fixed core from calling fprintf in
alarm signal missed quickpoll handler.
2013-08-29 21:15:42 -06:00
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00