f1b6f73719
empty &seeds= fix to not reset seeds
Matt Wells2016-11-16 10:34:25 -08:00
53ee1039b8
import final hung crawl fix into old gb. xor the firstip into the doledb key this time. seems to avoid all collisions now so we don't overwrite nodes in the doledb tree.
Matt Wells2016-11-07 09:11:27 -08:00
3d248732d0
fix to shut up app checker.
Matt2016-11-04 17:28:26 -06:00
c0b2cdb60a
hide the verify disk writes parm, seems to be causing cores when activated. and shouldn't really need to be used. is for debugging disk issues.
Matt2016-11-04 17:09:15 -06:00
93d5752ab7
import fix for jobs hanging from pro.
Matt Wells2016-11-01 11:18:40 -07:00
1542f7c57f
do not save doledb on exit to prevent corruption being propagated and in case we change default spider priorities in the url filters code which could cause hung jobs.
Matt Wells2016-10-24 14:23:13 -07:00
d5bc775ce5
fix errCount bug in url filters from errCount wrapping over to negative numbers.
Matt2016-09-20 17:02:51 -06:00
f23daf3e5e
more fixes for 'zeroing out' code.
Matt Wells2016-09-14 10:32:01 -07:00
93f878fbca
wow this thing is really being persnickety
Matt Wells2016-09-14 10:18:48 -07:00
8a5891f2ec
fix memory leak in Parms.cpp
Matt2016-09-12 13:41:07 -06:00
e9f0d067be
zero out docs that we do not process to save disk space. record content hash in 'zeroed out' content so we preserve it for deduping.
Matt Wells2016-08-30 11:06:59 -07:00
616bfeea86
show corrupt collection numbers for spiderdb corrupted recs.
Matt Wells2016-08-15 12:55:22 -07:00
7e77b900a6
fix a core from corrupted spider request in doledb in rdb::reclaimMemFromDeletedTreeNodes()
Matt Wells2016-06-21 11:54:09 -07:00
b03571c1a1
fix infinite loop from corrupt spider request
Matt Wells2016-06-06 08:06:20 -07:00
14e2f1f579
make trash subdir in case missing.
Matt Wells2016-05-30 19:37:03 -07:00
bd5618f1b7
fix spider request corruption detection in doledb
Matt Wells2016-05-28 08:56:36 -07:00
fe21fb1cc6
fix spider detection of corrupted requests. fix deduping so it doesn't core on docid based spider requests.
Matt Wells2016-05-28 08:40:59 -07:00
7bd2344f41
increase regular page download timeout from 30 seconds to 60 seconds to accommodate some slower websites.
Matt Wells2016-05-18 10:05:02 -07:00
494f7ca645
reduce diffbot timeout from 18000 secs to 240 secs
Matt Wells2016-05-18 09:55:15 -07:00
b0e015b97d
fix dns lookup bug that was causing us to get incorrect ips sometimes.
Matt Wells2016-05-17 11:57:21 -07:00
39e621f655
trash files of length 0 that are holding up a merge. if we can't merge files we end up stockpiling them and things get slow fast.
Matt Wells2016-05-13 13:21:43 -07:00
dfca68ec46
Merge pull request #99 from vonbetz/regexdebug
Gigablast2016-05-13 10:06:24 -06:00
c89d963d46
Merge pull request #100 from vonbetz/tldfixes
Gigablast2016-05-13 10:01:01 -06:00
d4dc85bf18
Fixes for new tlds. They can now contain '-' and numbers. Fix punycode url encoding: set max length before encoding each url chunk.
Zak Betz2016-05-12 16:04:21 -06:00
0ee1e6164c
Add input validation to regexs before crawlbot collections are created. Add ./gb egrep command to test regexes.
Zak Betz2016-05-11 14:03:45 -06:00
a246289238
Merge pull request #96 from vonbetz/diffbot-testing
Gigablast2016-05-11 09:32:21 -06:00
89f3344be5
Updated tld list with the most current list.
Zak Betz2016-05-04 11:17:51 -06:00
8c3eacc338
fix cores from XmlDoc::getLinkInfo1() returning -1 because of its call to getFirstIp() presumably.
Matt Wells2016-04-18 09:55:00 -07:00
c5de65a78a
more core dump fixes concerning -1 being returned for XmlDoc::getLinkInfo1()
Matt Wells2016-04-17 18:50:23 -07:00
65856e3b6a
tell malloc to trim 100MB at a time to prevent kernel destroying the cpu by defraging/compacting memory. fix core in Title.cpp.
Matt Wells2016-04-17 09:16:03 -07:00
95a3a261db
fix so host 8 doesn't jam things up so much. host 8 was too busy merging a large spiderdb for blouartinfo and unable to tend to other smaller merges and therefore the # of files was getting out of hand causing slowdowns. so merge spiderdb much less aggressively.
Matt2016-04-15 13:34:41 -06:00
165f724fd7
thanks to isj for the puny code fixes
Matt2016-04-06 11:10:42 -06:00
74cfde3e53
fix calling doneSendingNotification() with a just-freed memory ptr bug.
Matt2016-04-06 10:53:08 -06:00
d2983747b8
fix sending back reply that has some stuff on the stack that it references when XmDoc::getMsg20Reply() returns. thanks to isj for this fix.
Matt2016-04-06 10:43:43 -06:00
8891100c2a
fix add url on root page to set collnum properly. fix Summary::getBestWindow() underrun bug.
Matt2016-04-06 10:31:04 -06:00
48398d0cd7
Merge branch 'diffbot-testing' into testing
Matt2016-03-20 23:14:26 -06:00
136b8842db
fix more data corruption bugs. hopefully will dump out all the collections this time and not leave any in the tree, otherwise, especially if there are a lot left behind, they get corrupted.
Matt Wells2016-03-20 21:04:01 -07:00
61ef806dea
hash bang fix. detect more corruption. don't dump titledb and spiderdb at same time, seems to reduce corruption in rdbmem.
Matt Wells2016-03-20 12:50:43 -07:00
fc495a5bf5
fix dump core when collection deleted while dumping
Matt Wells2016-03-18 06:46:38 -07:00
8922b8e69c
Merge branch 'diffbot-testing' into testing
Matt2016-03-17 14:31:22 -06:00
8bc653c31c
after dump completes scan tree to ensure all nodes reference secondary mem ptr so they don't get their data overwritten.
Matt Wells2016-03-17 10:09:49 -07:00
0caf345850
if running ./gb start and another gb is already bound on the port then quickly exit(0) and have the bash keep alive loop exit the loop based on that return value. we can't use ./cleanexit file because it doesn't get remove and will mess up the main process that is running.
Matt Wells2016-03-16 16:56:48 -07:00
36fdbf2f5a
rename log files in the gb main.cpp code not in the bash loop. do not rename the log file if failed to start gb because socket was already bound. prevents us from double starts moving the log file, which is annoying.
Matt Wells2016-03-16 16:08:08 -07:00
a2e8a3a1fd
use ./cleanexit file to ensure gb doesn't restart after a graceful exit in the bash keep alive loop.
Matt Wells2016-03-16 14:57:19 -07:00
7396e57660
show docids of corrupted title recs found. show key range of each dump to disk. fix 'sentToDiffbot' bug for unchanged docs in status docs. make sure firstKeyInQueue is set properly from current key, so reset list ptr before doing that in RdbDump.cpp.
Matt Wells2016-03-16 13:53:08 -07:00
5e8c47adfd
Merge branch 'diffbot-testing' into testing
Matt2016-03-16 01:14:37 -06:00
Ivan Skytte Jørgensen
CompunixAu
Gigablast
Dmitry Smirnov
Shijuraj J
Zak Betz