32 Commits

Author SHA1 Message Date
4cdd726df0 Make statistics variable atomic 2017-05-01 18:09:40 +02:00
948a796e06 Remove crawlinfo 2017-04-24 10:36:29 +02:00
e3532f73b4 Make SpiderLoop::m_lockTable thread safe 2017-04-21 11:50:21 +02:00
a023239b78 Make SpiderLoop::m_lockTable private 2017-04-18 12:08:52 +02:00
22a931a8aa Move clearLocks logic from SpiderColl to SpiderLoop & change goto to for loop as well 2017-04-18 11:49:01 +02:00
299aba3670 Remove SpiderLoop::m_activeListModified. No additional usage other than what's already covered by SpiderLoop::m_activeListValid 2017-04-12 14:54:12 +02:00
e4855fc1fb Make SpiderLoop::m_activeListValid private. 2017-04-12 14:52:32 +02:00
f451b83689 Remove SpiderLoop::m_lastCallTime. We're now only calling SpiderLoop::spiderDoledUrls from a single place. Don't need to have logic catering for extra calls. 2017-04-12 13:22:14 +02:00
465e41c85d Remove unused SpiderLoop::m_processed 2017-04-12 13:17:44 +02:00
c936ea017c Remove redundant keyword class 2017-04-12 12:37:15 +02:00
5b87fc7430 Rename spiderUrl9 to spiderUrl. Remove SpiderLoop::m_sreq & SpiderLoop::m_doledbKey and instead modify spiderUrl2 to accept those parameters. 2017-04-12 12:16:40 +02:00
8cc17fec12 Remove SpiderLoop::m_collnum 2017-04-12 10:54:23 +02:00
589e94151e Make SpiderLoop::m_maxUsed private 2017-04-12 10:11:37 +02:00
7e5bb393cd Remove unused SpiderLoop::m_lockCache 2017-04-12 09:48:46 +02:00
01a4f9cca4 Make SpiderLoop::m_numSpidersOut private 2017-04-11 11:05:33 +02:00
2dc5b02cc6 Remove unused printLockTable 2017-04-07 11:27:57 +02:00
2d6686ba79 Remove unused variable. Make methods/variables private where possible 2017-04-06 12:47:47 +02:00
5043a1a22e Make static function part of SpiderLoop class 2017-04-06 11:50:57 +02:00
3dbfdf365f Rename SpiderLoop::startLoop to SpiderLoop::init 2017-04-05 11:35:04 +02:00
1349b48c41 Add locks to RdbTree.
Some code style changes/comment removal.
Moved treetest which was in main.cpp to unittest
2017-03-31 12:59:40 +02:00
4b02b5da6c Remove unused variables. Use logDebug instead 2017-03-28 14:30:29 +02:00
e80f0290dd Remove unused variable in SpiderLoop 2017-03-22 12:31:51 +01:00
f7d0c7e863 #include cleanup in SpiderLoop.* 2016-11-13 16:12:03 +01:00
aba937780d Stop #including Conf.h from header files 2016-11-12 20:24:20 +01:00
1cb65b951f Unused member m_outstanding1 removed from SpiderLoop 2016-10-18 22:01:55 +02:00
a040a78c99 Use key96_t instead of key_t and redefining std lib key_t (which breaks std lib functionality that uses key_t) 2016-09-02 14:49:06 +02:00
fad7c45282 Remove Msg12 2016-06-29 12:34:23 +02:00
9d632b6578 Remove undefined functions 2016-03-08 23:14:49 +01:00
ab0b9d03ea Standardize header guards 2016-03-08 22:16:02 +01:00
da8734a848 More Spider splitting 2016-01-30 21:29:32 +01:00
6cee130646 Separated Doledb from Spider.cpp 2016-01-30 20:11:15 +01:00
8463f36179 Split Spider.cpp into Spider, SpiderColl and SpiderLoop. More to come.. 2016-01-30 19:41:54 +01:00