Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • 0be8c50ed8 Avoid buffer overflow in calls to KEYSTR (use MAX_KEY_BYTES*2+1 for buffer) Ivan Skytte Jørgensen 2017-03-27 15:49:33 +02:00
  • 8173e2065a Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-27 15:41:09 +02:00
  • a9051ab705 Fix f80a29978f where dataCopy is used instead of data Ai Lin Chia 2017-03-27 15:40:04 +02:00
  • 75db44d81d Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-27 15:05:27 +02:00
  • 7f3a5f4cb8 Re-add warning suppression Ai Lin Chia 2017-03-27 15:05:10 +02:00
  • 7a21587ca6 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-27 15:03:19 +02:00
  • dba364ef22 Remove set but not used variable Ai Lin Chia 2017-03-27 15:01:32 +02:00
  • 02dbdfddef Fix clang warning: no newline at end of file Ai Lin Chia 2017-03-27 14:57:20 +02:00
  • 81c9d5b56e Remove now unused m_dataInPtrs variable Ai Lin Chia 2017-03-27 14:55:12 +02:00
  • 1bfa9d3317 Remove unused g_inHandler Ai Lin Chia 2017-03-27 14:55:00 +02:00
  • 2bb1e0e514 Remove unused variable Ai Lin Chia 2017-03-27 14:54:43 +02:00
  • dad3e273b8 Move clang++ if to before g++. clang++ matches g++ as well. Ai Lin Chia 2017-03-27 14:54:16 +02:00
  • ec9a1201c0 Remove now unused query statistics from Stats Ai Lin Chia 2017-03-27 14:30:37 +02:00
  • 2a705fb1a6 RdbList: more const Ivan Skytte Jørgensen 2017-03-27 14:40:37 +02:00
  • f80a29978f Made Rdb::addRecord() take const pointers Ivan Skytte Jørgensen 2017-03-27 14:29:03 +02:00
  • 2fc275fdd3 Remove separate initializeNeedle function and initialize string size in constructor instead Ai Lin Chia 2017-03-27 14:25:52 +02:00
  • 5b47247ec5 Fix delete. We can't delete void* ptr Ai Lin Chia 2017-03-27 14:06:52 +02:00
  • 80e56b7f0f Spider/SpiderColl: const Ivan Skytte Jørgensen 2017-03-27 13:57:29 +02:00
  • aba799fa80 Spider.*: const Ivan Skytte Jørgensen 2017-03-27 13:53:57 +02:00
  • d260ea6d6c Spider.*: const Ivan Skytte Jørgensen 2017-03-27 13:51:27 +02:00
  • 9efabf06e9 Fix memory leak when generating index Ai Lin Chia 2017-03-27 12:55:10 +02:00
  • 550b560dd7 SpiderColl::isInDupCache() constness Ivan Skytte Jørgensen 2017-03-27 12:56:28 +02:00
  • 8612611e04 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-27 12:49:30 +02:00
  • fa16dff3e6 Remove unused function Ai Lin Chia 2017-03-27 12:47:00 +02:00
  • 5cf4e51002 Add error code & qlang to query statistics Ai Lin Chia 2017-03-27 12:43:26 +02:00
  • b8629e24d2 Remove "allow cloud user" variable that is not working anyway. Ai Lin Chia 2017-03-27 12:43:05 +02:00
  • 92abbd153e Remove commented out code Ai Lin Chia 2017-03-27 10:54:40 +02:00
  • d5f9717c36 Don't register spider specific callbacks for query host Ai Lin Chia 2017-03-27 10:53:59 +02:00
  • 4cefd66135 Spider.h: const Ivan Skytte Jørgensen 2017-03-27 12:47:25 +02:00
  • 3613402acd RdbIndex::addKey/addRecord_unlocked: const Ivan Skytte Jørgensen 2017-03-27 12:44:04 +02:00
  • 5b13833655 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-27 12:06:48 +02:00
  • 8735946f42 Removed now-unused Rdb::getTree() Ivan Skytte Jørgensen 2017-03-27 12:06:34 +02:00
  • acff2e8ada Moved Msg5::getTreeList() logic+code to new Rdb::getTreeList() Ivan Skytte Jørgensen 2017-03-27 12:05:42 +02:00
  • 6096dac475 Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2017-03-26 10:02:52 +02:00
  • c90319b16d revert commit 7872cd5090 - 'Send only getcrawlinfo to hosts that have spidering enabled' because it had the side effect that spidering would not be paused if a nospider host died Brian Rasmusson 2017-03-26 10:00:58 +02:00
  • f3fe24fd05 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 23:03:48 +01:00
  • b25ddb57a1 We shouldn't continue with merge when there is pending generate global index in queue Ai Lin Chia 2017-03-25 23:02:21 +01:00
  • 33ca049297 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 21:44:37 +01:00
  • 3f73b77137 We should only remove from queue when we're done processing it Ai Lin Chia 2017-03-25 21:42:59 +01:00
  • 9cc555af2b Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 19:55:51 +01:00
  • 339f1a78ff More fixes for verify index Ai Lin Chia 2017-03-25 19:55:05 +01:00
  • 2cee927c6e Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 19:16:52 +01:00
  • be45454d0e Only verify index when it's a file merge Ai Lin Chia 2017-03-25 19:13:58 +01:00
  • 024a0dac4a Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 17:17:05 +01:00
  • 77d8106362 Only use base->getIndex if it's readable Ai Lin Chia 2017-03-25 17:16:32 +01:00
  • 30c88bebae Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 16:57:55 +01:00
  • ea2b286871 Fix verify index for the scenario that we're in the middle of a dump Ai Lin Chia 2017-03-25 16:57:18 +01:00
  • c417e592bb Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 15:23:31 +01:00
  • a5a09790cc Fix verifyIndex so cater for more scenarios Ai Lin Chia 2017-03-25 15:22:39 +01:00
  • b0139d489a Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 11:51:24 +01:00
  • a4df893078 Fix compilation error Ai Lin Chia 2017-03-25 11:51:11 +01:00
  • b1aac4f6d4 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-25 11:49:08 +01:00
  • 9fbc3275e5 Sort pending docIds before using lower_bound Ai Lin Chia 2017-03-25 11:48:19 +01:00
  • 6a4ef6e283 Remove spiderDoledUrls in addSpiderReply. Why are we even calling that?! Ai Lin Chia 2017-03-24 16:51:06 +01:00
  • f6d9f23e98 Move initialization of SpiderColl into SpiderColl constructor from Spider.cpp Ai Lin Chia 2017-03-24 16:30:43 +01:00
  • 68bff70362 Code style changes Ai Lin Chia 2017-03-24 16:27:53 +01:00
  • ac62890874 Remove write only variable SpiderColl::m_siteListHasNegatives Ai Lin Chia 2017-03-24 16:22:43 +01:00
  • d26faa4395 Update note for mnew & mdelete for SpiderColl Ai Lin Chia 2017-03-24 16:21:19 +01:00
  • 32a8af7168 Rename m_list2, m_nextKey2, etc.. to m_waitingTreeList2, m_waitingTreeNextKey, etc.. Ai Lin Chia 2017-03-24 16:18:03 +01:00
  • ea4b877068 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-24 16:30:04 +01:00
  • d77b46c72f Rework record size estimation in Msg5::readList() Ivan Skytte Jørgensen 2017-03-24 16:29:49 +01:00
  • eb7103547f Code style changes Ai Lin Chia 2017-03-24 16:02:26 +01:00
  • 8614dfe4b8 Remove commented out code Ai Lin Chia 2017-03-24 16:01:06 +01:00
  • e9c461078b Remove unused stats from Performance page Ai Lin Chia 2017-03-24 15:56:21 +01:00
  • 884c6fa2cd Code style changes Ai Lin Chia 2017-03-24 15:55:59 +01:00
  • 03153a501b Remove stats for PageInject (we already have statistics for spidering pages) Ai Lin Chia 2017-03-24 15:41:52 +01:00
  • 8ce0b0dfe2 Remove duplicate data between Statistics and g_stats.addStat_r Ai Lin Chia 2017-03-24 14:49:38 +01:00
  • 2bcae2e655 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-24 15:11:58 +01:00
  • 4aef3a9acd Made nice method Rdb::deleteTreeNode() so spiderloop don't have to know directly about Rdb::m_tree Ivan Skytte Jørgensen 2017-03-24 15:09:26 +01:00
  • 39ac1e9b93 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-24 14:40:50 +01:00
  • e408ee7268 Sync nomerge2 changes to master for main.cpp Ai Lin Chia 2017-03-24 14:39:48 +01:00
  • 37f8c64e34 Sync nomerge2 changes to master for Log.cpp Ai Lin Chia 2017-03-24 14:39:04 +01:00
  • 4563bdffa2 Sync RdbIndexQuery changes from nomerge2 to master Ai Lin Chia 2017-03-24 14:32:49 +01:00
  • 924bdef6d4 Fix rdbindex tools Ai Lin Chia 2017-03-24 14:32:30 +01:00
  • 94fad65e3c Sync changes from nomerge2 for get_titlerec Ai Lin Chia 2017-03-24 14:15:03 +01:00
  • 7da3b8ad48 Renamed RdbTree::getListSize() to estimateListSize() Ivan Skytte Jørgensen 2017-03-24 14:34:37 +01:00
  • 297a28452f Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-24 14:11:05 +01:00
  • e164f20c75 Sync changes from nomerge2 to master (move verify integrity from Rdb to Msg4In) Ai Lin Chia 2017-03-24 14:08:00 +01:00
  • 4a203531d1 Sync changes for RdbBuckets from nomerge2 Ai Lin Chia 2017-03-24 14:04:07 +01:00
  • a67f572894 Sync changes in nomerge2 to master for TopTree (comment removal) Ai Lin Chia 2017-03-24 14:02:40 +01:00
  • 0e167e4b1a Sync lock changes from nomerge2 to master Ai Lin Chia 2017-03-24 14:01:04 +01:00
  • cd05a362e2 Extract RdbCacheLock related code in nomerge2 to master Ai Lin Chia 2017-03-24 13:19:29 +01:00
  • d5f8161fb7 Removed RdbTree::setNumKeys() Ivan Skytte Jørgensen 2017-03-24 14:00:44 +01:00
  • c9fd19d754 Removed RdbTree::m_doBalancing Ivan Skytte Jørgensen 2017-03-24 13:24:22 +01:00
  • 23df724e29 removed 'dataInPtrs' paramteer to RdbTree::set() Ivan Skytte Jørgensen 2017-03-24 13:10:33 +01:00
  • d1153969fb Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-24 12:42:16 +01:00
  • 51fc0283cd Initial code for enabling processing of Msg4In in a thread. Currently disabled until dependencies are thread-safe Ai Lin Chia 2017-03-24 12:37:04 +01:00
  • c0c0bdc970 Move tryToDeleteSpiderColl from Spider to SpiderColl. Make wrapper function static function of SpiderColl Ai Lin Chia 2017-03-24 12:23:18 +01:00
  • 27d414bf04 Remove commented out code Ai Lin Chia 2017-03-24 11:21:52 +01:00
  • 0422b923bf Change goto to for loop Ai Lin Chia 2017-03-24 11:11:53 +01:00
  • 944c47cbd7 Remove unused function SpiderColl::makeWaitingTree Ai Lin Chia 2017-03-24 11:01:56 +01:00
  • bee39ada8a Make makeWaitingTreeKey static function Ai Lin Chia 2017-03-24 10:56:14 +01:00
  • de61d1b76a Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-23 17:50:34 +01:00
  • 390d74bbc4 Add lock to silence thread sanitizer Ai Lin Chia 2017-03-23 17:49:29 +01:00
  • dd48809831 Add more trace log & log for shardByTermId Ai Lin Chia 2017-03-23 17:33:12 +01:00
  • 66e264fe2a Remove unused/always false args from getMatches2 Ai Lin Chia 2017-03-23 12:09:48 +01:00
  • 6349c42c52 Manipulate Posdb entries shardByTermId delete key to doc delete key so we can delete those posdb entries when it's stored on a different shard Ai Lin Chia 2017-03-23 12:08:06 +01:00
  • 0d21817c78 #include cleanup in RdbTree Ivan Skytte Jørgensen 2017-03-23 17:28:09 +01:00
  • 2028d3ea5b Made RdbTree callbacks private Ivan Skytte Jørgensen 2017-03-23 17:26:14 +01:00
  • 62c53c8d43 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-23 16:50:52 +01:00