Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • 3d84aaf5d6 Removed unused Msge0::m_numTags Ivan Skytte Jørgensen 2017-03-13 12:11:33 +01:00
  • 77fa86fe0e Msge0::doneSending() should return void Ivan Skytte Jørgensen 2017-03-10 17:30:52 +01:00
  • 9adbd26eb7 Removed obsolete comments in Msge0.cpp Ivan Skytte Jørgensen 2017-03-10 17:29:00 +01:00
  • 60914d1ea5 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-13 12:13:22 +01:00
  • 23cc52eae4 Revert "temp fix to prevent core dump" Ai Lin Chia 2017-03-13 12:07:50 +01:00
  • d232d58c04 Use m_endKey instead of endKey. We have manipulated m_endKey Ai Lin Chia 2017-03-13 12:06:51 +01:00
  • 56a3a78ac9 Rename endKey to newEndKey to avoid shadowing another parameter Ai Lin Chia 2017-03-13 12:06:30 +01:00
  • 224bd301b3 Move titledb init variable to static function Ai Lin Chia 2017-03-13 12:05:09 +01:00
  • c371c4eda6 moved log related settings to 'Log controls' page Brian Rasmusson 2017-03-12 17:38:41 +01:00
  • 51dff74a19 log (configurable) UdpServer timings as warning Brian Rasmusson 2017-03-11 23:44:14 +01:00
  • ffe5c4e9d9 Use a more meaningful parameter name in Msge0 Ivan Skytte Jørgensen 2017-03-10 17:25:50 +01:00
  • f1f2f41daa Handle OOM correctly in Msge0 Ivan Skytte Jørgensen 2017-03-10 17:23:49 +01:00
  • 4075c0da0c Refactor slab handling in Msge0 into separate method Ivan Skytte Jørgensen 2017-03-10 17:17:57 +01:00
  • 27546ffc76 Reintroduce comment accidently removed Ai Lin Chia 2017-03-10 17:02:21 +01:00
  • 7be769711f Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 16:49:40 +01:00
  • bfad6188ca temp fix to prevent core dump Ai Lin Chia 2017-03-10 16:47:49 +01:00
  • 5e099b3443 More constness in Msge0 Ivan Skytte Jørgensen 2017-03-10 16:25:29 +01:00
  • 5a5ebb9bba 0 -> false Ivan Skytte Jørgensen 2017-03-10 16:22:21 +01:00
  • 213369a2f3 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 16:19:28 +01:00
  • 6d047b0e7b Even more trace logs Ai Lin Chia 2017-03-10 16:19:10 +01:00
  • 38de91f380 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 14:52:44 +01:00
  • 7b75932bce Be more specific in logs Ai Lin Chia 2017-03-10 14:52:06 +01:00
  • 76a6b5d82e Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-10 14:52:16 +01:00
  • ab5fe781ff If a response is not 200 then don't parse more than the Mime header, and certainly don't try to decompress response body etc. Ivan Skytte Jørgensen 2017-03-10 14:50:59 +01:00
  • c47ee68e96 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 14:47:48 +01:00
  • 02ffc91e95 Add more trace logs Ai Lin Chia 2017-03-10 14:47:33 +01:00
  • 766a381c2f Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 14:34:53 +01:00
  • b570b1f9f6 Log list last key Ai Lin Chia 2017-03-10 14:34:21 +01:00
  • daa83bec80 Add decode titledb key to tools Ai Lin Chia 2017-03-10 14:33:59 +01:00
  • 0596b7154a Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 13:32:33 +01:00
  • df3f33e52c Add more trace logs Ai Lin Chia 2017-03-10 13:31:39 +01:00
  • 804d838cd9 Msg13Request: use offsetof() instead of pointer calculations Ivan Skytte Jørgensen 2017-03-10 12:56:32 +01:00
  • e08ae2b052 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-10 12:22:42 +01:00
  • 7090500b2c Add trace log to RdbMerge, RdbDump, RdbMap Ai Lin Chia 2017-03-10 11:56:10 +01:00
  • 0f49ddd9ea newEndKey is not manipulated anymore. Use m_endKey directly Ai Lin Chia 2017-03-10 11:40:39 +01:00
  • 50a7bd8677 const for getMsgSize() Ivan Skytte Jørgensen 2017-03-10 11:52:34 +01:00
  • bd21e43189 Log a bit more when a corrupt compressed stream is detected Ivan Skytte Jørgensen 2017-03-10 11:43:38 +01:00
  • 67cdafd505 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-09 17:47:49 +01:00
  • 199cd8f23d Avoid deadlock when UdpServer::sendReply() needs to call UdpServer::sendErrorReply() Ivan Skytte Jørgensen 2017-03-09 17:47:33 +01:00
  • 1bc06e97c5 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-09 17:02:43 +01:00
  • fb6e34a044 Don't submit merge job if we don't need to merge Ai Lin Chia 2017-03-09 16:59:11 +01:00
  • 56f963141e Fix bug where sometimes we are adding invalid docId into RdbIndex Ai Lin Chia 2017-03-09 16:58:43 +01:00
  • 7592d260c6 Move generateGlobalIndex to thread; Use updateGlobalIndex after merging instead of regenerating global index Ai Lin Chia 2017-03-09 16:56:59 +01:00
  • e3d5061f9e Made CollectioNRec::m_maxTextDocLen and m_maxOtherDocLen work Ivan Skytte Jørgensen 2017-03-09 16:44:49 +01:00
  • 85b60ac049 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-08 16:45:46 +01:00
  • 3b6f9099fc Revert "Make sure we're not blocking main thread while merging pending doc ids" Ai Lin Chia 2017-03-08 16:45:28 +01:00
  • 5a6845cc2f Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-08 16:21:24 +01:00
  • 868ddb4616 Reserve vector to avoid slow reallocation Ai Lin Chia 2017-03-08 16:19:13 +01:00
  • 03854b8490 Use g++-5 for coverity scans as well Ai Lin Chia 2017-03-08 15:57:16 +01:00
  • b6488a7ea0 Remove commented out code Ai Lin Chia 2017-03-08 15:52:45 +01:00
  • 59261209eb Use logDebug instead Ai Lin Chia 2017-03-08 15:52:27 +01:00
  • 20d0e4b87f Remove unused args from makeLinkInfo Ai Lin Chia 2017-03-08 15:49:32 +01:00
  • 41818c1974 Remove commented out code Ai Lin Chia 2017-03-08 15:42:32 +01:00
  • c5693bf0f0 Remove noop code. icount is not used Ai Lin Chia 2017-03-08 15:34:58 +01:00
  • 47ddaa03bd Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-08 12:10:25 +01:00
  • b5f4e003d7 Make sure we're not blocking main thread while merging pending doc ids Ai Lin Chia 2017-03-08 12:09:30 +01:00
  • 7a2d9a91a7 Don't add linkdb record for simplified redir & canonical Ai Lin Chia 2017-03-08 11:14:30 +01:00
  • da25ad1cfd Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2017-03-07 19:58:28 +01:00
  • 86402d3dd7 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-07 17:14:01 +01:00
  • aa116ad0c3 Produce valid JSON in PageHosts Ivan Skytte Jørgensen 2017-03-07 17:07:53 +01:00
  • 5253f4c5f7 m_numRequests should be zeroed in Msge0::reset() Ivan Skytte Jørgensen 2017-03-07 15:15:02 +01:00
  • 64efc60967 Removed unused 'starti' parameter to Msge0::launchRequests() Ivan Skytte Jørgensen 2017-03-07 15:03:39 +01:00
  • 17643eb99f goto -> for() Ivan Skytte Jørgensen 2017-03-07 14:42:13 +01:00
  • 6de18085b2 Simplify logic in DailyMerge Ivan Skytte Jørgensen 2017-03-07 12:57:06 +01:00
  • 6b65776baf Only try resuming interrupted merges on first call to RdbBase::attemptMerge() Ivan Skytte Jørgensen 2017-03-07 12:49:37 +01:00
  • 5c34265e97 Add more test for urlblocklist Ai Lin Chia 2017-03-07 12:43:39 +01:00
  • 950e6b7165 Removed obsolete commented-out code Ivan Skytte Jørgensen 2017-03-07 12:15:01 +01:00
  • 8ee096f84c Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-07 12:11:43 +01:00
  • 0d1d4b6247 Make sure all instance of filterTitledbList is commented out Ai Lin Chia 2017-03-07 11:40:30 +01:00
  • 925ad57311 Fix bug with HttpMime where empty attribute value for domain will cause segfault Ai Lin Chia 2017-03-07 11:35:20 +01:00
  • 0122638abc Add blocking of path by host to urlblocklist Ai Lin Chia 2017-03-07 11:34:42 +01:00
  • fa436998c4 Use _exit() instead of exit() when told to shut down Ivan Skytte Jørgensen 2017-03-07 12:00:53 +01:00
  • 7a09cd8870 PageThreads: verify-data job type Ivan Skytte Jørgensen 2017-03-06 17:24:19 +01:00
  • 70a683b4f4 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-06 17:11:13 +01:00
  • 56ab2c751c First relinquish merge-space lock when files have been file has been moved/renamed Ivan Skytte Jørgensen 2017-03-06 17:10:31 +01:00
  • 112911c588 Better log in MergeSpaceCoordinator.cpp Ivan Skytte Jørgensen 2017-03-06 16:56:22 +01:00
  • c38a9ac08f Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-06 16:57:04 +01:00
  • 0ba85ea733 Cater for pthread_cond_wait spurious wakeup Ai Lin Chia 2017-03-06 16:55:25 +01:00
  • eeaf3a69d1 Make sure we don't try to merge a deleted index Ai Lin Chia 2017-03-06 16:36:07 +01:00
  • ad7168737a Move RdbDump list-verification to a separate job Ivan Skytte Jørgensen 2017-03-06 16:31:13 +01:00
  • 58fd815e79 Fix compilation error Ai Lin Chia 2017-03-06 16:34:11 +01:00
  • d4c150b761 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-06 16:31:46 +01:00
  • 93e0dbb4b5 Move RdbDump list-verification to a separate job Ivan Skytte Jørgensen 2017-03-06 16:31:13 +01:00
  • 978988373c Rename spider-dedup threads to merge-filter. Add filterTitledbList (uncalled) to filter based on urlblocklist Ai Lin Chia 2017-03-06 16:17:49 +01:00
  • b54cd86191 Remove privacore specific tld checks Ai Lin Chia 2017-03-03 11:34:05 +01:00
  • 087f817f0d Add support of tld to UrlBlockList Ai Lin Chia 2017-03-02 10:38:41 +01:00
  • aebc2e8ff8 Cater for multiple criteria types for UrlBlockList to speed things up Ai Lin Chia 2017-03-01 15:37:31 +01:00
  • 71c8deb6dc Move dedup spiderdb to cpu thread Ai Lin Chia 2017-03-01 14:22:21 +01:00
  • 8183e3cfa3 Move some string util function to GbUtils Ai Lin Chia 2017-03-01 14:11:39 +01:00
  • d60ed6818c Remove urlblocklist test based on real url Ai Lin Chia 2017-03-01 14:09:50 +01:00
  • aa08b184da Removed niceness parameter to RdbDump::dumpList() because it was always the same value as m_niceness previous assigned in RdbDump::set() Ivan Skytte Jørgensen 2017-03-06 16:02:41 +01:00
  • 213c68dbe4 Made 3-parameter RdbDump::dumpList() private so we hav econtrol of when it is a recall Ivan Skytte Jørgensen 2017-03-06 15:55:10 +01:00
  • 5c4e255f66 Move empty-list optimization to earlier Ivan Skytte Jørgensen 2017-03-06 15:52:15 +01:00
  • dd160dfd47 Fix calls to RdbList::checkList_r() in currently #defined-away code Ivan Skytte Jørgensen 2017-03-06 14:54:23 +01:00
  • 9672cc1a54 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-06 13:11:21 +01:00
  • 8e925da185 Don't double-verfiy lists being dumped Ivan Skytte Jørgensen 2017-03-06 13:10:42 +01:00
  • af83cadc24 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-06 12:57:33 +01:00
  • d554847dd7 All file merges should be done in the merge-thread pool (not just posdb) Ivan Skytte Jørgensen 2017-03-06 12:54:18 +01:00
  • a97e964554 Add thread pool for merging Ai Lin Chia 2017-03-06 11:43:49 +01:00
  • 4a9df34f76 Add thread pool for merging Ai Lin Chia 2017-03-06 11:43:49 +01:00