Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • f6cbd070a2 Initial step of running unit test in jenkins Ai Lin Chia 2017-06-14 11:25:45 +02:00
  • 96b45ef0ca Use declarative pipeline instead Ai Lin Chia 2017-06-14 11:17:37 +02:00
  • 6e11a144df Fix RdbListTest by disabling rdbindex check Ai Lin Chia 2017-06-14 11:07:03 +02:00
  • 38bc710b13 Revert "Fix RdbList unit test by disabling index for posdb. We're testing the logic of RdbList here, not the logic of Posdb" Ai Lin Chia 2017-06-14 11:04:35 +02:00
  • e3a9e92cea Fix RdbList unit test by disabling index for posdb. We're testing the logic of RdbList here, not the logic of Posdb Ai Lin Chia 2017-06-14 11:03:20 +02:00
  • 36b8ca4ecb Add compact_enc_det as third-party module Ai Lin Chia 2017-06-14 10:39:01 +02:00
  • cef9a1c9c2 Clean workspace before checkout Ai Lin Chia 2017-06-13 23:25:15 +02:00
  • c512be2d7e Let's try to get submodule working in Jenkins Ai Lin Chia 2017-06-13 23:03:56 +02:00
  • c0fcdeb16c Try to init submodule as well Ai Lin Chia 2017-06-13 15:05:28 +02:00
  • 23d1ef6b1c Initial commit of Jenkinsfile Ai Lin Chia 2017-06-13 14:58:11 +02:00
  • 691e2774cc Bugfix shutdown in PosdbTable::mergeTermSubListsForDocId() Ivan Skytte Jørgensen 2017-06-13 12:04:36 +02:00
  • 44e704324a Bugfix shutdown in PosdbTable::mergeTermSubListsForDocId() Ivan Skytte Jørgensen 2017-06-13 12:04:36 +02:00
  • c6270cc837 Check for empty toptree in PosdbTable::intersectLists() Ivan Skytte Jørgensen 2017-06-12 16:56:46 +02:00
  • 1d577d7446 Bugfix core in intersectLists10_r() Ivan Skytte Jørgensen 2017-06-12 16:53:22 +02:00
  • 490ee03383 Check for empty toptree in PosdbTable::intersectLists() Ivan Skytte Jørgensen 2017-06-12 16:56:46 +02:00
  • e9a7bc3557 Bugfix core in intersectLists10_r() Ivan Skytte Jørgensen 2017-06-12 16:53:22 +02:00
  • 45e443bd1c Fix allocation of working-area in PosdbTable::intersectLists10_r() Ivan Skytte Jørgensen 2017-06-12 16:14:28 +02:00
  • 6e95ffa4a7 Removed commented-out code in topTree.* Ivan Skytte Jørgensen 2017-06-12 13:14:19 +02:00
  • edc214d1fa Fix segfault for specific pattern url (when urlcomponent is normalized) Ai Lin Chia 2017-06-12 12:15:59 +02:00
  • b8a340c55b Moved memory allocation from msg39 processing thread to intersection thread Ivan Skytte Jørgensen 2017-06-09 16:46:54 +02:00
  • c9eacfe472 Move save map/index after merging to a thread Ai Lin Chia 2017-06-09 16:28:38 +02:00
  • cc74736285 Rearranged and documented members of PosdbTable Ivan Skytte Jørgensen 2017-06-09 15:39:27 +02:00
  • d4a95b2d10 Move addList to a thread for RdbDump Ai Lin Chia 2017-06-09 13:47:49 +02:00
  • 623a11f74f Change goto to while loop Ai Lin Chia 2017-06-09 12:37:54 +02:00
  • 438a1ac641 Moved allocation of working memory from msg39 to intersect Ivan Skytte Jørgensen 2017-06-09 15:00:23 +02:00
  • 2ca828369b Made PosdbTable::m_topTree private Ivan Skytte Jørgensen 2017-06-09 13:55:36 +02:00
  • 7100ed38ab Better encapsulation of PosdbTable Ivan Skytte Jørgensen 2017-06-09 13:52:54 +02:00
  • b8c4274d65 Removed unused PosdbTable::m_collnum Ivan Skytte Jørgensen 2017-06-09 13:45:29 +02:00
  • 26ecd5fb9d Removed unused PosdbTable::m_maxScores Ivan Skytte Jørgensen 2017-06-09 13:44:27 +02:00
  • 28cdc30632 Removed unused PosdbTable::m_numSlots Ivan Skytte Jørgensen 2017-06-09 13:43:04 +02:00
  • abe22bb633 Removed unused PosdbTable::m_estimatedTotalHits Ivan Skytte Jørgensen 2017-06-09 13:42:25 +02:00
  • 1a3dd8306f Removed unused PosdbTable::m_docsInColl Ivan Skytte Jørgensen 2017-06-09 13:41:20 +02:00
  • 5574aba451 Removed unsused PosdbTable::m_logstate Ivan Skytte Jørgensen 2017-06-09 13:37:53 +02:00
  • 5b344a0a30 Better encapsulation of PosdbTable Ivan Skytte Jørgensen 2017-06-09 13:33:38 +02:00
  • 64619390f9 Renamed miniMergedList/miniMergedEnd to miniMergedListStart/miniMergedListEnd Ivan Skytte Jørgensen 2017-06-09 12:36:08 +02:00
  • da72115e55 Add more logs/config for RdbIndex & RdbMap addList timings from merge Ai Lin Chia 2017-06-08 20:35:06 +02:00
  • 72d3b61ce9 Add more comments (mostly todo comments) dev-rwlock Ai Lin Chia 2017-06-08 16:37:02 +02:00
  • 5465b81f0f Tell PosdbTable::mergeTermSubListsForDocId() where the minimergebuf ends instead of letting it guess with the magic value 299000 Ivan Skytte Jørgensen 2017-06-08 15:05:53 +02:00
  • 7891d3a431 Remove unused function RdbBucket::getNumNegativeKeys Ai Lin Chia 2017-06-08 14:12:40 +02:00
  • 58242b5521 Add libprotobuf-dev to travis Ai Lin Chia 2017-06-08 13:38:49 +02:00
  • 44ee471fc7 Fix building of libcld3.so Ai Lin Chia 2017-06-08 13:34:05 +02:00
  • 6d9959e6e5 Add protobuf-compiler package to travis Ai Lin Chia 2017-06-08 12:34:49 +02:00
  • da2511ab5c Make sure library is build first Ai Lin Chia 2017-06-08 12:26:50 +02:00
  • 1635ad2d50 Code style changes Ai Lin Chia 2017-06-08 12:20:41 +02:00
  • 8828b998c1 Merge branch 'master' into dev-language Ai Lin Chia 2017-06-08 12:15:24 +02:00
  • b36dd779e2 Add initial implementation of getting lang from CLD3 Ai Lin Chia 2017-06-08 12:13:07 +02:00
  • 3f25a5b7da Encapsulate Msg5 slightly more. Don't expose member variables directly Ai Lin Chia 2017-06-08 12:03:52 +02:00
  • 775e06f937 Remove unused RdbTree::getBytesWritten_unlocked & RdbTree::getBytesRead_unlocked Ai Lin Chia 2017-06-07 16:51:09 +02:00
  • e85847f303 Remove unused RdbBuckets::getNumKeys_unlocked Ai Lin Chia 2017-06-07 16:49:17 +02:00
  • 12fa7d8855 Use rwlock for RdbTree Ai Lin Chia 2017-06-07 16:32:44 +02:00
  • 2e7412abbd Use rwlock for RdbBuckets Ai Lin Chia 2017-06-06 17:15:48 +02:00
  • 883571a3ad Don't use spider: for current_spiders statistics Ai Lin Chia 2017-06-06 16:58:42 +02:00
  • f6551ecc8c Fix typo in variable Ai Lin Chia 2017-06-06 15:19:34 +02:00
  • a10c61bfe1 Fix -Wreorder warning Ai Lin Chia 2017-06-06 14:51:03 +02:00
  • cb5fcbd730 Don't submit acquire lock job more than once Ai Lin Chia 2017-06-06 14:39:35 +02:00
  • 38de902335 Code style changes. Remove commented out code Ai Lin Chia 2017-06-06 12:36:08 +02:00
  • f68061166f Spelling fix in log Ivan Skytte Jørgensen 2017-06-06 14:20:09 +02:00
  • ba475ce1f8 Fidx vagus keepalive interval always being twice Ivan Skytte Jørgensen 2017-06-06 13:50:55 +02:00
  • a7a0b8d2de Set max spider per ip to 1 Ai Lin Chia 2017-06-06 11:55:11 +02:00
  • 31888b185a Make RdbBuckets::getLock private Ai Lin Chia 2017-06-02 14:53:40 +02:00
  • c0dd3265d2 Abort if we have not seen ourself alive for the past x minutes (defaults to 5) Ai Lin Chia 2017-06-02 12:20:50 +02:00
  • 8b20afcd8d Fix bug where we may wait indefinately while destroying RdbIndex after merge. Issue with not locking m_pendingMerge Ai Lin Chia 2017-06-02 11:33:25 +02:00
  • 56b3d94312 Addec comment explaining "nwp[mink][2] & 0x03" actually is Ivan Skytte Jørgensen 2017-06-01 17:28:48 +02:00
  • ab595df934 Moved sanity check inside if() (nullptr is never>) Ivan Skytte Jørgensen 2017-06-01 16:56:20 +02:00
  • 54f939a1aa Turned PosdbTable::m_qdist into a normal local+parameter Ivan Skytte Jørgensen 2017-06-01 16:46:14 +02:00
  • 389d448860 Simplify ocde in findMinTermPairScoreInWindow() a bit Ivan Skytte Jørgensen 2017-06-01 16:23:56 +02:00
  • 9f2356b234 Fix bug where early return due to empty pendingDocIds cause m_pendingMerge not to be reset Ai Lin Chia 2017-06-01 16:46:36 +02:00
  • e647249872 Move RdbMerge::m_mergeSpaceCoordinator->acquire to a thread (it always sleeps 1 second when acquiring lock is successfull) Ai Lin Chia 2017-06-01 16:27:08 +02:00
  • 61d99d8d68 Only shrink sublists once Ivan Skytte Jørgensen 2017-06-01 16:06:45 +02:00
  • 70d597dbe0 Remove unused RdbMerge::getRdbId function Ai Lin Chia 2017-06-01 15:55:28 +02:00
  • 14b3cedb45 Use logDebug/logTrace in RdbMerge. Add more log lines Ai Lin Chia 2017-06-01 15:53:18 +02:00
  • 353fd339fc Modify vagus dead/alive log slightly Ai Lin Chia 2017-06-01 15:14:19 +02:00
  • acf54bfaaa Use g_conf.m_logTraceRdbMerge instead of g_conf.m_logTraceRdbDump inside RdbMerge Ai Lin Chia 2017-06-01 15:02:09 +02:00
  • 5a06d25448 Always submit a mergePendingDocIds job without locking RdbIndex::m_pendingDocIdsMtx to avoid blocking the main thread Ai Lin Chia 2017-06-01 14:30:48 +02:00
  • 9f3de50ad2 Add debug vagus log (config). Change some vague log to info instead Ai Lin Chia 2017-06-01 13:49:47 +02:00
  • 9674238da3 Remove additional dash when checking for "-h" Ai Lin Chia 2017-06-01 11:10:43 +02:00
  • a4af30519c Fix compilation error for validate_rdbindex tool Ai Lin Chia 2017-06-01 11:09:28 +02:00
  • cc53df3615 Fix calculation of endMergeFileNum when resuming a killed merge Ai Lin Chia 2017-05-31 17:22:44 +02:00
  • b691937dd4 Cater for newly dumped file in a different way while merging list Ai Lin Chia 2017-05-31 16:33:23 +02:00
  • eee0725843 verifyFileSharding is not really doing a file merge Ai Lin Chia 2017-05-31 16:07:51 +02:00
  • 15a55d4fd5 Add sanity for UdpSlot, message size shouldn't change across datagrams Ai Lin Chia 2017-05-31 15:01:19 +02:00
  • 356895efc5 Rename startFileNum to startFileIndex. Check only up to files that are being merged when verifying RbdIndex. We should have newly dump files that could have the docId, but not in global index yet Ai Lin Chia 2017-05-31 14:21:04 +02:00
  • 04a589e806 Move preparation of global index to thread as well Ai Lin Chia 2017-05-31 12:07:07 +02:00
  • 3404178dad Fix issue with page term display that is not displaying some terms that are hashed Ai Lin Chia 2017-05-31 11:41:19 +02:00
  • d52bdac673 Don't do a timedMerge when we're generating index Ai Lin Chia 2017-05-30 15:49:48 +02:00
  • dde0184785 Remoed unused QueryTerm::m_underNOT Ivan Skytte Jørgensen 2017-05-30 15:31:32 +02:00
  • b9519ca2f4 Initialize with some sane default value as we don't always load config file (when using ./gb cmd) Ai Lin Chia 2017-05-30 15:15:47 +02:00
  • 15f35b2ba4 Don't finalize some process twice. Remove double call to g_jobScheduler.disallow_new_jobs. Ai Lin Chia 2017-05-30 13:10:40 +02:00
  • 17f1cf8a2e Removed unused Query::m_hasQuotaField Ivan Skytte Jørgensen 2017-05-30 15:14:26 +02:00
  • fab45a7395 Removed Query::m_hasUOR Ivan Skytte Jørgensen 2017-05-30 15:07:21 +02:00
  • 49d0cf4d23 Removed QueryTerm::m_isUORed and QueryTerm::m_UORedTerm Ivan Skytte Jørgensen 2017-05-30 12:41:58 +02:00
  • 81fb24d82a Change RdbIndex::m_pendingMerge to atomic to avoid locking RdbIndex::m_pendingDocIdsMtx when checking if it's set Ai Lin Chia 2017-05-30 12:32:26 +02:00
  • 4da8a3594e Minor reformatting in PosdbTable::setQueryTermInfo() Ivan Skytte Jørgensen 2017-05-30 12:28:53 +02:00
  • 9b233ca842 Removed obsolete comments from PosdbTable::setQueryTermInfo() Ivan Skytte Jørgensen 2017-05-30 12:22:27 +02:00
  • cc0967510b Add logging when loop callback hit time threshold. Remove some unused function, remove undefined function (only defined in header) Ai Lin Chia 2017-05-30 12:11:10 +02:00
  • 4940258a15 Add log when we're unable to submit regenerate file job Ai Lin Chia 2017-05-29 16:56:58 +02:00
  • a6774454d3 Cater for Rdb where index is not used (for resuming killed merge) Ai Lin Chia 2017-05-29 16:44:38 +02:00
  • 90a4e398ac Fix bug where we will never stop regenerating map/index when resuming a killed merge Ai Lin Chia 2017-05-29 16:40:41 +02:00
  • 0ce5dc0dbd Delay generation of RdbIndex/RdbMap for killed merge until when we need it Ai Lin Chia 2017-05-29 16:27:39 +02:00
  • 14a9f82971 goto -> for() Ivan Skytte Jørgensen 2017-05-29 15:13:37 +02:00