Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • dbcd7b6a6d Don't close fd directly after rename, but instead redirect fd to new file Ai Lin Chia 2016-12-30 12:28:32 +01:00
  • 7e4914bb10 Add more trace logs Ai Lin Chia 2016-12-30 12:27:44 +01:00
  • 249d52b9b2 Code style changes Ai Lin Chia 2016-12-30 11:52:12 +01:00
  • 2d204ddfe7 Modify logs to use logError Ai Lin Chia 2016-12-30 11:25:14 +01:00
  • 7e020d76ec Add constness to HttpMime Ai Lin Chia 2016-12-29 12:35:08 +01:00
  • 88a551a02e Remove gb injecttest Ai Lin Chia 2016-12-28 16:24:30 +01:00
  • 2a4209dc2f Statsdb: use Rdb::addList() instead of low-level Rdb::addRecord() Ivan Skytte Jørgensen 2016-12-30 12:24:35 +01:00
  • bf9cd714f4 Remvoed niceness parameter from Rdb::dumpTree() Ivan Skytte Jørgensen 2016-12-29 12:25:51 +01:00
  • b4f5ed72ad Renamed MAXHOSTBUFSIZE to MINHOSTBUFSIZE (which is was it is) Ivan Skytte Jørgensen 2016-12-29 12:11:06 +01:00
  • c2c43e5024 Removed obsolete comment Ivan Skytte Jørgensen 2016-12-29 12:09:57 +01:00
  • 7c767a35c9 Use correct memory label for Msg4 buffers Ivan Skytte Jørgensen 2016-12-29 11:59:19 +01:00
  • da011094a3 Removed bogus comment Ivan Skytte Jørgensen 2016-12-29 11:34:41 +01:00
  • 57e644d255 Use static qualifier on internal functions consistenly in Msg4.cpp Ivan Skytte Jørgensen 2016-12-29 11:33:13 +01:00
  • b890960b6e constness on Msg4::isInLinkedList parameter Ivan Skytte Jørgensen 2016-12-29 11:28:37 +01:00
  • 849225700a Merge branch 'master' into nomerge2 Ai Lin Chia 2016-12-28 11:01:42 +01:00
  • 58fa5030c6 Fix out of memory bug when generating index Ai Lin Chia 2016-12-28 11:00:48 +01:00
  • 3e89301f0a Removed inconvenience variable Ivan Skytte Jørgensen 2016-12-27 16:00:10 +01:00
  • 473bfd5673 More postfix ++ operators Ivan Skytte Jørgensen 2016-12-27 15:56:13 +01:00
  • 6feaf442fc postfix ++ operator for u_int128_t Ivan Skytte Jørgensen 2016-12-27 15:50:55 +01:00
  • 40d6a15197 Added postfix incrmenet operator to u_int96_t Ivan Skytte Jørgensen 2016-12-27 15:43:25 +01:00
  • 6bbfa6d0aa Changed Repair::m_isDelete to local variable Ivan Skytte Jørgensen 2016-12-27 15:10:47 +01:00
  • 9cedaec82b Cleanup in Repair.cpp Ivan Skytte Jørgensen 2016-12-27 15:08:44 +01:00
  • 7490421dc0 Always log both pre- and post-counts of positive/negative records after a merge Ivan Skytte Jørgensen 2016-12-27 13:59:23 +01:00
  • 7398f656ce Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2016-12-27 12:08:54 +01:00
  • 386daecb58 Bugfix map rebuild of posdb Ivan Skytte Jørgensen 2016-12-27 12:05:44 +01:00
  • 27dde00635 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-23 13:16:03 +01:00
  • 0aa45daa9b Use initial-cap in advanced controls for tabs (makes it easier to locate) Ivan Skytte Jørgensen 2016-12-23 13:15:41 +01:00
  • 96147649bf Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-23 13:07:40 +01:00
  • 81ae44e62a Fix infinite loop in HttpMime::getNextLine() Ivan Skytte Jørgensen 2016-12-23 13:06:56 +01:00
  • cb7b0bd638 Revert "Spider nofollow link" Ai Lin Chia 2016-12-22 17:39:58 +01:00
  • ebb2a6fb86 Spider nofollow link Ai Lin Chia 2016-12-22 17:09:22 +01:00
  • d714e0e0ef Merge branch 'master' into nomerge2 Ai Lin Chia 2016-12-22 15:29:59 +01:00
  • b3164d629f Move RdbIndex merge to cpu thread Ai Lin Chia 2016-12-22 15:28:49 +01:00
  • e72f1a397a Made diacritic detection clearer for synonym generation Ivan Skytte Jørgensen 2016-12-22 13:28:22 +01:00
  • b8aa998dd8 More const in Synonyms and Wiktionary Ivan Skytte Jørgensen 2016-12-22 12:52:19 +01:00
  • 495ab0cabd Cater for hashbang url Ai Lin Chia 2016-12-22 12:42:45 +01:00
  • 66a8d0acd8 Merge branch 'master' of github.com:privacore/open-source-search-engine Ivan Skytte Jørgensen 2016-12-22 12:35:58 +01:00
  • acdf780316 Honour SearchIntpum_queryExpansion when generating summaries with snippets Ivan Skytte Jørgensen 2016-12-22 12:35:18 +01:00
  • 16e5fe6581 Include Latin in the default set of languages to crawl+index Ivan Skytte Jørgensen 2016-12-22 12:34:37 +01:00
  • 3bb6851049 Fix compilation error Ai Lin Chia 2016-12-22 12:29:49 +01:00
  • ec30353afe Remove stripPound variable (default to always true now). Don't store url fragments. Fix relative url resolution for url fragments (we drop them completely now). Ai Lin Chia 2016-12-22 12:26:35 +01:00
  • d57c860af9 Add quotes when printing string for UrlParser::print() Ai Lin Chia 2016-12-21 16:30:27 +01:00
  • d812cb7465 Code style changes Ai Lin Chia 2016-12-21 16:27:20 +01:00
  • 5ae21e0296 Fix relative url resolution except url fragments (#) Ai Lin Chia 2016-12-21 16:26:00 +01:00
  • 37fd0907b5 Only copy url when we need to Ai Lin Chia 2016-12-20 14:30:33 +01:00
  • 0f3008da47 Add some url normalization(scheme & domain case normalization) into UrlParser Ai Lin Chia 2016-12-20 14:25:36 +01:00
  • 0cc2401e40 Store location of port in UrlParser Ai Lin Chia 2016-12-20 14:25:01 +01:00
  • b794dfa65e Parse url even when there is no path in url Ai Lin Chia 2016-12-20 14:24:07 +01:00
  • db13f1e0b8 Code style changes Ai Lin Chia 2016-12-20 14:22:27 +01:00
  • 3983e845dc Code style changes Ai Lin Chia 2016-12-20 12:30:05 +01:00
  • eac5b8d06e Code style changes Ai Lin Chia 2016-12-20 12:23:03 +01:00
  • 181922298c Remove ftp default port as we don't index ftp sites anyway Ai Lin Chia 2016-12-20 12:15:26 +01:00
  • 456812cafb Code style changes Ai Lin Chia 2016-12-20 12:02:10 +01:00
  • 702f40f596 Removed default value of parameter useQueryStopWords in Query::set2() Ivan Skytte Jørgensen 2016-12-22 12:11:15 +01:00
  • 0533502cc2 More constness in summary Ivan Skytte Jørgensen 2016-12-20 16:35:31 +01:00
  • b1b296c990 More constness in summary Ivan Skytte Jørgensen 2016-12-20 16:33:26 +01:00
  • 9fdea668c9 Removed last reference to O_ASYNC Ivan Skytte Jørgensen 2016-12-20 14:59:59 +01:00
  • 3bdc04a51c Really last use of O_ASYNC/O_NONBLOCK on files Ivan Skytte Jørgensen 2016-12-20 14:59:28 +01:00
  • bec2c03d02 Removed last use of O_NONBLOCK / O_ASYNC on files. Ivan Skytte Jørgensen 2016-12-20 14:55:47 +01:00
  • 5d35697c0e Removed last use of O_NONBLOCK in BigFile Ivan Skytte Jørgensen 2016-12-20 14:21:23 +01:00
  • 4357934d61 Don't log blocking/nonblocking in log message Ivan Skytte Jørgensen 2016-12-20 14:01:16 +01:00
  • e1e1359128 BigFile::read/write: changed blocking/nonblocking check. Ivan Skytte Jørgensen 2016-12-20 13:59:34 +01:00
  • 9d9919afd5 Removed now-unused BigFile::setBlocking() and setNonBlocking() Ivan Skytte Jørgensen 2016-12-20 13:34:39 +01:00
  • 26bf6c92d7 Removed calls to BigFile:.setBlocking() in thrutest(): the effect was nil Ivan Skytte Jørgensen 2016-12-20 13:31:47 +01:00
  • ddee7548eb Removed unused BigFile::getFlags() Ivan Skytte Jørgensen 2016-12-20 12:39:34 +01:00
  • 76167355b1 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-20 12:10:02 +01:00
  • c703a85861 Removed no-op code (leftrover and defunct code for setting up SIGIO) Ivan Skytte Jørgensen 2016-12-20 12:09:26 +01:00
  • 3e1bf94863 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-20 12:04:05 +01:00
  • b263887c29 Better encapsulation of RdbCache Ivan Skytte Jørgensen 2016-12-20 11:54:49 +01:00
  • 277718331a Use StackBuf<> Ivan Skytte Jørgensen 2016-12-20 11:33:52 +01:00
  • a6f05334b9 Use a switch instead of if-series Ivan Skytte Jørgensen 2016-12-19 17:29:18 +01:00
  • bf1985bc9e Use StackBuf<> Ivan Skytte Jørgensen 2016-12-19 17:24:57 +01:00
  • 1bd9d77574 Fix mass search&replace error in comments (as int32_t as --> as long as) Ivan Skytte Jørgensen 2016-12-19 17:14:13 +01:00
  • c40f52e2de Make local function decl+def consistent Ivan Skytte Jørgensen 2016-12-19 16:41:38 +01:00
  • 1dabd81a83 goto -> for()-loops Ivan Skytte Jørgensen 2016-12-19 16:06:07 +01:00
  • 362c5037a1 Merge branch 'nomerge2' of github.com:privacore/open-source-search-engine into nomerge2 Ivan Skytte Jørgensen 2016-12-19 15:01:44 +01:00
  • e003f35eab Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-19 15:01:31 +01:00
  • 40be14e752 Merge branch 'master' into nomerge2 Ai Lin Chia 2016-12-19 15:00:17 +01:00
  • 886e4071c4 Make sure RdbIndex std::vector memory is freed Ai Lin Chia 2016-12-19 13:57:56 +01:00
  • d4ba5e8a07 Code style changes Ai Lin Chia 2016-12-16 16:24:02 +01:00
  • fdd2d622b7 Log how much was loaded from high-freq-terms and page-temperature Ivan Skytte Jørgensen 2016-12-19 13:56:08 +01:00
  • cb0105307b bugfix page temperature loading Ivan Skytte Jørgensen 2016-12-19 13:54:07 +01:00
  • e159d801a3 Use page temperature in ranking Ivan Skytte Jørgensen 2016-12-19 12:58:33 +01:00
  • add175c973 handle empty or non-existing page temperatures better Ivan Skytte Jørgensen 2016-12-19 12:58:21 +01:00
  • 4e6080d28c Scale page temperature to [0..1] Ivan Skytte Jørgensen 2016-12-19 12:34:42 +01:00
  • 08b20fddce Prep. for page temperature in ranking Ivan Skytte Jørgensen 2016-12-19 12:24:44 +01:00
  • d98aa780f0 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-12-19 12:00:12 +01:00
  • 445c819273 close fd after file move (because if the move was across filesystem then we have a new file) Ivan Skytte Jørgensen 2016-12-19 11:59:09 +01:00
  • ee199a45df Page temperature registry (preliminary) Ivan Skytte Jørgensen 2016-12-16 17:44:39 +01:00
  • 7a320c773b NUL-terminate the url correctly Ivan Skytte Jørgensen 2016-12-16 15:35:03 +01:00
  • 1f16b65cce Remove no-op method registerMsgHandler3 Ai Lin Chia 2016-12-16 12:34:49 +01:00
  • 3fd7073bc1 Remove no-op Msg40::registerHandler Ai Lin Chia 2016-12-16 12:34:07 +01:00
  • d02a6a3c70 Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2016-12-16 11:10:05 +01:00
  • 24a2049bf0 fix buffer overrun in tld indexing for very long tlds Brian Rasmusson 2016-12-16 11:07:02 +01:00
  • 343079b974 Merge branch 'master' into nomerge2 Ai Lin Chia 2016-12-15 17:10:46 +01:00
  • 7340a15eb4 Added timedMerged to sleep callback to avoid calling gettimeofdayInMilliseconds at every RdbIndex::addRecord Ai Lin Chia 2016-12-15 17:09:19 +01:00
  • 6e0867fe34 Try to optimize Rdb::addRecord when index is enabled. We don't need to delete opposing key when we're not adding a NEG key, or a special key. Ai Lin Chia 2016-12-15 14:57:27 +01:00
  • c43a0ff303 Fix error where repair page wasn't defined Ai Lin Chia 2016-12-15 12:30:35 +01:00
  • c0288fd36c Fix compilation error for release-safe target Ai Lin Chia 2016-12-15 12:23:11 +01:00
  • 09ef1f3fcd Add more trace logs for Summary Ai Lin Chia 2016-12-15 12:17:18 +01:00