Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • 27d90d89d1 Fix compilation error Ai Lin Chia 2016-07-26 15:30:58 +02:00
  • d4c4d55510 Add trace logs for XmlDoc::getUrlFilterNum Ai Lin Chia 2016-07-26 15:18:49 +02:00
  • 907bd15003 Only cleanup spiderdb when we're compiling with PRIVACORE_SAFE_VERSION (tmp code to clear out unwanted tld) Ai Lin Chia 2016-07-26 15:17:13 +02:00
  • 89a247c110 Modify log line to be more similar so we can grep all url that are unwanted for indexing Ai Lin Chia 2016-07-26 14:57:23 +02:00
  • 253ea1fd47 Change log level Ai Lin Chia 2016-07-26 14:56:15 +02:00
  • 1ad80a2b6d Suppress more clang warnings Ai Lin Chia 2016-07-26 13:57:25 +02:00
  • b0fa2eb0cc Remove more qa stuff Ai Lin Chia 2016-07-26 13:57:09 +02:00
  • 37d203f540 Add privacore tld blacklist Ai Lin Chia 2016-07-26 12:53:05 +02:00
  • fd6e8cbb21 Remove more qa specific code Ai Lin Chia 2016-07-26 12:20:22 +02:00
  • c2279f0c4a Renamed Msg5::threadDoneWrapper to something more descriptive Ivan Skytte Jørgensen 2016-07-26 14:57:39 +02:00
  • fbe6b7c0ec use Msg3::areAllScansCompleted() internally Ivan Skytte Jørgensen 2016-07-26 13:30:23 +02:00
  • bd6a976fb9 Encapsulate Msg3 a bit more Ivan Skytte Jørgensen 2016-07-26 12:52:25 +02:00
  • 39db950803 Removed dead code Ivan Skytte Jørgensen 2016-07-26 12:12:08 +02:00
  • 46892c9588 removed qa.cpp and immediate callers Ivan Skytte Jørgensen 2016-07-26 11:32:11 +02:00
  • 8b85e3c654 Removed qatest123 from Posdb.cpp Ivan Skytte Jørgensen 2016-07-26 11:06:12 +02:00
  • f9d9334d99 Remove commented out code and unused variable Ai Lin Chia 2016-07-26 09:08:44 +02:00
  • 919602b82b Rearranged some parts of UdpServer header. Remove commented out code Ai Lin Chia 2016-07-25 21:36:24 +02:00
  • d03334ca07 Change log level Ai Lin Chia 2016-07-25 17:05:27 +02:00
  • 7d9bc5f729 Encapsulate UdpServer by changing private methods/variables to private Ai Lin Chia 2016-07-25 17:05:05 +02:00
  • 35d4becbbc Remove g_stats.addSpiderPoint and related spider statistics Ai Lin Chia 2016-07-25 17:03:51 +02:00
  • 6e1e22edca Remove unused Msg17 Ai Lin Chia 2016-07-25 16:51:03 +02:00
  • 5f14fb2cee More msgType changes Ai Lin Chia 2016-07-25 16:00:20 +02:00
  • a6f18184ed Use abort() instead of kill8self,sigsegv) Ivan Skytte Jørgensen 2016-07-25 15:35:21 +02:00
  • c60e2862e8 Include msg counts for full range of message type Ai Lin Chia 2016-07-25 14:49:52 +02:00
  • 95d9e8a01c Change direct usage of m_msgType to getMsgType Ai Lin Chia 2016-07-25 14:49:17 +02:00
  • dd800eb19c Delete comment Ai Lin Chia 2016-07-25 14:41:18 +02:00
  • 1ce5b59670 Code style changes. More changes of hardcoded msg_type_t value to enum. Log levels. Ai Lin Chia 2016-07-25 14:31:46 +02:00
  • 5904307ddf Remove unused Dns::cancel Ai Lin Chia 2016-07-25 14:23:49 +02:00
  • 970c04ba05 Change log level Ai Lin Chia 2016-07-25 14:10:56 +02:00
  • bde87ec9a3 Modify hardcoded msg type to a proper msg_type_t enum Ai Lin Chia 2016-07-25 12:22:04 +02:00
  • 547da23469 More sanity check in Msg5::gotList2() Ivan Skytte Jørgensen 2016-07-25 14:59:50 +02:00
  • d5b0cf7de0 Encapsulate msg2 more Ivan Skytte Jørgensen 2016-07-25 12:06:52 +02:00
  • f3f4285283 Remove unused/always false isHandlerHot from UdpServer Ai Lin Chia 2016-07-22 16:48:31 +02:00
  • 1ff2bd2103 Remove reference to msg 0x50 Ai Lin Chia 2016-07-22 16:43:33 +02:00
  • c887588163 Remove reference to msg 0x3b Ai Lin Chia 2016-07-22 16:42:11 +02:00
  • 51e36af261 Remove reference to msg 0x35 Ai Lin Chia 2016-07-22 16:41:35 +02:00
  • b8dcaa8128 Remove reference to msg 0x23 Ai Lin Chia 2016-07-22 16:40:33 +02:00
  • e37c36b593 Remove reference to msg 0x24 Ai Lin Chia 2016-07-22 16:38:30 +02:00
  • 11b181d995 Remove reference to msg 0x2c Ai Lin Chia 2016-07-22 16:37:43 +02:00
  • 62be7fa523 Remove reference to msg 0x36 Ai Lin Chia 2016-07-22 16:36:10 +02:00
  • ef1e708dc5 Remove reference to msg 0x02 Ai Lin Chia 2016-07-22 16:34:01 +02:00
  • 3a22fd3a61 Remove reference to msg 0x34 Ai Lin Chia 2016-07-22 16:33:09 +02:00
  • 83ad86c91e Remove commented out code Ai Lin Chia 2016-07-22 16:32:43 +02:00
  • e3d5eda784 Remove reference to msg 0x08 Ai Lin Chia 2016-07-22 16:31:31 +02:00
  • 91ed387123 Remove reference to msg 0x09 Ai Lin Chia 2016-07-22 15:51:26 +02:00
  • fdcf48e003 Remove reference to msg 0x06 Ai Lin Chia 2016-07-22 15:50:39 +02:00
  • 4867e4f0b3 Remove reference to msg 0x12 Ai Lin Chia 2016-07-22 15:49:48 +02:00
  • d0498a2874 Remove reference to msg 0x0d Ai Lin Chia 2016-07-22 15:49:09 +02:00
  • 306e66026e Remove reference to msg 0x10 Ai Lin Chia 2016-07-22 15:46:27 +02:00
  • 3082ed9c36 Remove reference to msg 0x8b Ai Lin Chia 2016-07-22 15:42:07 +02:00
  • ddf9257840 Rename m_tmpBuf to m_hostname which is what it really stores Ai Lin Chia 2016-07-22 15:05:14 +02:00
  • abedd0c4cd Remove commented out token logic from UdpServer Ai Lin Chia 2016-07-22 14:31:06 +02:00
  • e1862d46f7 Removed unnecessary forward-decl of static inline functions in UnicodeProperties.h Ivan Skytte Jørgensen 2016-07-22 23:59:06 +02:00
  • 6e7c4a636a Avoid duplicated #defines Ivan Skytte Jørgensen 2016-07-22 23:52:54 +02:00
  • 2d54f60231 Removed calls to setrlimit(...core,infinite) Ivan Skytte Jørgensen 2016-07-22 17:38:24 +02:00
  • 2fcbd6ae6f Removed non-functional code from printStackTrace() Ivan Skytte Jørgensen 2016-07-22 17:24:15 +02:00
  • 09e358524a Simplified Msg5::needsRecall() by using early-return instead of gotos Ivan Skytte Jørgensen 2016-07-22 16:44:36 +02:00
  • 3f2097f9d1 Converted goto-loop into do-while in Msg5.cpp Ivan Skytte Jørgensen 2016-07-22 16:40:25 +02:00
  • ffd67619be More enapsulation of Msg5 (repairLists_r/mergeLists_r) Ivan Skytte Jørgensen 2016-07-22 16:32:28 +02:00
  • 4bc8febff7 Moved call to RdbList:.resetListPtr() to just after merge_r() where it is needed Ivan Skytte Jørgensen 2016-07-22 16:23:01 +02:00
  • b807e18c47 Changed static functiosn into static member functions in UdpServer Ivan Skytte Jørgensen 2016-07-22 15:46:13 +02:00
  • 6592f65be7 Simplify sanity check in Msg5 Ivan Skytte Jørgensen 2016-07-22 15:44:38 +02:00
  • ecfbdc33c6 Remove unused variable Ai Lin Chia 2016-07-22 14:25:31 +02:00
  • e9eee473dc Remove diffbot specific code. -> maxcrawl, maxprocess, crawl pattern, crawl done notification, etc... Ai Lin Chia 2016-07-22 12:36:59 +02:00
  • 0207dedf83 Add static to s_init Ai Lin Chia 2016-07-22 12:22:17 +02:00
  • 90eba9d323 Fix s_init check in Bits.cpp. It should be a static Ai Lin Chia 2016-07-22 12:21:47 +02:00
  • b2090e063e Remove unreachable code Ai Lin Chia 2016-07-22 12:21:24 +02:00
  • 0fdbe713de Ignore lastcore* files Ai Lin Chia 2016-07-22 12:20:02 +02:00
  • 8d4ac301c2 Fix bug where we continue even if adding to tree fails (caught by compiler warnings) Ai Lin Chia 2016-07-22 12:19:27 +02:00
  • 483f3eb719 Rename Statistics header guard to prevent warning from clang Ai Lin Chia 2016-07-22 11:55:54 +02:00
  • 594c6194fd Remove unused regex structure & related safebuf Ai Lin Chia 2016-07-22 00:18:59 +02:00
  • 570838d7f6 Remove m_isCustomCrawl & logic surrounding it Ai Lin Chia 2016-07-22 00:09:03 +02:00
  • cc825d5ced Remove customCrawl logic from add new collection Ai Lin Chia 2016-07-21 23:35:17 +02:00
  • adc6665c94 Remove now unused ECUSTOMCRAWLMISMATCH Ai Lin Chia 2016-07-21 23:19:06 +02:00
  • c460ffb442 Remove addCrawl & addBulk of custom crawl logic Ai Lin Chia 2016-07-21 23:16:31 +02:00
  • 7e3ce87318 Ignore statistics file Ai Lin Chia 2016-07-21 17:40:52 +02:00
  • 3726fdaa7b Remove diffbot specific CollectionRec::m_notifyUrl Ai Lin Chia 2016-07-21 17:24:02 +02:00
  • 59d147f1be Code style changes Ai Lin Chia 2016-07-21 17:19:13 +02:00
  • 57e741a115 Remove crawlbot page and 'bulk' download of titlerec in csv files Ai Lin Chia 2016-07-21 17:18:11 +02:00
  • 026b286c98 Remove commented out code Ai Lin Chia 2016-07-21 16:45:22 +02:00
  • ec7bf45c35 Fix typo Ai Lin Chia 2016-07-21 15:52:20 +02:00
  • 1157e76275 Remove unused header Ai Lin Chia 2016-07-21 15:51:54 +02:00
  • a1b0f2bc64 fx_fetld only contains tld now Ai Lin Chia 2016-07-21 15:36:39 +02:00
  • f69057d340 Move PRIVACORE_SAFE_VERSION to cover a bigger scope (functions doesn't need to be defined when enabled) Ai Lin Chia 2016-07-21 15:36:06 +02:00
  • 4ef9db6cf2 Code style changes & simplify code Ai Lin Chia 2016-07-21 15:35:27 +02:00
  • f11d223185 Revert "TagRec::m_lists[] is a regular array and there is no need for calling ::constructor() on them" Ivan Skytte Jørgensen 2016-07-22 13:58:24 +02:00
  • 14c0f874dd Better encapsulation of Msg5 Ivan Skytte Jørgensen 2016-07-22 13:40:55 +02:00
  • e5392ef08a Get rid of 'retRecPtr' in RdbCache methods (not thread safe) Ivan Skytte Jørgensen 2016-07-21 16:08:14 +02:00
  • cdb443ec17 Add force flag to symbolic link creation Ai Lin Chia 2016-07-21 15:34:21 +02:00
  • 499633e40e Mem::printBreeches() should always force a shutdown/abort Ivan Skytte Jørgensen 2016-07-21 15:13:17 +02:00
  • 43afae853b Removed commented-out code Ivan Skytte Jørgensen 2016-07-21 14:41:19 +02:00
  • 91c0e3a068 Changed log-level on too-many-clients to error Ivan Skytte Jørgensen 2016-07-21 14:40:56 +02:00
  • 4903d9fefd Don't move files to trash Ai Lin Chia 2016-07-20 18:15:27 +02:00
  • 8e4b84aa96 Code style changes. Change log level to WARN for errors Ai Lin Chia 2016-07-20 17:48:42 +02:00
  • cd0e78ac5c Ignore slacktee.* Ai Lin Chia 2016-07-20 17:46:33 +02:00
  • 038c764e06 Don't restart gb if uptime is less than 5 minutes. Use attachment for slack notifications. Ai Lin Chia 2016-07-20 17:43:41 +02:00
  • 830e9016c8 Include slacktee.sh when building dist tarball Ai Lin Chia 2016-07-19 17:49:23 +02:00
  • aae526c186 Add slacktee as third-party git submodule Ai Lin Chia 2016-07-19 17:48:14 +02:00
  • a95a4d6063 Use logTrace instead Ai Lin Chia 2016-07-19 15:41:34 +02:00
  • 74c0468f59 set g_errno on new[] failure Ivan Skytte Jørgensen 2016-07-19 16:23:03 +02:00