Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • cbcda64cb5 const version of Xml::getString Ivan Skytte Jørgensen 2016-12-01 12:15:32 +01:00
  • 3c478d01fd More constness Ivan Skytte Jørgensen 2016-12-01 11:48:27 +01:00
  • 8cf0b4990c #include cleanup in Url.cpp Ivan Skytte Jørgensen 2016-12-01 11:47:44 +01:00
  • 81f079466c More constness in Xml Ivan Skytte Jørgensen 2016-12-01 11:38:23 +01:00
  • c8b692dc62 fix handling of 'base href' where the url does not contain a domain. Sites use base href='/', and that caused gb to use the current url as base instead, which generated wrong URLs if it contained a path. Now builds the URL based on current URL scheme+host and the supplied base href. Brian Rasmusson 2016-11-30 16:49:46 +01:00
  • 8cea6ff906 Added adult checks on TLDs themselves (eg .xxx) Ivan Skytte Jørgensen 2016-11-29 15:55:14 +01:00
  • e92742ddfc fixed core due to 'long' paths when saving rdb cache Brian Rasmusson 2016-11-29 15:50:48 +01:00
  • 07b6909676 Renamed Url::isSpam() to isAdult() Ivan Skytte Jørgensen 2016-11-29 15:06:14 +01:00
  • d3995a07e4 Moved Url::isSpam() logic to AdultCheck.cpp: isAdultUrl() Ivan Skytte Jørgensen 2016-11-29 15:01:31 +01:00
  • aa7f8d24fe Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-11-29 14:51:35 +01:00
  • 416db85d5a Moved isAdult() from Lang.cpp to AdultCheck.cpp Ivan Skytte Jørgensen 2016-11-29 14:50:06 +01:00
  • 99cd9c3392 Fix commit 44e793e905 so we can start unit test again Ai Lin Chia 2016-11-29 14:45:12 +01:00
  • c1bd870cf4 Rename some variables. Remove commented out codes Ai Lin Chia 2016-11-28 17:35:19 +01:00
  • 56027c7e62 Remove unused m_boundaryLen Ai Lin Chia 2016-11-28 15:28:02 +01:00
  • d50ae327c4 Remove commented out code Ai Lin Chia 2016-11-28 10:49:33 +01:00
  • 4cc6435fed Made Url::isSpam (private version) static Ivan Skytte Jørgensen 2016-11-29 14:33:54 +01:00
  • 6a803841e7 Claned up Speller somewhat Ivan Skytte Jørgensen 2016-11-29 14:29:32 +01:00
  • 2e4ea53dfd Made spider crawl-delay configurable. Made robots.txt cache use m_maxRobotsCacheAge parameter instead of hard coded max age Brian Rasmusson 2016-11-29 13:48:25 +01:00
  • c0e15dbb38 gbcompress(): removed method parameter, minor cleanup Ivan Skytte Jørgensen 2016-11-28 15:24:56 +01:00
  • 893758ef82 const for gbuncompress/gbcompress Ivan Skytte Jørgensen 2016-11-28 15:13:42 +01:00
  • 106b0e2baa More constness in Titledb.* Ivan Skytte Jørgensen 2016-11-28 14:56:03 +01:00
  • 4c4385dbeb Remove optional parameters. Move some methods to local static. Make some methods private. Remove unused methods. Ai Lin Chia 2016-11-28 10:47:46 +01:00
  • b5e7892688 Fix comment Ai Lin Chia 2016-11-25 15:36:24 +01:00
  • 0ba98f2b26 Make HttpMime member variable private Ai Lin Chia 2016-11-25 15:31:26 +01:00
  • 569f9b087d Code style changes Ai Lin Chia 2016-11-25 15:30:45 +01:00
  • e46913386f Remove redundent move log file from main.cpp. We're already doing that in Log::init() Ai Lin Chia 2016-11-25 15:24:27 +01:00
  • dafc3c90b4 Rename MsgType.h to msgtype_t.h (to keep things consistent) Ai Lin Chia 2016-11-25 12:22:27 +01:00
  • 780542978a Document titledb key better Ivan Skytte Jørgensen 2016-11-25 17:37:54 +01:00
  • 0bd5b44b78 Renamed Titledb::getGlobalNumDocs() to estimateGlobalNumDocs() Ivan Skytte Jørgensen 2016-11-25 17:20:29 +01:00
  • 9b261d5640 fix the previous fix Brian Rasmusson 2016-11-25 17:11:06 +01:00
  • 91cc53ba76 If it is detected that the DocId version in the current file is not the newest, then only advance the term list pointers in the INTERSECT_SCORING pass. Fixes coredump. Brian Rasmusson 2016-11-25 16:27:33 +01:00
  • ad2e5a1592 Removed write-only Multicast::m_lastLaunchHost Ivan Skytte Jørgensen 2016-11-25 16:20:42 +01:00
  • b129d9c7a6 Removed read-only Multicase::m_retryCount Ivan Skytte Jørgensen 2016-11-25 16:18:46 +01:00
  • e26e078c39 Fix multicast sleep callback interval. Ivan Skytte Jørgensen 2016-11-25 16:00:20 +01:00
  • bfc9f6dc89 Multicast: removed assumption that a group at most has 2 hosts Ivan Skytte Jørgensen 2016-11-25 15:42:56 +01:00
  • a03a965d36 Moved most of Multicast::sleepWrapper1() logic to a non-static member function sleepWrapper2() Ivan Skytte Jørgensen 2016-11-25 15:41:19 +01:00
  • e1117d8c03 Renamed Multicast::sendToGroup() to sendToWholeGroup() (that is what it does) Ivan Skytte Jørgensen 2016-11-25 15:18:31 +01:00
  • 479438bd93 Collected Multicast member arrays into more clear structs Ivan Skytte Jørgensen 2016-11-25 14:59:57 +01:00
  • b9a4b78bca Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-11-25 14:18:08 +01:00
  • 44e793e905 hack away core dump in Mem.cpp when something calls Mem::rmMem() after g_mem has been destroyed Ivan Skytte Jørgensen 2016-11-25 14:17:49 +01:00
  • afc0b88817 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-11-25 14:00:22 +01:00
  • e6211d547b Stop statistics thread when shutting down gracefully Ivan Skytte Jørgensen 2016-11-25 13:50:06 +01:00
  • b09636aba5 Store titlerec with no content to keep linkdb links intact for documents with EDOCSIMPLIFIEDREDIR error Ai Lin Chia 2016-11-24 16:22:47 +01:00
  • 657f52c1c5 Make Msg8a/Msge0 coorperation more clear Ivan Skytte Jørgensen 2016-11-24 17:00:12 +01:00
  • 7b74f5e23c Slightly clearer logic in PageHosts.cpp Ivan Skytte Jørgensen 2016-11-24 16:48:26 +01:00
  • f13f5d687d Renamed Dns::getResponsibleHost() to getIPLookupHost() Ivan Skytte Jørgensen 2016-11-24 16:31:08 +01:00
  • 8e34f85829 Removed 'shortcut' to g_dns Ivan Skytte Jørgensen 2016-11-24 16:06:08 +01:00
  • ccd211261b Make MsgC/Msge1 coorperation more clear Ivan Skytte Jørgensen 2016-11-24 15:33:07 +01:00
  • 4f9017056e Cleaned up paranoid sanity-checks Ivan Skytte Jørgensen 2016-11-24 14:51:37 +01:00
  • 25d0df06b9 Add missed out newline to hostsconf preamble Ai Lin Chia 2016-11-24 14:26:05 +01:00
  • 5573d72880 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2016-11-24 14:11:53 +01:00
  • 06112ab78b Handle 6-byte keys too in high-freq-term-list Ivan Skytte Jørgensen 2016-11-24 13:44:32 +01:00
  • eff33fe85a Code style changes dev-redirection Ai Lin Chia 2016-11-24 11:27:57 +01:00
  • f7cf402add Remove unused XmlDoc::getLastRedirUrl Ai Lin Chia 2016-11-23 16:31:27 +01:00
  • 3c9e0e74aa Remove commented out code Ai Lin Chia 2016-11-23 15:53:40 +01:00
  • e4602d330e Remove always 0 argument Ai Lin Chia 2016-11-23 15:52:03 +01:00
  • c76ce227da Remove unused niceness from args Ai Lin Chia 2016-11-23 15:47:05 +01:00
  • 3a991bfaea Code style changes & add some todo comment Ai Lin Chia 2016-11-23 15:14:10 +01:00
  • 129a26e991 Add trace logs to XmlDoc::getCanonicalRedirUrl Ai Lin Chia 2016-11-23 15:08:09 +01:00
  • 1c81580d38 Remove double semicolon at end of line Ai Lin Chia 2016-11-23 13:34:48 +01:00
  • fb24cba26b Fix content type from test/xml to text/xml Ai Lin Chia 2016-11-23 12:49:49 +01:00
  • 4c5cef50cd Move variable declaration closer to where it's use. Combine two identical if blocks Ai Lin Chia 2016-11-23 12:49:10 +01:00
  • 9170bba7cc Remove commented out code Ai Lin Chia 2016-11-23 12:48:32 +01:00
  • fb981fde30 Remove unused member variable Ai Lin Chia 2016-11-23 11:59:18 +01:00
  • e67a1ab66f Remove old misc/urlinfo.cpp. Use tools/print_urlinfo.cpp instead Ai Lin Chia 2016-11-23 11:55:41 +01:00
  • 230cfa9346 Remove code to handle <meta name=usefakeips content=1> which is only used when importing gb dmoz url Ai Lin Chia 2016-11-23 11:46:37 +01:00
  • a106d1b1ff Remove code to handle <meta name=noindex content=1> which is only used when importing gb dmoz url Ai Lin Chia 2016-11-23 11:42:33 +01:00
  • c151d2672b use correct message offsets in multicast.cpp for msg39 decoding Ivan Skytte Jørgensen 2016-11-22 16:21:02 +01:00
  • 1439a925ce postpone local variable decl+def until first use Ivan Skytte Jørgensen 2016-11-22 15:58:09 +01:00
  • d45474deaf Add more trace logs for redirection Ai Lin Chia 2016-11-22 16:16:13 +01:00
  • d9b2b0cc38 Code style changes Ai Lin Chia 2016-11-22 16:14:50 +01:00
  • 87355a8e01 Add some comments Ai Lin Chia 2016-11-22 16:11:31 +01:00
  • 260f816528 Remove unused member variable m_nextPtr Ai Lin Chia 2016-11-22 12:40:14 +01:00
  • 7cee4ea8e3 Fix clang unreachable code warnings Ai Lin Chia 2016-11-22 12:31:44 +01:00
  • 279eb30017 Revert "Moved members from Hostdb to statics in Parms.cpp" Ivan Skytte Jørgensen 2016-11-22 15:17:22 +01:00
  • 6af6308462 Revert "Revert "Removed unused local function isMyIp()"" Ivan Skytte Jørgensen 2016-11-22 15:16:39 +01:00
  • 7b20e9b146 Revert "Removed unused local function isMyIp()" Ivan Skytte Jørgensen 2016-11-22 15:13:11 +01:00
  • 4bfccc8d09 Moved members from Hostdb to statics in Parms.cpp Ivan Skytte Jørgensen 2016-11-22 15:10:59 +01:00
  • 543dc2e7c2 Merge branch 'master' of https://github.com/privacore/open-source-search-engine Brian Rasmusson 2016-11-22 15:00:17 +01:00
  • 6ee00bc624 made mergeBufSize config parameter visible on Rdb Controls tab Brian Rasmusson 2016-11-22 14:59:58 +01:00
  • 0494839b0e Removed unused Host/Hostdb machineNum/numMahcnie fields, methods and calculation Ivan Skytte Jørgensen 2016-11-22 14:49:07 +01:00
  • 3aaf0265d4 Fix Hostdb::saveHostsConf() (would write hosts.conf in wrong format Ivan Skytte Jørgensen 2016-11-22 14:42:07 +01:00
  • 74f706b9e2 Dump utf8content for get_titlerec tool Ai Lin Chia 2016-11-22 12:18:55 +01:00
  • 654d993e59 Rename unused member variable reserved3 to m_reserved33 Ai Lin Chia 2016-11-22 12:18:15 +01:00
  • d0dc199ae4 Remove commented out code Ai Lin Chia 2016-11-22 12:17:35 +01:00
  • 4b4cfe1f70 Use logDebug instead Ai Lin Chia 2016-11-22 12:16:52 +01:00
  • 7fc953028b Print uh48 for dumpSpiderdb in hex instead of decimal Ai Lin Chia 2016-11-22 12:16:16 +01:00
  • 0d832e32ad Removed unused local function isMyIp() Ivan Skytte Jørgensen 2016-11-21 14:20:01 +01:00
  • 8f006d083d Remove special hack for nytimes. general fix has been applied in commit a5217ae456 Ai Lin Chia 2016-11-21 11:16:38 +01:00
  • f19112bdee Remove Linkdb::setLostDate_uk Ai Lin Chia 2016-11-18 15:51:40 +01:00
  • 0b688008a5 Removed superfluous null check on 'url' (was tested earlier) Ivan Skytte Jørgensen 2016-11-20 14:56:26 +01:00
  • 6502541baf Make local functions static Ivan Skytte Jørgensen 2016-11-19 15:20:00 +01:00
  • 99ad00d7f8 Removed default values on parameters to Multicast:.send() Ivan Skytte Jørgensen 2016-11-18 16:14:18 +01:00
  • a10d38ae57 Remove hopcount from Linkdb::makeKey_uk Ai Lin Chia 2016-11-18 14:52:30 +01:00
  • 8127c023a0 Remove unused functions/args/variables Ai Lin Chia 2016-11-18 13:33:22 +01:00
  • a9339718ed Remove commented out code Ai Lin Chia 2016-11-18 11:47:49 +01:00
  • d333627511 Fix parameter name in declaration of to Hostdb::pickBestHost() Ivan Skytte Jørgensen 2016-11-18 14:53:16 +01:00
  • cf69971ffe Dropped 'hostNumToTry' parameter to Multicast::sendToHostLoop() Ivan Skytte Jørgensen 2016-11-18 14:51:19 +01:00
  • 62aad5f05f Made logic in Hostdb::isDead() clearer Ivan Skytte Jørgensen 2016-11-18 14:36:29 +01:00
  • 43696be481 Don't check if Hostdb::getShard() reutnrs NULL (it never does) Ivan Skytte Jørgensen 2016-11-18 14:29:48 +01:00