Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity

Commit Graph

  • 5ec750970a Don't insert sreq->m_firstIp into SpiderColl::m_localTable Ivan Skytte Jørgensen 2017-07-24 15:35:13 +02:00
  • c39b5e2a37 Remoed 'quotaTable' parameter from getUrlFilterNum() because it was always the spidercoll:m_localTable Ivan Skytte Jørgensen 2017-07-24 15:22:15 +02:00
  • 4a798cb535 Use /etc/hosts Ai Lin Chia 2017-07-24 14:57:49 +02:00
  • ed46992e70 Add DNS permanent error code to url filters Ai Lin Chia 2017-07-24 11:33:08 +02:00
  • 671fafa15d Merge branch 'dev-dns' Ai Lin Chia 2017-07-24 11:03:07 +02:00
  • f68e3e685e SpiderColl: more const Ivan Skytte Jørgensen 2017-07-21 17:15:36 +02:00
  • 274ec9540d Cater for IP hosts Ai Lin Chia 2017-07-21 11:54:09 +02:00
  • 28f387346a Make tools compile Ai Lin Chia 2017-07-20 17:13:36 +02:00
  • 5727abc191 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-21 16:59:47 +02:00
  • 883a649c4b Merge branch 'staging' into stable stable Ai Lin Chia 2017-07-21 16:16:34 +02:00
  • 586dcb9c5d Merge branch 'master' into staging Ai Lin Chia 2017-07-21 16:16:18 +02:00
  • c5590223fa Don't supporess messages to apparently dead hosts when doing command-line stuff (because in that case we are not talking to vagus) Ivan Skytte Jørgensen 2017-07-21 16:16:00 +02:00
  • 25873f1c49 hovsa Ivan Skytte Jørgensen 2017-07-21 16:03:07 +02:00
  • 91e7af306c Merge branch 'staging' into stable Ai Lin Chia 2017-07-21 15:44:29 +02:00
  • 008bc7a8d6 Merge branch 'master' into staging Ai Lin Chia 2017-07-21 15:42:50 +02:00
  • 81f10d5caa Update usage Ai Lin Chia 2017-07-21 14:45:54 +02:00
  • 192cc0c752 Minor display update on Spider-queue admin page Ivan Skytte Jørgensen 2017-07-21 15:41:31 +02:00
  • f10a9f6b65 Fix name/url of admin page DoledbIP table Ivan Skytte Jørgensen 2017-07-21 15:12:39 +02:00
  • 6c5f7bbe14 Added admin page for DoledbIP table Ivan Skytte Jørgensen 2017-07-21 15:06:27 +02:00
  • 711e4019ce Call setHashTable before using s_tagDesc Ai Lin Chia 2017-07-21 13:40:10 +02:00
  • ea6b2b0c56 Fix compilation error in previous commit Ai Lin Chia 2017-07-21 12:07:05 +02:00
  • 2e262b19ca We should only filter positive keys. It shouldn't matter for Linkdb but for Tagdb, we will be reading invalid recSize. So to keep things consistent, we'll only filter positive records. Negative records should go away when it's merged to file #0 anyway Ai Lin Chia 2017-07-21 11:59:04 +02:00
  • 324ef8f389 fix #include placement in Pages.h Ivan Skytte Jørgensen 2017-07-20 17:39:55 +02:00
  • 2e341627df Print human-readable timestamp in dump waitingtree Ivan Skytte Jørgensen 2017-07-20 15:14:00 +02:00
  • 0e822c1cfb Fix coredump in cmdline dump-waitingtree (unlocked mutex) Ivan Skytte Jørgensen 2017-07-20 15:07:38 +02:00
  • 825c6c7055 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-20 11:44:53 +02:00
  • 217eace473 Fix compilation warning (warning: enum constant in boolean context) Ai Lin Chia 2017-07-20 11:44:30 +02:00
  • 12042f11c4 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-20 11:34:23 +02:00
  • d6494c37a2 Fix coverity warning. We should never get NULL for getUtf8Content at the point of time when getContentLangIdCLD2 is called, but will just add the check to silent coverity. Ai Lin Chia 2017-07-20 11:32:49 +02:00
  • 4782aa23e0 Don't finalize multiple times Ai Lin Chia 2017-07-19 15:54:23 +02:00
  • 10450c514d Add GbThreadQueue to GbDns to process request (rather than blocking main thread) Ai Lin Chia 2017-07-19 15:38:09 +02:00
  • bddd1080b8 Add scopelock around usage of s_channel Ai Lin Chia 2017-07-19 14:24:21 +02:00
  • b58f01d96b Store hostname in DnsItem Ai Lin Chia 2017-07-19 14:08:03 +02:00
  • 2ea2dc5e79 More const in Parms.* Ivan Skytte Jørgensen 2017-07-18 16:33:04 +02:00
  • c9800f01b6 Reestablish urlfilter rebuild hack in Parms.cpp Ivan Skytte Jørgensen 2017-07-18 16:08:11 +02:00
  • e1a09e1707 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-18 15:40:28 +02:00
  • ed68fd41df Use getUtf8Content for CLD2 instead of getRawUtf8Content Ai Lin Chia 2017-07-18 15:39:35 +02:00
  • 9bba41eeca Initialize variables passed into CLD2. This sometimes will not be initialized. Ai Lin Chia 2017-07-18 15:36:30 +02:00
  • aa4e3cb3eb Avoid buffer-overflow from CGI parameter names Ivan Skytte Jørgensen 2017-07-18 15:38:00 +02:00
  • 6c761570e5 Skip dead hosts when updating parametres Ivan Skytte Jørgensen 2017-07-18 15:14:36 +02:00
  • 6ead9f2ed5 Merge branch 'master' of github.com:privacore/open-source-search-engine Ivan Skytte Jørgensen 2017-07-18 14:57:26 +02:00
  • 6eabd12071 Use the delay for pagereindex even when there is a spider reply Ai Lin Chia 2017-07-18 11:29:23 +02:00
  • 92974ebb40 Split DNS errors into more error codes Ai Lin Chia 2017-07-18 11:03:03 +02:00
  • 4444e176ba Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-17 16:16:58 +02:00
  • 22d373f2b4 Modify smerge to spmerge. smerge used to be section merge, spmerge is for spiderdb which is what we want Ai Lin Chia 2017-07-17 16:14:27 +02:00
  • 0d16c5bb4e Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-17 16:04:36 +02:00
  • 3f480b488d Remove dead code Ai Lin Chia 2017-07-17 16:01:26 +02:00
  • 9f408b67ad added -f to rm of entities.inc to avoid make clean failing Brian Rasmusson 2017-07-17 15:57:51 +02:00
  • 535354b852 added url filters to remove docs with errcount for badip and bad http status Brian Rasmusson 2017-07-17 15:57:28 +02:00
  • 85fb3ff997 const in PageReindex.* Ivan Skytte Jørgensen 2017-07-17 15:43:30 +02:00
  • 267c620dbf Added trace-log option to query-reindex Ivan Skytte Jørgensen 2017-07-17 15:24:30 +02:00
  • e0bc452967 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-17 15:24:25 +02:00
  • 2abf9ea017 Add config for spiderReindexDelay instead of misusing url filters Ai Lin Chia 2017-07-17 15:23:08 +02:00
  • d2c1d24f49 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-17 15:09:02 +02:00
  • 9ea719a99b Throttle page reindex based on spiderIpWait configured in url filters Ai Lin Chia 2017-07-17 14:53:27 +02:00
  • 23f37c672f Use spider frequency as configured in url filters Ai Lin Chia 2017-07-17 14:51:59 +02:00
  • 0b6e1ab1e0 Remove commented out code. Code style changes. Move declaration to where it's assigned Ai Lin Chia 2017-07-17 14:51:25 +02:00
  • dda611ac8d Properly disabled synonyms for query-reindex Ivan Skytte Jørgensen 2017-07-17 14:47:04 +02:00
  • cbd7ecea39 Separate log setting for reindex (was caught by setting for repair) Ivan Skytte Jørgensen 2017-07-17 14:20:40 +02:00
  • 6fdf4e07e1 Better log in query-reindex Ivan Skytte Jørgensen 2017-07-17 14:14:12 +02:00
  • 762aa513b4 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-14 17:26:50 +02:00
  • e9a2358c29 RdbTree::checkTree(): check spiderrequetsts correctly Ivan Skytte Jørgensen 2017-07-14 17:17:58 +02:00
  • 6ee77a706a Updated .gitignore to ignore libced.so and libcld3.so Ivan Skytte Jørgensen 2017-07-14 15:36:15 +02:00
  • 502e666a26 Use correct term-freq-weight for bigrams and synonyms Ivan Skytte Jørgensen 2017-07-14 14:32:02 +02:00
  • ef7db99106 bugfix: When reciging msg4 for rdb*2 (repair/rebuild) databases then back off properly until Repai.cpp has been initialized Ivan Skytte Jørgensen 2017-07-14 14:26:12 +02:00
  • 830bb32708 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-14 11:09:13 +02:00
  • e1720775b8 Optimization to get NS record from A reply. More logs Ai Lin Chia 2017-07-13 17:17:07 +02:00
  • 7f941de995 Include termFrequencyWeight in XML and JSON results Ivan Skytte Jørgensen 2017-07-13 17:06:27 +02:00
  • 5257d096c3 Update c-ares to new commit Ai Lin Chia 2017-07-13 16:57:39 +02:00
  • b5728fe030 Used a clearer variable name that 'wts' which was used for either wiki phrase weight or plain score Ivan Skytte Jørgensen 2017-07-13 16:23:58 +02:00
  • abc04b810b Moved decl. for locals score and dist closer to first use Ivan Skytte Jørgensen 2017-07-13 15:33:26 +02:00
  • 93e03bd698 Fixed compilation error Ai Lin Chia 2017-07-13 15:17:44 +02:00
  • 016b9cab1c Merge branch 'master' into staging Ai Lin Chia 2017-07-13 15:03:56 +02:00
  • 3a58bff1f0 Extend log messages. Use status instead of parse_status Ai Lin Chia 2017-07-13 15:02:32 +02:00
  • ca259b873c bugfix b783587429 when term(s) doesn't exist Ivan Skytte Jørgensen 2017-07-13 14:13:42 +02:00
  • 5afc4d43f0 Merge branch 'master' into staging Ai Lin Chia 2017-07-12 14:58:42 +02:00
  • dea56cd23e Pass ARES error code to XmlDoc Ai Lin Chia 2017-07-11 16:03:58 +02:00
  • 807bb67ef8 Free allocated memory Ai Lin Chia 2017-07-11 16:03:11 +02:00
  • 2ec4937749 Don't exit from thread with select error Ai Lin Chia 2017-07-11 16:02:21 +02:00
  • 132b0cc68e Check url/dns block list when url is redirected Ai Lin Chia 2017-07-11 15:26:13 +02:00
  • 6f5595528d Remove declared but not defined functions Ai Lin Chia 2017-07-11 15:00:12 +02:00
  • 3e243bb1ce Modify trace logs to use g_conf.m_logTraceDns for GbDns Ai Lin Chia 2017-07-11 11:55:00 +02:00
  • 2ef35d1d7e Add reload of dns server settings Ai Lin Chia 2017-07-11 11:39:34 +02:00
  • afffa35e56 Merge branch 'master' into dev-dns Ai Lin Chia 2017-07-10 23:02:56 +02:00
  • cd2ac85f00 Modify logs, add more trace logs Ai Lin Chia 2017-07-10 22:57:56 +02:00
  • 558f2d8052 Factored out pair-score-matrix to a local class Ivan Skytte Jørgensen 2017-07-10 14:47:41 +02:00
  • 72bdd3e4a1 Double up m_dir size. Jenkins workspace could have dir longer than 128 Ai Lin Chia 2017-07-10 13:30:12 +02:00
  • b783587429 Factored out find-first-body-entry in PosdbTable::getMinTermPairScoreSlidingWindow() Ivan Skytte Jørgensen 2017-07-10 13:19:28 +02:00
  • e03945c1f6 Move include to top of file Ai Lin Chia 2017-07-10 13:01:09 +02:00
  • a618cb6a68 Link c-ares in unittest Ai Lin Chia 2017-07-10 13:00:44 +02:00
  • eaaf688a8b Add libcares as dependency to GbDns so we make sure that required files are there before compiling GbDns.cpp Ai Lin Chia 2017-07-10 12:49:53 +02:00
  • bcfd05c8ce Fix include criteria for c-ares Ai Lin Chia 2017-07-10 12:15:07 +02:00
  • b830574fbc moved two-term logic from getMaxPossibleScore() to seaprate function Ivan Skytte Jørgensen 2017-07-07 17:39:03 +02:00
  • 4c5d710334 Make code clearer by eliminating noop-if and information-sparse comment Ivan Skytte Jørgensen 2017-07-07 17:29:46 +02:00
  • 3a16202418 Changed how and when QueryTermInfo::m_termFreqWeight is set Ivan Skytte Jørgensen 2017-07-07 16:08:31 +02:00
  • 94977b2ac2 spiderdb dump would dump errcodes from last reply for all following requests, meaning it would dump invalid errcounts for unspidered requests Brian Rasmusson 2017-07-07 16:43:13 +02:00
  • f66c9f6edb spiderdb dump would dump errcodes from last reply for all following requests, meaning it would dump invalid errcounts for unspidered requests Brian Rasmusson 2017-07-07 16:43:13 +02:00
  • 085f861249 Added comment about why QueryWord::m_phraseLen may be less than the lengthy of the source phrase due to non-alfanum words Ivan Skytte Jørgensen 2017-07-07 14:43:14 +02:00
  • 22d2a83d34 Rewrite API-like queries Ivan Skytte Jørgensen 2017-07-07 14:19:45 +02:00
  • 80b04d5e29 Fix indentation Ivan Skytte Jørgensen 2017-07-07 14:01:46 +02:00