Mirrors/privacore-open-source-search-engine
1
0
Fork 1
mirror of https://github.com/privacore/open-source-search-engine.git synced 2025-06-22 23:36:08 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • ed072b22bc Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-03 13:11:06 +01:00
  • d81d96d0a2 Fix PageReindex broken in 37483e550d Ai Lin Chia 2017-03-03 13:10:27 +01:00
  • 1da1310893 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-03 13:07:13 +01:00
  • 8a5a906f4e Added more safety checks in Url::isAdult() Ivan Skytte Jørgensen 2017-03-03 13:06:19 +01:00
  • 4722056eaa Remove privacore specific tld checks Ai Lin Chia 2017-03-03 11:34:05 +01:00
  • f4b8a32c14 Merge branch 'master' into nomerge2 Ai Lin Chia 2017-03-03 11:13:10 +01:00
  • 502a063315 Disable tagdb cache for now Ai Lin Chia 2017-03-03 10:40:33 +01:00
  • 6c75ce7c49 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-03-02 13:30:54 +01:00
  • 10791220cd Log more debug in query parsing Ivan Skytte Jørgensen 2017-03-02 13:30:31 +01:00
  • 0a18383465 Fix compilation when valgrind is enabled Ivan Skytte Jørgensen 2017-03-02 13:13:36 +01:00
  • 5de8c9f29c Log with ERR when detecting query inconsistencies Ivan Skytte Jørgensen 2017-03-02 13:10:50 +01:00
  • 8ca6dc1e47 Fix get_titlerec Ai Lin Chia 2017-03-02 12:18:56 +01:00
  • 93a81a2fed Add support of tld to UrlBlockList Ai Lin Chia 2017-03-02 10:38:41 +01:00
  • 56915e22ea Cater for multiple criteria types for UrlBlockList to speed things up Ai Lin Chia 2017-03-01 15:37:31 +01:00
  • 7f044bd0e8 Move dedup spiderdb to cpu thread Ai Lin Chia 2017-03-01 14:22:21 +01:00
  • 057b417137 Remove old posdb test. Now we always use index for posdb Ai Lin Chia 2017-03-01 14:20:24 +01:00
  • d7355f9cd4 Fix unit test compilation error Ai Lin Chia 2017-03-01 14:14:30 +01:00
  • f7b7c8572d Move some string util function to GbUtils Ai Lin Chia 2017-03-01 14:11:39 +01:00
  • 0d7f92e9a8 Remove urlblocklist test based on real url Ai Lin Chia 2017-03-01 14:09:50 +01:00
  • 67dc4b01de Use correct part of URL for storing tags Ivan Skytte Jørgensen 2017-02-28 16:08:32 +01:00
  • 5f50290074 set html form input field width based on value range Ivan Skytte Jørgensen 2017-02-28 14:20:15 +01:00
  • c7e04d100a Check for erros when finishing merge Ivan Skytte Jørgensen 2017-02-28 13:45:51 +01:00
  • 5762d829cc bugfix typo in parm units display Ivan Skytte Jørgensen 2017-02-28 13:35:40 +01:00
  • a5f5c3a2b1 Don't filter out control characters when logging Ivan Skytte Jørgensen 2017-02-28 12:53:08 +01:00
  • 98b1dbacf6 Don't use eg. '...in milliseconds' in parameter descriptions. Just use Parm::m_units Ivan Skytte Jørgensen 2017-02-27 18:09:59 +01:00
  • 4e2f88c599 Show Parm::m_units Ivan Skytte Jørgensen 2017-02-27 17:51:41 +01:00
  • fde3bc3947 Group cpu/thread/job parameters Ivan Skytte Jørgensen 2017-02-27 17:08:51 +01:00
  • 1a467b9231 iOptionally check if job clenaup takes too long Ivan Skytte Jørgensen 2017-02-27 17:06:23 +01:00
  • 07b8a510f4 Simplified code in Loop::doPoll() a bit Ivan Skytte Jørgensen 2017-02-27 16:37:40 +01:00
  • 6cc4142541 Docuemnt restrictions on /tmp/gb_merge_space more clearly in Parms.cpp Ivan Skytte Jørgensen 2017-02-27 16:15:38 +01:00
  • a7ab0fd0c7 Demote logging ERR to WRN when rollback-move fails when unlinking non-existing destination file Ivan Skytte Jørgensen 2017-02-27 15:42:24 +01:00
  • 12a2d98b2e Removed unused fields from Tagdb.cpp:State12 (m_isLocal+m_mergeTags) Ivan Skytte Jørgensen 2017-02-27 14:03:01 +01:00
  • 05c17400f5 temporarily remove spiderdb cleanup code running in the main thread when merging Brian Rasmusson 2017-02-24 20:10:20 +01:00
  • bc94a03ce6 Add check for posdb neg key Ai Lin Chia 2017-02-24 15:37:19 +01:00
  • 6b1bec5054 Don't log timestamp when the log system already does that Ivan Skytte Jørgensen 2017-02-24 14:49:46 +01:00
  • c948b39102 When logging first 1024 bytes of an incoming HTTP request indicate if the log chunk is truncated Ivan Skytte Jørgensen 2017-02-24 14:47:05 +01:00
  • 51484abe43 spider-priority drowndown in URL filters didn't work Ivan Skytte Jørgensen 2017-02-24 14:28:51 +01:00
  • 7bda4d8986 Log when leftover merge files are found in merge dir Ivan Skytte Jørgensen 2017-02-24 13:56:54 +01:00
  • e2789a436b Update README.md Brian Rasmusson 2017-02-24 11:19:22 +01:00
  • b33b458273 Split out function prepareBuffer in Msg4 Ai Lin Chia 2017-02-23 17:07:42 +01:00
  • 14fc613460 Print the hidden parameter for ./gb dump as well Ai Lin Chia 2017-02-23 14:32:58 +01:00
  • 7ae45ba10e Fix url-classification shutdown when it hasn't been initialized Ivan Skytte Jørgensen 2017-02-23 14:19:25 +01:00
  • b72ff2fdaa Clear g_collectiondb before exiting when dumping - otherwise destructos end up using destroyed data and mutexes Ivan Skytte Jørgensen 2017-02-23 14:08:29 +01:00
  • a7efe6f280 More encoding fixes for query-stopwords (german/italian/norwegian) Ivan Skytte Jørgensen 2017-02-23 13:53:56 +01:00
  • 6fcf63b770 Fix trace log. 2 variables were swapped. Use logError instead to log errors Ai Lin Chia 2017-02-22 15:24:28 +01:00
  • 53193748c1 Remove unused input args from storeRec Ai Lin Chia 2017-02-22 14:32:57 +01:00
  • 26b98e07cd Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2017-02-23 10:15:21 +01:00
  • 4e54def10a Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2017-02-23 10:14:32 +01:00
  • 2620ea3031 fix data corruption bug when deleting an RdbBucket Brian Rasmusson 2017-02-23 10:13:09 +01:00
  • ebacf6aef7 Moved m_memoryLabelPtrs initialization. We are using it in load as well. Ai Lin Chia 2017-02-22 15:15:28 +01:00
  • ed74e4dba3 Remove commented out codes Ai Lin Chia 2017-02-20 15:06:07 +01:00
  • 2cb61e64db Remove unused MAX_MACHINES define Ai Lin Chia 2017-02-20 15:05:35 +01:00
  • b2d743a135 RdbBucket::getFirstKey() should return const Ivan Skytte Jørgensen 2017-02-21 11:59:43 +01:00
  • 621fa0cbfc Fix deadlock in SpiderColl::evalIpLoop Ai Lin Chia 2017-02-21 11:39:13 +01:00
  • 64bc15158b Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-02-20 20:39:50 +01:00
  • f29c6aaf8b bugfix clusterign allocatign 0 records and expecing non-null pointer from mmalloc() Ivan Skytte Jørgensen 2017-02-20 20:39:33 +01:00
  • 696b7c5ab2 Merge branch 'master' of github.com:privacore/open-source-search-engine Ivan Skytte Jørgensen 2017-02-20 18:13:21 +01:00
  • 26ae00bb79 Removed Query.* FIELD_QUOTA and FIELD_GBSECTIONHASH Ivan Skytte Jørgensen 2017-02-20 17:16:42 +01:00
  • eab379f3b5 Merge branch 'master' into nomerge2 Ivan Skytte Jørgensen 2017-02-20 16:09:13 +01:00
  • 4a29cca774 Mark bigrams generated from two highfreqterms in 2-word queries as required. Ivan Skytte Jørgensen 2017-02-20 16:07:25 +01:00
  • 2fa935b0e3 Fix bug with RdbIndexQuery::documentIsInFile where it doesn't check m_treeIndexData when fileNum is not m_numFiles Ai Lin Chia 2017-02-20 16:04:05 +01:00
  • 63bae5c001 Made QueryTerm::m_qword const Ivan Skytte Jørgensen 2017-02-20 15:12:17 +01:00
  • b022163676 Fix compilation error Ai Lin Chia 2017-02-20 14:51:16 +01:00
  • 1186e89585 Remove redundant cast Ai Lin Chia 2017-02-20 12:51:28 +01:00
  • 61c51ae74b Use log warn for warning messages Ai Lin Chia 2017-02-20 12:51:13 +01:00
  • 7a26e134d1 Remove commented out code Ai Lin Chia 2017-02-20 12:50:57 +01:00
  • f61099a764 Remove unused input args from Msg0::getList Ai Lin Chia 2017-02-20 12:48:09 +01:00
  • 041d01ef32 Remove commented out code Ai Lin Chia 2017-02-20 12:27:47 +01:00
  • 3940721eb5 Remove always true m_sendToSelf variable Ai Lin Chia 2017-02-20 12:27:33 +01:00
  • 5b67f59680 Unwrap log text in Query.cpp for easier grepping Ivan Skytte Jørgensen 2017-02-20 14:58:48 +01:00
  • 20f9632981 More const in Query.cpp Ivan Skytte Jørgensen 2017-02-20 14:55:15 +01:00
  • 7226b146da Removed unhelpful comments Ivan Skytte Jørgensen 2017-02-20 14:38:06 +01:00
  • 649624d3df Changed scope of 'list' local variable so it isn't use for different thigns all over the place Ivan Skytte Jørgensen 2017-02-20 14:25:06 +01:00
  • e6c0f8d4fa bugfix msg2 site-whitelist handling in rdb tree (posdb rdbbuckets) Ivan Skytte Jørgensen 2017-02-20 11:41:17 +01:00
  • 03db913cdc Made Mem more private Ivan Skytte Jørgensen 2017-02-19 16:41:55 +01:00
  • d1da8bd20a Mem::dup() and mdup() should return void* instead of char* Ivan Skytte Jørgensen 2017-02-19 16:38:06 +01:00
  • eea835c9b2 Handle no content in HttpServer::sendReply2() Ivan Skytte Jørgensen 2017-02-19 16:28:21 +01:00
  • 7047d4a1be Added make.depend to .gitignore Ivan Skytte Jørgensen 2017-02-19 16:21:17 +01:00
  • e82156bc67 Fix (unlikely) error in Msg13 for spidering if all hosts aer down Ivan Skytte Jørgensen 2017-02-19 16:20:04 +01:00
  • 092cfe6e25 Removed unused method Mem::strdup() Ivan Skytte Jørgensen 2017-02-19 15:25:59 +01:00
  • ac6619b70d Fix typo in parm description Ivan Skytte Jørgensen 2017-02-19 15:07:24 +01:00
  • 9ce07a3772 Stop returning (void *)0x7fffffff for zero-sized allocations Ivan Skytte Jørgensen 2017-02-19 15:03:54 +01:00
  • e33deafa0b Removed checks for low address in Mem::gbmalloc() Ivan Skytte Jørgensen 2017-02-18 23:05:23 +01:00
  • 56762d25b7 Dropped Mem.cpp:freeCacheMem() Ivan Skytte Jørgensen 2017-02-18 23:03:13 +01:00
  • 0ba68cedd5 Merge remote-tracking branch 'origin/master' into nomerge2 Brian Rasmusson 2017-02-18 21:12:37 +01:00
  • 86fb16e5fb Verify result from Query::set2() (coverity complained) Ivan Skytte Jørgensen 2017-02-17 14:42:26 +01:00
  • 1d4c05b652 constness in images.* Ivan Skytte Jørgensen 2017-02-17 14:38:04 +01:00
  • b2c43dec63 Check for #terms in harcoded query (make coverity happy) Ivan Skytte Jørgensen 2017-02-17 14:28:58 +01:00
  • 63392dd835 Add skiphash as parameter for injecting document so we can force a reindex even when content is the same Ai Lin Chia 2017-02-17 12:49:35 +01:00
  • 19db02a16e Changed Query::m_gbuf/m_gnext/m_qwordsAllocSize into a simple SmallBuf<> Ivan Skytte Jørgensen 2017-02-16 17:54:42 +01:00
  • 77f48c0bf0 Add langId as parameter for injecting document so we can override the document language Ai Lin Chia 2017-02-16 17:26:14 +01:00
  • 6ad9429779 Optimize when URL realtime classification is down or disabled Ivan Skytte Jørgensen 2017-02-16 16:47:29 +01:00
  • 44d6b5d202 Merge branch 'master' of github.com:privacore/open-source-search-engine Ivan Skytte Jørgensen 2017-02-16 16:24:29 +01:00
  • 7c24ce7dd0 Fix memory leak of msg8astate Ai Lin Chia 2017-02-16 14:40:22 +01:00
  • 4ff266570c Code style changes Ai Lin Chia 2017-02-16 14:39:26 +01:00
  • 5c35d99967 Remove unused inject variables Ai Lin Chia 2017-02-16 14:39:03 +01:00
  • 97e336fd07 Remove commented out code Ai Lin Chia 2017-02-16 11:26:42 +01:00
  • 2cf1140d7b Use logError instead of log Ai Lin Chia 2017-02-16 11:26:25 +01:00
  • 3c8a5135cd Renamed Query::m_stackBuf to something sensible Ivan Skytte Jørgensen 2017-02-15 22:40:26 +01:00
  • 62d96d5fdf Encapsulate Query more Ivan Skytte Jørgensen 2017-02-15 22:39:06 +01:00