Forks/open-source-search-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

need to allow clients to not restrict to

Browse Source 
seed domains.
This commit is contained in:
Matt Wells 2014-02-26 22:27:22 -08:00
parent 927f4626ee
commit af9eb8fb73
1 changed files with 8 additions and 5 deletions

13
Collectiondb.cpp

@ -2139,11 +2139,14 @@ bool CollectionRec::rebuildUrlFilters ( ) {
// 2nd default filter
// always turn this on for now. they need to add domains they want
// to crawl as seeds so they do not spider the web.
//if ( m_restrictDomain ) {
m_regExs[i].set("!isonsamedomain && !ismanualadd");
m_spiderPriorities [i] = SPIDER_PRIORITY_FILTERED;
i++;
//}
// no because FTB seeds with link pages that link to another
// domain. they just need to be sure to supply a crawl pattern
// to avoid spidering the whole web.
if ( m_restrictDomain ) {
m_regExs[i].set("!isonsamedomain && !ismanualadd");
m_spiderPriorities [i] = SPIDER_PRIORITY_FILTERED;
i++;
}
m_regExs[i].set("errorcount>=1 && !hastmperror");
m_spiderPriorities [i] = 15;