Forks/open-source-search-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

do not allow custom crawls to spider the web any more.

Browse Source
This commit is contained in:
Matt Wells 2014-02-26 10:26:09 -08:00
parent 6716d8f21b
commit a0697e1bb5
1 changed files with 7 additions and 5 deletions

12
Collectiondb.cpp

@ -2137,11 +2137,13 @@ bool CollectionRec::rebuildUrlFilters ( ) {
i++;
// 2nd default filter
if ( m_restrictDomain ) {
m_regExs[i].set("!isonsamedomain && !ismanualadd");
m_spiderPriorities [i] = SPIDER_PRIORITY_FILTERED;
i++;
}
// always turn this on for now. they need to add domains they want
// to crawl as seeds so they do not spider the web.
//if ( m_restrictDomain ) {
m_regExs[i].set("!isonsamedomain && !ismanualadd");
m_spiderPriorities [i] = SPIDER_PRIORITY_FILTERED;
i++;
//}
m_regExs[i].set("errorcount>=1 && !hastmperror");
m_spiderPriorities [i] = 15;