Forks/open-source-search-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

have diffbot retry non tmp errors to make up

Browse Source 
for bug of calling valid urls malformed EBADURLs
This commit is contained in:
Matt Wells 2015-12-14 17:30:41 -08:00
parent 4b9240e42e
commit e1cfeb4c82
1 changed files with 4 additions and 3 deletions

7
Collectiondb.cpp

@ -3794,10 +3794,11 @@ bool CollectionRec::rebuildUrlFiltersDiffbot() {
m_regExs[i].set("errorcount>=1 && !hastmperror");
m_regExs[i].set("errorcount>=1 && !hastmperror && errcount<=4");
m_spiderPriorities [i] = 15;
m_spiderFreqs [i] = 0.0;
m_maxSpidersPerRule [i] = 0; // turn off spiders if not tmp error
m_spiderFreqs [i] = 0.1;
m_maxSpidersPerRule [i] = 1; // turn off spiders if not tmp error
// no, try to fix bug of EBADURL when it wasn't really...
i++;
// and for docs that have errors respider once every 5 hours