Forks/open-source-search-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

do not give proxy for diffbot to use just yet.

Browse Source 
need to fix https CONNECT support to download the
whole page first and then send that back. need to
tell diffbot phantomjs to not worry about certificates
then.
This commit is contained in:
Matt 2015-03-05 15:32:57 -08:00
parent 367c4d5783
commit ccd85ada31
1 changed files with 7 additions and 0 deletions

7
XmlDoc.cpp
View File

@ -15237,6 +15237,13 @@ SafeBuf *XmlDoc::getDiffbotReply ( ) {
if ( cr->m_forceUseFloaters ) useProxies = true;
// we gotta have some proxy ips that we can use
if ( ! g_conf.m_proxyIps.hasDigits() ) useProxies = false;
// until we fix https CONNECT support for https urls diffbot can't
// go through gb. we should fix that by downloading the whole page
// ourselves and sending it back, and tell diffbot's phantomjs not
// to do the certificate check.
useProxies = false;
// if we used a proxy to download the doc, then diffbot should too
// BUT tell diffbot to go through host #0 so we can send it to the
// correct proxy using our load balancing & backoff algos.