dfc069aaa1
do away with filtered/banned spider priorities. add checkbox to signify force deletes to remove urls from index if in the index, or not allow them in.
mwells2015-03-17 20:27:23 -06:00
dea534827e
langidbits init bug leftover from searchinput reset memset fix i think.
Matt2015-03-17 15:04:31 -06:00
f830eb43f7
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
Matt2015-03-17 14:31:33 -06:00
a54471849b
sitemap.xml support for harvesting loc urls. parse xml docs as pure xml again but set nodeid to TAG_LINK etc. so Linkdb.cpp can get links again. added isparentsitemap url filter to prioritize urls from sitemaps. added isrssext to url filters to prioritize new possible rss feed urls. added numinlinks to url filters to prioritize popular urls for spidering. use those filters in default web filter set. fix filters that delete urls from the index using the 'DELETE' priority. they weren't getting deleted.
Matt2015-03-17 14:26:16 -06:00
29b2707ad7
fix searchinput::clear() bug. final fix for fhtqt memleak bug.
Matt Wells2015-03-15 07:48:29 -07:00
3b39b1d37a
fix facet mem leak from QueryTerm::m_facetHashTable and safebuf when doing federated queries over a token.
Matt Wells2015-03-15 07:18:32 -07:00
e71af6d26c
recompute active list every 3 secs. otherwise it seems buggy and drops collections it shouldn't.
Matt Wells2015-03-12 22:22:19 -07:00
83be5d7d46
fix links parser so it harvests outlinks from rss feeds' <link> tags. it was doing this before, now it is doing it again.
Matt2015-03-12 17:35:47 -07:00
a8dfa56098
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
Matt Wells2015-03-12 13:19:16 -07:00
485c600c7c
fix for changing maxtocrawl/process/rounds. fix for thinking a crawl is done when it is just taking a while to populate doledb from the waiting tree for that SpiderColl. we just call populateDoledbFromWaiting in doneSleepingWrapper avery 50ms. it loops over every coll so it could be more efficient.
Matt Wells2015-03-12 13:15:52 -07:00
a4e95899bd
makefile updates for building pkgs
Matt2015-03-10 21:43:37 -07:00
8e72d6e4cc
fix a couple critical xml parsing bugs. fixes parsing of rss feeds better and xml in general. fixed qa tests to ignore collection list when doing diff.
Matt2015-03-10 19:13:21 -07:00
052f2b2009
fix for bug #2092 commented out to avoid qa testing for now.
Matt2015-03-10 15:45:33 -07:00
c6fd5571d2
if "links":[ is specified in diffbot reply then crawlbot will parse out those links as if they were on the page.
Matt2015-03-10 14:36:44 -07:00
ef66e1e69f
speed up overflow check for firstip with a little cache thingy. reduce log spam from ssl msgs.
Matt2015-03-10 13:24:03 -07:00
e346a14a47
added logic to retry diffbot reply on connection reset, connection timed out or gateway timed out (http status 504) msgs. added logic to detect truncated json (missing final }) and not print it. also, at index time, we set a diffbot missing curly error to g_errno so the whole url can be retried later.
Matt Wells2015-03-09 20:54:34 -07:00
eccb969e5b
put in some fixes to deal with doledb tree that seems to have m_data[i] and m_data[j] pointing to the same thing. wtf? anyway, deal with that. it should fix the tree or something automatically at startup?
Matt Wells2015-03-08 20:36:13 -07:00
5e651997e7
fixed langid based query stop words.
Matt2015-03-08 15:44:23 -07:00
2413a9b9b1
query stop words now based on selected langid.
Matt2015-03-08 15:16:24 -07:00
a25c358817
fix core dump caused by some corruption of some sort. for deleting json objs associated with a particular url.
Matt Wells2015-03-08 12:18:47 -07:00
4506b93ed3
fixed missing sites in sitelinks.txt
Matt Wells2015-03-05 20:32:01 -08:00
56d65a7c55
adjust dump tagdb cmdline cmd to start at a specified site to aid us in fixing sitelinks.txt missing some sites bug.
Matt Wells2015-03-05 19:27:36 -08:00
ccd85ada31
do not give proxy for diffbot to use just yet. need to fix https CONNECT support to download the whole page first and then send that back. need to tell diffbot phantomjs to not worry about certificates then.
Matt2015-03-05 15:32:57 -08:00
367c4d5783
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
Matt2015-03-05 11:10:54 -08:00
dfd6d8b2cf
fix critical spider bug that was deleting pages because of bogus SpiderReply::m_langId values!
Matt2015-03-05 08:49:39 -08:00
7ae5518c1d
if no spider launched for a collection, decrement priority and try again. if we hit -1 priority then advance to next coll in the linked list. reset priority to max at start of each collection's round.
Matt2015-03-05 08:15:50 -08:00
38caa517f2
add switches to disable injections or querying from the master controls, for all collections.
Matt Wells2015-03-04 10:49:37 -08:00
3fc9abe222
do not show any non-public page if allow cloud users is false and no permission. return 403.
Matt Wells2015-03-02 08:13:38 -08:00
93b505e7bb
fix isCollAdmin() function to return false if not using coll passwords. they'll have to be master admin.
Matt Wells2015-03-02 07:47:05 -08:00
051a8f0ad6
dont attempt merge in quickpoll. just return do not core.
Matt Wells2015-03-02 07:26:38 -08:00
a97f1fcad0
add skippedshards and totalshards to search results in xml/json so you know if you are missing results from a dead shard or not. all hosts must be dead in shard for shard to be dead.
mwells2015-02-27 08:17:32 -07:00
f8db6288ae
ignore dead shards when doing queries so they remain fast.
mwells2015-02-27 08:02:19 -07:00
f5383d98db
if a shard is dead skip it when searching.
Matt2015-02-27 07:28:41 -07:00
bb5e0c9c63
another MAX_DGRAMS fix.
Matt2015-02-27 06:30:17 -07:00
ee672bb3a3
say 'Injected url already indexed' and not 'Injection abandoned' for clarity's sake.
Matt Wells2015-02-26 16:04:51 -07:00
064d022d6f
call mkdir on 'gb install' cmd.
Matt2015-02-25 19:49:36 -07:00
d4f67285ce
keep spiders maxed out all the time.
mwells2015-02-25 18:40:50 -07:00
5ae3ae60d3
Merge branch 'diffbot-testing' of github.com:gigablast/open-source-search-engine into diffbot-testing
mwells2015-02-25 11:40:00 -07:00
4355665fa9
fix ssl handshake hanging forever. time it out now. spider performance fix for when a firstip has massive # of urls and can only have one spidered in the future.
mwells2015-02-25 11:39:13 -07:00
3f753f8114
sometimes tree data is NULL for some reason... fix core.
Matt2015-02-24 20:34:41 -07:00
a5e66da609
fix crawls where the url crawl pattern and page process pattern are supplied, but not the url process pattern. such was the dribbble crawl.
Matt Wells2015-02-24 14:27:38 -07:00
f41877c2e3
compute MIN hop count over all spider requests for the same url in Spider.cpp.
Matt Wells2015-02-24 13:41:18 -07:00
692c2932e8
fixed bug of gb not saving
mwells2015-02-22 13:11:20 -07:00
4e485b6649
increase dolebuf cache time from 2 to 5 mins for better performance. cache empty dolebufs if winner tree list was not from cache, so in case we have a huge spiderdb scan list of urls we aren't spidering we can cache it, like twitter.com e.g. do not call strstr in getUrlFilterNum2() for .css? or /print/ since it was taking way too much cpu time.
mwells2015-02-21 15:17:28 -07:00
a57de3289c
Merge branch 'diffbot' into diffbot-testing
Matt Wells2015-02-21 09:43:33 -08:00
b80a70a6fd
fix for https urls through proxies using newly updated tcp/loop code.
Matt Wells2015-02-21 09:25:54 -08:00
mwells
Matt
Matt Wells