379 Commits

Author SHA1 Message Date
2e7412abbd Use rwlock for RdbBuckets 2017-06-06 17:15:48 +02:00
3c621c4f31 Add debug-test as target 2017-05-10 16:25:46 +02:00
1d769203e7 Merge branch 'nomerge2' 2017-05-08 11:59:43 +02:00
1c117d3aa1 Dropped PingInfo + PingServer entierely.
The remaining functionality of the pingserver was to ping normalt and "shotgun" IPs to determine the fastests and working network path. This is better done at L2/L3 or OS level - not in the application.

The email-sending functionality was dropped too. We recommend using 3rd-party tools for monitoring and statistics-collection/triggering.
2017-05-05 15:00:31 +02:00
61dc9e4573 Keep our own Host::m_flags up-to-date 2017-05-04 14:37:19 +02:00
d3ecbda478 Use Vagus for some host/instnace information
hosts.conf CRC and total_docs_indexed is now exchanged over Vagus. rest will follow in separate commits.
The vagus cluster identifier defaults to "gb-"$USER
2017-05-01 17:21:34 +02:00
97a480a0f6 Merge branch 'master' into nomerge2 2017-04-18 14:12:38 +02:00
53e89c2d15 Revert watchdog functionality.
Reverting commit:
	00d91dc2e3
	a57aa86bb6

The watchdog functionality worked fine but added a non-neglible background load on idle setups and scaled just as poorly as the pinginfo. We need a better mechanism.
2017-04-18 14:10:45 +02:00
211b210ccb Merge branch 'master' into nomerge2 2017-04-10 13:51:36 +02:00
c86745045d Removed Msg1f / remote log fetching
Program logs too much for last 20KB log of being any use. Viewing log in a web interface is better done with a 3rd-party log thing (logstash/syslog-ng/fluentd/...)
2017-04-10 13:47:33 +02:00
50b656f93a Merge branch 'master' into nomerge2 2017-04-10 12:38:13 +02:00
766facf75d Remove blaster/blasterdiff command that seems to be processing gb output log and injecting url. We should do that outside of gb and use the admin/inject interface if needed. 2017-04-06 19:16:39 +02:00
a57aa86bb6 Send watchdog messge to hosts we haven't interacted with recently.
Distinquish between hotss we normalyl talk to and hosts we rarely talk to. Use agressive conditional watchdog messages, and detect unresponsive hosts based on that.
2017-04-06 17:18:56 +02:00
813c4e2350 Remove now unused Timezone.cpp 2017-04-06 14:45:48 +02:00
8ae62f5e22 Move SpiderCache to separate file.
Move spider page related code to PageSpider.cpp
2017-04-05 16:13:51 +02:00
524cd5ad85 Remove now unused Msg1 2017-04-04 16:22:54 +02:00
68ac6ed451 Add debug-safe, debug-test config target 2017-04-03 20:55:24 +02:00
75db44d81d Merge branch 'master' into nomerge2 2017-03-27 15:05:27 +02:00
7f3a5f4cb8 Re-add warning suppression 2017-03-27 15:05:10 +02:00
7a21587ca6 Merge branch 'master' into nomerge2 2017-03-27 15:03:19 +02:00
dad3e273b8 Move clang++ if to before g++. clang++ matches g++ as well. 2017-03-27 14:54:16 +02:00
213cb460cd Merge branch 'master' into nomerge2 2017-03-22 20:46:44 +01:00
1778d66315 Remove statsdb 2017-03-22 20:46:28 +01:00
f27190e251 Merge branch 'master' into nomerge2 2017-03-17 15:12:58 +01:00
98192ba79d Move generateGlobalIndex to thread 2017-03-17 14:58:26 +01:00
d9b47779cc Merge 2017-03-16 17:21:44 +01:00
e90e1d210b Split Msg4.* into Msg4In and Msg4Out 2017-03-16 16:04:27 +01:00
aebc2e8ff8 Cater for multiple criteria types for UrlBlockList to speed things up 2017-03-06 16:12:43 +01:00
56915e22ea Cater for multiple criteria types for UrlBlockList to speed things up 2017-03-01 15:37:31 +01:00
ac875d9e15 Merge branch 'master' into nomerge2 2017-02-09 12:19:55 +01:00
1def0b382a Add thin wrapper around pcre.h 2017-02-07 16:38:54 +01:00
484d4e6ae7 Added real-time URL classification (not used yet from msg40) 2017-02-07 16:22:49 +01:00
adc20e3129 Fix timeing bug in parallele make of Version.o 2017-02-07 16:00:03 +01:00
728046f472 Make sure warning suppression work with custom version of g++ (eg: on travis) 2017-02-06 15:44:27 +01:00
5c329eed42 Fix makefile to allow for custom CXX in version check 2017-02-06 15:26:27 +01:00
538c75b205 Merge branch 'master' into nomerge2
Conflicts:
	Parms.cpp
	main.cpp
2017-01-25 20:29:30 +01:00
d725ded930 Refactor scoring weights into a struct 2017-01-20 16:27:16 +01:00
8fea89b797 Add url block list 2017-01-18 17:17:33 +01:00
55a91ea697 Merge branch 'master' into nomerge2
Conflicts:
	Makefile
	Msg2.cpp
	Msg40.cpp
2017-01-09 14:25:37 +01:00
91c24650f5 Moved gbcompress/gbuncompress from XmlDoc.* to separate file 2017-01-09 14:00:34 +01:00
7cb999330b Merge branch 'master' into nomerge2
Conflicts:
	PosdbTable.cpp
2017-01-06 17:11:24 +01:00
d541968da7 Load docid2flagsandsitemap.dat 2017-01-06 17:06:33 +01:00
e003f35eab Merge branch 'master' into nomerge2 2016-12-19 15:01:31 +01:00
08b20fddce Prep. for page temperature in ranking 2016-12-19 12:24:44 +01:00
5a06b25687 Merge branch 'master' into nomerge2
Conflicts:
	Msg2.cpp
	Msg3.cpp
	Msg39.cpp
	Msg51.cpp
	RdbList.cpp
2016-11-03 17:36:50 +01:00
ca0f5d52e6 Disable gcc suggestion warnings. 2016-11-01 12:35:16 +01:00
9e07acbce9 changed Makefile so config=release-safe must be used to disable features unsafe for our production environment 2016-10-31 20:54:21 +01:00
5bad84aa52 Merge branch 'master' into nomerge2
Conflicts:
	Msg3.cpp
	Msg39.cpp
	Rdb.cpp
	RdbCache.h
	XmlDoc.cpp
2016-10-31 11:31:03 +01:00
9454e3df02 Added moveFile2Phase1()/moveFile2Phase2() 2016-10-24 15:15:35 +02:00
a93e34e085 Added makePath() function 2016-10-21 13:21:49 +02:00