Forks/privacore-open-source-search-engine
1
0
Fork 0
forked from Mirrors/privacore-open-source-search-engine
Code Pull Requests Activity
12,475 Commits 7 Branches 0 Tags 254 MiB
master
Commit Graph

37 Commits

Author SHA1 Message Date
Ivan Skytte Jørgensen
beeddcf35d Got rid of gb-include.h 2018-07-26 17:29:51 +02:00
Ai Lin Chia
26f4d7b1be Make RdbMap::m_numPositiveRecs & RdbMap::m_numNegativeRecs atomic 2017-05-04 15:42:59 +02:00
Ai Lin Chia
df3f33e52c Add more trace logs 2017-03-10 13:32:20 +01:00
Ivan Skytte Jørgensen
5825378da5 Use merge-space while merging 
No more BigFile .part* deletion during a merge to preserve disk space. Instead MergeSpaceCoordinator is used for coordinating access to a large and possibly cheap storage with room for a whole resulting mergefile.
When a mrge file has been finished the reads are allowed from that and reads from the source files disallowed, which are then deleted. Then the file is renamed/moved from merge-space to regular collection storage using the 2-phase commit feature of GbMoveFile.cpp, and finally reads are done from the finished file.

Details:
  RdbBase: Use MergeSpaceCoordinator and merge space for temporary target merge file.
  RdbBase: better cleanup of crashed merges
  RdbBase: more mutex locing while manipulatin m_fileInfo array
  RdbBase: keep track of thraeds/jobs
  RdbMerge: ditto
  RdbMerge: Dont call file->chopHead()
  Msg5/Msg3: no more "compensate for merge" flag
  Msg3: Skip over RdbBase files that have reads disallowed
 2016-10-31 18:16:40 +01:00
Ivan Skytte Jørgensen
58b9015fcd Removed 'force' functionalit yon file rename operations 
In normal circumstances the destination file should not exist. spidering/merging/querying works fine. recovery logic may have to get fixed too.
 2016-10-25 16:12:17 +02:00
Ivan Skytte Jørgensen
5eab079963 Removed 'force' parameter from BigFile::rename (variant 2) 2016-10-25 15:06:06 +02:00
Ivan Skytte Jørgensen
02fa090f37 Removed default parameter from RdbMap::rename 2016-10-25 14:40:35 +02:00
Ivan Skytte Jørgensen
26c5d7b2d8 Removed 'force' parameter from RdbMap::rename() (always true) 2016-10-25 13:02:55 +02:00
Ivan Skytte Jørgensen
e4afd2320f Removed non-const verions of getFilename() in BigFile/RdbIndex/RdbMap 2016-10-24 16:03:38 +02:00
Ivan Skytte Jørgensen
ba81b7a7d6 Use proper English: alloced -> allocated 2016-10-17 11:57:37 +02:00
Ivan Skytte Jørgensen
9c384d2651 Fix accidental recursive call 2016-10-11 12:09:38 +02:00
Ivan Skytte Jørgensen
fccf4e4c7e Removed commented-out code from RdbMap 2016-10-10 17:09:59 +02:00
Ivan Skytte Jørgensen
6b55f31a44 More constness in RdbMap 2016-10-10 17:02:38 +02:00
Ai Lin Chia
5a8736a41d Remove unused prototype 2016-09-09 23:31:18 +02:00
Ai Lin Chia
a040a78c99 Use key96_t instead of key_t and redefining std lib key_t (which breaks std lib functionality that uses key_t) 2016-09-02 14:49:06 +02:00
Ai Lin Chia
9b41b7a614 Remove now unused RdbMap::m_generatingMap variable 2016-08-31 11:13:31 +02:00
Ai Lin Chia
0256f76c18 Remove suspicious looking code that fails addRecord once a second when there is error. 2016-08-31 11:13:31 +02:00
Ivan Skytte Jørgensen
5449f42a56 Rearranged+removed unrechable code following gbshutdownAbort() in RdbMap::setKey 2016-08-16 00:08:00 +02:00
Ivan Skytte Jørgensen
56409fd0dd constness in RdbMap 2016-08-01 17:08:53 +02:00
Ai Lin Chia
4bab41e32a Remove commented out MAX_SEGMENTS code 2016-07-19 14:50:48 +02:00
Ai Lin Chia
f86e945d49 Add force/unforced rename and fix bug where first file was renamed 2016-07-05 16:54:02 +02:00
Ai Lin Chia
bd9ff20d78 Fix unreachable-code warnings 2016-06-28 11:48:30 +02:00
Brian Rasmusson
780696c0d1 More emergency shutdown streamlining 2016-06-20 19:29:10 +02:00
Ai Lin Chia
8dab0db467 Remove extra semicolons 2016-05-19 18:37:26 +02:00
Ai Lin Chia
db94bf618c Add PRIVACORE_TEST_VERSION defines which reduces the part/page size for testing 2016-05-11 13:45:40 +02:00
Ivan Skytte Jørgensen
0b4bded117 More constness in RdbMap.* 2016-04-14 18:18:24 +02:00
Ivan Skytte Jørgensen
69b13e2bca Removed unused RdbMap::addIndexList 2016-04-12 15:47:59 +02:00
Ai Lin Chia
ab0b9d03ea Standardize header guards 2016-03-08 22:16:02 +01:00
Matt
f6207b75c7 Merged 6e12f96aeab29778b3b6ba64c5fe594e7e6989cd 2015-11-20 12:01:09 +01:00
Matt Wells
74ec812959 try to fix core from adding a file that already exists. 
just return an error now. hopefully merge will try again later.
also core if you try to write recs to an rdbmap that
has already had its memory footprint reduced so we can find
that overrun bug.
 2015-08-21 14:00:40 -07:00
Matt Wells
840ca3fea1 fix rdbmap reduce mem thing 2015-08-08 15:43:09 -07:00
Matt
adcef39376 Merge branch 'diffbot-testing' into diffbot-matt 
Conflicts:
	Collectiondb.cpp
	Collectiondb.h
	Conf.cpp
	Conf.h
	Msg39.cpp
	PageEvents.cpp
	PageResults.cpp
	PageTurk.cpp
	Pages.cpp
	Parms.cpp
	Posdb.cpp
	Proxy.cpp
	Query.cpp
	Query.h
	RdbBase.cpp
	RdbMap.cpp
	Repair.cpp
	Repair.h
	SafeBuf.cpp
	Spider.cpp
	Tagdb.cpp
	TopTree.cpp
	XmlDoc.cpp
	main.cpp
 2014-11-20 16:53:07 -08:00
Matt
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
Matt Wells
444ed14cde reduce mem usage in rdbmap. useful 
for when there are thousands of tiny collections.
 2014-11-07 08:49:08 -08:00
Matt Wells
e7dd8f7956 replace long long with int64_t 2014-10-30 13:36:39 -06:00
Matt Wells
4e803210ee tons of changes from live github on neo. 
lots of core fixes.
took out ppthtml powerpoint convert, it hangs.
dynamic rdbmap to save memory per coll.
fixed disk page cache logic and brought it
back.
 2014-01-17 21:01:43 -08:00
Matt Wells
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00