Ivan Skytte Jørgensen
beeddcf35d
Got rid of gb-include.h
2018-07-26 17:29:51 +02:00
Ivan Skytte Jørgensen
9785fec43b
It compiles, links starts and can put seeds into the sqlite database
2017-10-10 15:54:16 +02:00
Ai Lin Chia
40fb177adb
Only set RdbMerge::m_isMerging to false after we relinquish the lock
2017-08-15 11:06:50 +02:00
Ai Lin Chia
f6551ecc8c
Fix typo in variable
2017-06-06 15:19:34 +02:00
Ai Lin Chia
a10c61bfe1
Fix -Wreorder warning
2017-06-06 14:51:03 +02:00
Ai Lin Chia
cb5fcbd730
Don't submit acquire lock job more than once
2017-06-06 14:39:52 +02:00
Ai Lin Chia
e647249872
Move RdbMerge::m_mergeSpaceCoordinator->acquire to a thread (it always sleeps 1 second when acquiring lock is successfull)
2017-06-01 16:28:11 +02:00
Ai Lin Chia
14b3cedb45
Use logDebug/logTrace in RdbMerge. Add more log lines
2017-06-01 15:53:18 +02:00
Ai Lin Chia
acf54bfaaa
Use g_conf.m_logTraceRdbMerge instead of g_conf.m_logTraceRdbDump inside RdbMerge
2017-06-01 15:02:09 +02:00
Ai Lin Chia
cc0967510b
Add logging when loop callback hit time threshold. Remove some unused function, remove undefined function (only defined in header)
2017-05-30 12:12:32 +02:00
Ai Lin Chia
4940258a15
Add log when we're unable to submit regenerate file job
2017-05-29 16:56:58 +02:00
Ai Lin Chia
a6774454d3
Cater for Rdb where index is not used (for resuming killed merge)
2017-05-29 16:44:38 +02:00
Ai Lin Chia
90a4e398ac
Fix bug where we will never stop regenerating map/index when resuming a killed merge
2017-05-29 16:40:41 +02:00
Ai Lin Chia
0ce5dc0dbd
Delay generation of RdbIndex/RdbMap for killed merge until when we need it
2017-05-29 16:27:39 +02:00
Ai Lin Chia
e797620783
Remove unused allowPageCache
2017-05-08 16:01:19 +02:00
Ai Lin Chia
5d83be6c4a
Remove unused maxCacheAge from Msg5
2017-05-08 16:01:19 +02:00
Ai Lin Chia
63796110ae
Remove unused retryNum from Msg5::getList
2017-05-08 16:01:19 +02:00
Ai Lin Chia
1c8be52e78
Remove unused cacheKeyPtr from Msg5::getList
2017-05-08 16:01:19 +02:00
Ivan Skytte Jørgensen
a20cc2b93f
Dropped unused 'syncPoint' parameter to Msg5::getList()
2017-05-08 12:59:06 +02:00
Ai Lin Chia
12bb0e2acf
Introduce Process::isShuttingDown and use it instead of checking Process::EXIT_MODE
2017-04-20 14:02:06 +02:00
Ai Lin Chia
cd22a03de9
Move setInDumpLoop out from RdbDump back into Rdb. Pass rdbId into RdbDump instead of Rdb.
2017-04-03 20:55:24 +02:00
Ai Lin Chia
b475a71b69
Remove power on/off logic in gb
2017-03-31 16:46:26 +02:00
Ai Lin Chia
464e9b889f
Remove treeIndex from RdbDump. We don't delete from tree until we're finish dumping. So we should only update treeIndex there.
2017-03-30 18:06:51 +02:00
Ai Lin Chia
cc4c3ac8b3
Remove collectionless logic
2017-03-29 11:51:20 +02:00
Ai Lin Chia
b25ddb57a1
We shouldn't continue with merge when there is pending generate global index in queue
2017-03-25 23:02:21 +01:00
Ai Lin Chia
98192ba79d
Move generateGlobalIndex to thread
2017-03-17 14:58:26 +01:00
Ivan Skytte Jørgensen
a9229cf0a9
Cannot pass non-POD m_list to varargs function logTrace()
2017-03-15 16:04:43 +01:00
Ivan Skytte Jørgensen
d76e2192a8
Remoed RdbMerge::m_endKey (it was always KEYMAX)
2017-03-14 15:40:26 +01:00
Ai Lin Chia
6d047b0e7b
Even more trace logs
2017-03-10 16:19:10 +01:00
Ai Lin Chia
7090500b2c
Add trace log to RdbMerge, RdbDump, RdbMap
2017-03-10 12:22:11 +01:00
Ai Lin Chia
0f49ddd9ea
newEndKey is not manipulated anymore. Use m_endKey directly
2017-03-10 12:22:11 +01:00
Ai Lin Chia
0d1d4b6247
Make sure all instance of filterTitledbList is commented out
2017-03-07 12:11:06 +01:00
Ivan Skytte Jørgensen
56ab2c751c
First relinquish merge-space lock when files have been file has been moved/renamed
2017-03-06 17:10:44 +01:00
Ai Lin Chia
978988373c
Rename spider-dedup threads to merge-filter. Add filterTitledbList (uncalled) to filter based on urlblocklist
2017-03-06 16:17:49 +01:00
Ai Lin Chia
71c8deb6dc
Move dedup spiderdb to cpu thread
2017-03-06 16:11:04 +01:00
Ivan Skytte Jørgensen
aa08b184da
Removed niceness parameter to RdbDump::dumpList() because it was always the same value as m_niceness previous assigned in RdbDump::set()
2017-03-06 16:02:41 +01:00
Ivan Skytte Jørgensen
213c68dbe4
Made 3-parameter RdbDump::dumpList() private so we hav econtrol of when it is a recall
2017-03-06 15:55:10 +01:00
Ivan Skytte Jørgensen
30b2a5d6fe
Avoid implicit conversion of enum 'rdbid_t' to integral type 'int'
2017-01-26 11:19:37 +01:00
Ivan Skytte Jørgensen
8c406162e5
Make unused function paarmeters unnamed
2017-01-08 17:30:26 +01:00
Brian Rasmusson
947b58ed3c
add support for custom merge lock dir
2017-01-06 09:46:54 +01:00
Ivan Skytte Jørgensen
bec2c03d02
Removed last use of O_NONBLOCK / O_ASYNC on files.
...
O_ASYNC doesn't work on files. O_NONBLOCK is generally useless onless dealing with mandatory file locks.
2016-12-20 14:55:47 +01:00
Ivan Skytte Jørgensen
44c4ee7fb5
Moved g_merge from RdbBase.cpp to RdbMerge.cpp, thereby also cleaning up #includes in RdbBase.h
2016-11-14 22:35:12 +01:00
Ivan Skytte Jørgensen
0d245b5ca3
Removed unused RdbMerge::m_numThreads
2016-11-14 22:31:16 +01:00
Ivan Skytte Jørgensen
dbd9f86fee
Removed unused RdbMerge::unlinkPartWrapper()
2016-11-14 22:30:31 +01:00
Ai Lin Chia
b6570ff536
Allow instance specific mergeDir override
2016-11-14 16:35:25 +01:00
Ivan Skytte Jørgensen
ff97b92fbb
Make merge initialization code clearer
2016-11-14 14:37:32 +01:00
Ivan Skytte Jørgensen
c966c96ddc
Handle more merge errors correctly
2016-11-14 13:47:25 +01:00
Ivan Skytte Jørgensen
c279cccce1
merge: check if we could add a sleep calback. if not then handle error
2016-11-14 13:25:14 +01:00
Ivan Skytte Jørgensen
5825378da5
Use merge-space while merging
...
No more BigFile .part* deletion during a merge to preserve disk space. Instead MergeSpaceCoordinator is used for coordinating access to a large and possibly cheap storage with room for a whole resulting mergefile.
When a mrge file has been finished the reads are allowed from that and reads from the source files disallowed, which are then deleted. Then the file is renamed/moved from merge-space to regular collection storage using the 2-phase commit feature of GbMoveFile.cpp, and finally reads are done from the finished file.
Details:
RdbBase: Use MergeSpaceCoordinator and merge space for temporary target merge file.
RdbBase: better cleanup of crashed merges
RdbBase: more mutex locing while manipulatin m_fileInfo array
RdbBase: keep track of thraeds/jobs
RdbMerge: ditto
RdbMerge: Dont call file->chopHead()
Msg5/Msg3: no more "compensate for merge" flag
Msg3: Skip over RdbBase files that have reads disallowed
2016-10-31 18:16:40 +01:00
Ivan Skytte Jørgensen
6f4c7e63d7
Changed Process::m_mdoe into an enum
2016-10-31 11:51:56 +01:00
