8b20afcd8d
Fix bug where we may wait indefinately while destroying RdbIndex after merge. Issue with not locking m_pendingMerge
2017-06-02 11:35:27 +02:00
81fb24d82a
Change RdbIndex::m_pendingMerge to atomic to avoid locking RdbIndex::m_pendingDocIdsMtx when checking if it's set
2017-05-30 12:33:09 +02:00
0ce5dc0dbd
Delay generation of RdbIndex/RdbMap for killed merge until when we need it
2017-05-29 16:27:39 +02:00
88ecd3153c
Protect m_needWrite variable with a mutex to avoid RdbIndex not writing when it needs to (in preparation for moving Msg4In to a thread)
2017-04-21 11:44:37 +02:00
ba8a260c91
Added RdbIndex::clear method
2017-03-30 16:47:34 +02:00
3613402acd
RdbIndex::addKey/addRecord_unlocked: const
2017-03-27 12:44:04 +02:00
cb8e24b0b1
Check pending docIds as well when verifying index
2017-03-22 15:20:18 +01:00
21efc82b90
Don't preallocate m_pendingDocIds when we're not going to use it
2017-03-15 15:58:52 +01:00
8265aa398a
Don't free memory when it's RdbIndex for tree/bucket
2017-03-13 13:39:10 +01:00
3b6f9099fc
Revert "Make sure we're not blocking main thread while merging pending doc ids"
...
This reverts commit b5f4e003d7
2017-03-08 16:45:28 +01:00
b5f4e003d7
Make sure we're not blocking main thread while merging pending doc ids
2017-03-08 12:09:30 +01:00
eeaf3a69d1
Make sure we don't try to merge a deleted index
2017-03-06 16:36:25 +01:00
b3164d629f
Move RdbIndex merge to cpu thread
2016-12-22 15:29:09 +01:00
886e4071c4
Make sure RdbIndex std::vector memory is freed
2016-12-19 14:57:59 +01:00
7340a15eb4
Added timedMerged to sleep callback to avoid calling gettimeofdayInMilliseconds at every RdbIndex::addRecord
2016-12-15 17:09:19 +01:00
bb26228442
More #include cleanup of SafeBuf.h
2016-11-04 12:47:34 +01:00
5825378da5
Use merge-space while merging
...
No more BigFile .part* deletion during a merge to preserve disk space. Instead MergeSpaceCoordinator is used for coordinating access to a large and possibly cheap storage with room for a whole resulting mergefile.
When a mrge file has been finished the reads are allowed from that and reads from the source files disallowed, which are then deleted. Then the file is renamed/moved from merge-space to regular collection storage using the 2-phase commit feature of GbMoveFile.cpp, and finally reads are done from the finished file.
Details:
RdbBase: Use MergeSpaceCoordinator and merge space for temporary target merge file.
RdbBase: better cleanup of crashed merges
RdbBase: more mutex locing while manipulatin m_fileInfo array
RdbBase: keep track of thraeds/jobs
RdbMerge: ditto
RdbMerge: Dont call file->chopHead()
Msg5/Msg3: no more "compensate for merge" flag
Msg3: Skip over RdbBase files that have reads disallowed
2016-10-31 18:16:40 +01:00
58b9015fcd
Removed 'force' functionalit yon file rename operations
...
In normal circumstances the destination file should not exist. spidering/merging/querying works fine. recovery logic may have to get fixed too.
2016-10-25 16:12:17 +02:00
5eab079963
Removed 'force' parameter from BigFile::rename (variant 2)
2016-10-25 15:06:06 +02:00
1efe9f7e9e
Removed default values from RdbInde::rename()
2016-10-25 14:34:58 +02:00
4123c28f41
Removed 'force' parameter from RdbIndex::rename() (always true)
2016-10-25 13:05:09 +02:00
e4afd2320f
Removed non-const verions of getFilename() in BigFile/RdbIndex/RdbMap
2016-10-24 16:03:38 +02:00
8ce3b603da
More constness in Rdb*
2016-10-06 15:21:11 +02:00
92cb25d30f
Use version number instead of storing dataFileSize. It's practically impossible to get accurate bucket record count by just counting the number of addRecords
2016-09-29 16:06:05 +02:00
75d46a736f
Add unlocked version of RdbIndex::mergePendingDocIds
2016-09-29 16:06:05 +02:00
b6574f7730
Add support for delete key to RdbIndex
2016-09-26 16:19:12 +02:00
44d849bc35
Swap parameter to be more consistent with other Rdb* class. Passing in collnum as the first parameter
2016-09-23 15:19:49 +02:00
8454da1376
Use rdbid_t for rdbId instead of char
2016-09-20 13:41:23 +02:00
460a5b487b
Add some simple validation on RdbIndex generated file
2016-08-31 15:36:50 +02:00
a23abef4a1
Generate index file while merging and dumping tree
2016-08-31 11:13:31 +02:00
ff6269f93b
Add logic to support time based & size merging of pendingDocIds
2016-08-31 11:13:31 +02:00
b211a82786
Use RdbQueryIndex and removed functions replaced by RdbQueryIndex
2016-08-31 11:13:31 +02:00
90664eff53
Use std::shared_ptr<std::vector<uint64_t>> instead of plain std::vector<uint64_t> for m_docIds and global index
2016-08-31 11:13:31 +02:00
54cf5c7300
Encapsulate RdbBuckets
2016-08-31 11:13:31 +02:00
127df3593f
Add code for validation of rdbindex
2016-08-31 11:13:31 +02:00
1bfd964468
First implementation of global presence index
2016-08-31 11:13:31 +02:00
b1d1a6de97
Fix compilation error
2016-08-31 11:13:31 +02:00
4639be504c
Move add key logic into RdbIndex::addRecord
2016-08-31 11:13:31 +02:00
5655484366
Tie Rdb tree/bucket index to a specific collection. First implementation of moving m_index from Rdb to RdbBase
2016-08-31 11:13:31 +02:00
c8332b3f41
Logic to regenerate index file (not fully implemented)
2016-08-31 11:13:31 +02:00
163d0955f3
base code for posdb index files
2016-08-05 15:42:20 +02:00
