706790e78c
Add http interface to docprocess for ease of testing
2017-12-01 17:59:55 +01:00
139296451d
Move reloading of UrlMatchList to a separate thread
2017-11-02 16:19:44 +01:00
c39c5830ab
Merge branch 'master' into nomerge2
2017-03-31 12:29:49 +02:00
87a79381c0
Cancel all queued jobs when shutting down
2017-03-31 12:21:02 +02:00
2e4d80c80e
Merge branch 'master' into nomerge2
2017-03-14 15:41:32 +01:00
4af955e174
Move document indexing from main thraed to a job
...
XmlDoc::indexDoc/indexDoc2/getMetaList() was being called from a callback in main thraed. Move that work to a thrad. Due to the multiple async calls done by getMetaList() there will be some overhead by this but at least the main thread won't be clogged when encountering 5MB+ documents anymore.
2017-03-14 15:23:15 +01:00
1bc06e97c5
Merge branch 'master' into nomerge2
2017-03-09 17:02:43 +01:00
7592d260c6
Move generateGlobalIndex to thread; Use updateGlobalIndex after merging instead of regenerating global index
2017-03-09 16:59:43 +01:00
ad7168737a
Move RdbDump list-verification to a separate job
2017-03-06 16:34:44 +01:00
d4c150b761
Merge branch 'master' into nomerge2
2017-03-06 16:31:46 +01:00
93e0dbb4b5
Move RdbDump list-verification to a separate job
2017-03-06 16:31:24 +01:00
978988373c
Rename spider-dedup threads to merge-filter. Add filterTitledbList (uncalled) to filter based on urlblocklist
2017-03-06 16:17:49 +01:00
71c8deb6dc
Move dedup spiderdb to cpu thread
2017-03-06 16:11:04 +01:00
a97e964554
Add thread pool for merging
2017-03-06 12:38:36 +01:00
4a9df34f76
Add thread pool for merging
2017-03-06 12:07:36 +01:00
7f044bd0e8
Move dedup spiderdb to cpu thread
2017-03-01 14:22:21 +01:00
d714e0e0ef
Merge branch 'master' into nomerge2
2016-12-22 15:29:59 +01:00
b3164d629f
Move RdbIndex merge to cpu thread
2016-12-22 15:29:09 +01:00
dab892fc05
Merge branch 'master' into nomerge2
...
Conflicts:
JobScheduler.cpp
JobScheduler.h
Msg2.cpp
Msg39.cpp
Msg40.cpp
UdpServer.cpp
main.cpp
2016-11-15 12:13:43 +01:00
e32919794a
Separate job queue for file rename/unlink
2016-11-10 14:47:34 +01:00
323602f314
Dropped thread_type_unlink enum element
2016-11-10 14:38:24 +01:00
50bd004b7a
Merge branch 'master' into nomerge2
...
Conflicts:
Makefile
Msg3.cpp
RdbList.cpp
XmlDoc.cpp
2016-10-07 16:01:34 +02:00
41c705a673
Removed obsolete todo from JobScheduler.h
2016-10-06 12:35:10 +02:00
1a5f180e14
Moved summary generation to separate threads.
2016-09-27 14:27:20 +02:00
9ff99c369f
Merge branch 'master' into nomerge2
...
Conflicts:
RdbList.cpp
RdbList.h
2016-09-05 14:54:49 +02:00
96f2298b55
Maintain statistics for per-job-type times in various phases
2016-09-05 14:50:29 +02:00
5ca6c5dd98
Copied over changes from broken 'nomerge' branch
2016-08-01 14:53:50 +02:00
af4e70a051
Revert "Copied over changes from broken 'nomerge' branch"
...
This reverts commit 9e48989e93
2016-08-01 14:52:50 +02:00
9e48989e93
Copied over changes from broken 'nomerge' branch
2016-08-01 14:52:09 +02:00
766003b6c7
Remove all trace of twin-sync
2016-07-29 13:53:56 +02:00
79ae62881a
Remove all trace of hdtemp
2016-07-29 13:53:56 +02:00
a8a0a6e922
Header guard defines
2016-05-24 16:08:55 +02:00
85795f0b45
Make PageThreads show something in html again
2016-05-03 16:33:12 +02:00
be9840d1c0
Replaced Threads.* with a jobscheduler
...
Threads were being created and destroyed which can be expensive. The
thread-per-job model has been changed to a job scheduler that manages the job
queues and threads in pools. The submission of a job now specifies start/finish
routines, state, and as precisely what kind of job it is. The job scheduler then
takes care of the rest. it is hidden how many queues and pools there are.
2016-04-29 14:27:27 +02:00
