Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-07-23 09:24:39 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • f97971b63b fixed NPE problems doing a shutdown from command-line orbiter 2008-04-09 22:59:17 +00:00
  • 7a35126e91 http timeouts von alten httpc wieder gesetzt danielr 2008-04-09 11:02:14 +00:00
  • 2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet) - fixed broken downloads (flush was missing) - different problem handling when download is corrupted - different default values in yacy.init orbiter 2008-04-08 21:36:33 +00:00
  • d96e2badc7 - fixed POST in proxy - prepared http connection tracking - refactoring (mainly moving StreamTools to serverFileUtils) danielr 2008-04-08 21:17:40 +00:00
  • 14404d31a8 - enhanced performance graph (more info) - added conditions for rarely used logging lines to prevent unnecessary CPU usage for non-printed info orbiter 2008-04-08 14:44:39 +00:00
  • 696b8ee3f5 fix for http://forum.yacy-websuche.de/viewtopic.php?p=6806#p6806 - removed all InputStream.available() because this does not work for files > 2GB - iterator terminate when a IOException occurs - added handling of non-executing index.add methods to enhance assert usage - added index for file indexes > 2GB, to be used in new indexHeap orbiter 2008-04-08 11:55:59 +00:00
  • 94d3d3a86f fixed Proxy (for GET, POST still does not work!) danielr 2008-04-08 09:34:20 +00:00
  • 081ed1d3ec HTTPLoader: reduced stackTraces danielr 2008-04-07 16:56:15 +00:00
  • 8b2efb6f8c fixed garbage in HTCACHE danielr 2008-04-07 16:46:45 +00:00
  • 225f9fd429 various fixes - shutdown behavior (killing of client sessions) - EcoFS reading better - another synchronization in balancer.size() orbiter 2008-04-07 13:12:58 +00:00
  • 6e36c156e8 added more logging to EcoFS orbiter 2008-04-07 09:52:25 +00:00
  • fb541f9162 HTTPC: default timeout half-hour danielr 2008-04-07 09:48:49 +00:00
  • a94f6cdca4 HTTPC: allowed self-signed certs danielr 2008-04-07 09:21:43 +00:00
  • ab330cfdca Network.html: removed ; from location danielr 2008-04-07 08:13:38 +00:00
  • 319144f4b2 fix for outofbounds-excception in EcoFS chunk iterator orbiter 2008-04-06 22:28:17 +00:00
  • 41e9c5723c try to fix shown location (instead of 'Europe/de) JakartaHttpClient/3.') danielr 2008-04-06 22:17:09 +00:00
  • ac8592a102 eclipse build path update orbiter 2008-04-06 20:35:05 +00:00
  • a9cf6cf2f4 generalization of index container-heap class. orbiter 2008-04-06 20:31:16 +00:00
  • f099061944 protection against bad dht-flush word selection orbiter 2008-04-06 20:25:05 +00:00
  • 5e4fddc1e6 more logging for new EcoFS.ChunkIterator to find bug for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1024&hilit=&p=6806#p6806 orbiter 2008-04-06 18:47:49 +00:00
  • 4894df71ab *) moved commons-httpclient from libx to lib (seems to work on my system, I hope one everybody else's too) low012 2008-04-06 18:24:38 +00:00
  • 117ae78001 speed enhancement for reading of eco-table indexes orbiter 2008-04-06 11:50:15 +00:00
  • e96ecd269f *) hopefully fixed build script (included commons-httpclient) low012 2008-04-06 09:49:45 +00:00
  • 7c149a4ee8 - undo less 'binary data found' - removed duplicate stackTrace danielr 2008-04-05 17:46:11 +00:00
  • 96cce8bed9 reduced 'Binary data found' errors danielr 2008-04-05 14:20:01 +00:00
  • 2aef1414f5 removed test (in yacy.init) danielr 2008-04-05 13:49:25 +00:00
  • 5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;) danielr 2008-04-05 13:17:16 +00:00
  • daa04f5db9 added additional check in file handler to prevent that url attacks are hidden in url path encodings orbiter 2008-04-04 12:15:27 +00:00
  • 783a4c9edb strong speed enhancements for the index cache dump and restore: storage and loading is 30 times faster! a cache of 100000 RWIs needed 180 seconds to store and 100 seconds to restore; now the same cache needs only 6 seconds to store and 3 seconds to restore. The cache size has decreased now by 30% (95 MB instead of 150 MB). orbiter 2008-04-02 13:18:23 +00:00
  • 442204a1c8 fix for concurrentModificationException orbiter 2008-04-01 21:21:37 +00:00
  • d2f4926951 - more logging for balancer to get a hint where the problem is - fix for new concurrency method in kelondroSplitTable orbiter 2008-03-31 18:45:27 +00:00
  • 20dadba426 - added a deadlock prevention function in cache flushing - removed unused methods in collection index orbiter 2008-03-31 17:51:51 +00:00
  • 764a40e37d speed enhancements for crawler and url retrieval (affects also search speed) - concurrency for LURL-fetching: this can be done using a concurrent lookup into the separated url databases. Concurrency is possible because there is no IO during lookup. The more LURL-Tables are present, the better is the speedup. More CPUs will increase speed - because a large number of LURL-lookups are made during crawling (for double-check), the LURL-Lookup speed enhancements enhances also crawling speed - search speed also profits from LURL-lookup enhancement - changed some flushing parameters in word index caching which should make better use of large word index caches and should speed up indexing - removed flush chunksize parameter, because this was only useful for IO path enhancement feature which was removed some weeks ago to prevent blocking and deadlocks during search requests orbiter 2008-03-31 15:41:19 +00:00
  • 3ce3a4a3a1 added stub for new index container heap data structure (purpose: index folding) orbiter 2008-03-30 22:58:42 +00:00
  • 2c34038912 addition/correction to last commit: usage of concurrent-classes orbiter 2008-03-30 21:17:12 +00:00
  • b2150057d2 removed unnecessary cleanup method orbiter 2008-03-30 20:32:08 +00:00
  • 76eac114ed * define global javascript-variable with var to get rid of warnings f1ori 2008-03-30 19:51:19 +00:00
  • b63cf2fc1c *) added button to Crawl Profile Editor to delete all terminated crawl jobs (only visible if there are terminated crawl jobs) low012 2008-03-30 15:15:56 +00:00
  • 2aed6bb3f7 * return valid xml in xml-bookmarks f1ori 2008-03-30 14:58:29 +00:00
  • c4c0d54b22 * added regex extended blacklistengine * removed my own engines lulabad 2008-03-30 08:50:09 +00:00
  • 368593e449 enhanced the concurrency handling of indexing process (better queue size control, better data concept, better shutdown behavior) orbiter 2008-03-30 00:03:44 +00:00
  • 4c3f1b67ad *) refactoring of Blacklist_p.java (moving entries might be slightly slower, but the code is more tidy now) *) added edit functionality for blacklist entries low012 2008-03-29 20:39:46 +00:00
  • 466d49e90c * added login-parameter to be able to force authentication f1ori 2008-03-29 11:10:04 +00:00
  • be58135b3e possible fix for deadlock in search execution orbiter 2008-03-29 07:50:37 +00:00
  • c67350f138 * use putXML with forXML-parameter to ensure urls are valid xml (problem was & in url) f1ori 2008-03-28 22:50:33 +00:00
  • 98a03249f3 - optimize selectall-javascript-function (in firefox more then 10 times faster now) f1ori 2008-03-28 16:28:36 +00:00
  • 0241d070bc added concurrency to indexing process: - the methods {parsing, semantic analysis (condensing), structure analysis (web structure)} in the serialized indexing path had been made concurrent. - four BlockingQueues handle concurrency and hand-over of the indexing objects, the last object in the queue is stored into a blockingQueue of maximum size 1 to serialize the process for storage (which uses IO and therefore here should not be deserialized) - a concurrency of (CPUs + 1) is default. Single-CPU users will profil from the change because large files cannot block the indexing process any more. - removed the secondary indexing thread, which is superfluous now. Concurrency is default for all users. orbiter 2008-03-28 11:56:28 +00:00
  • 9fb5d661f2 added my Blacklistengines lulabad 2008-03-28 08:18:21 +00:00
  • bca87f1e38 - refactoring of serverThreads: renaming to distinguish busy-threads and blocking-threads - added blockingThreads which are threads that are not driven by pause times but by BlockingQueue lookup orbiter 2008-03-27 12:03:16 +00:00
  • 968c775025 - preparation of parsing/indexing queue for concurrent execution - remote crawl receipts are now transmitted concurrently in separate threads (makes remove crawls much faster!) orbiter 2008-03-26 22:43:38 +00:00
  • 9b0e20fb06 next refactoring step in document indexing to prepare concurrency environment for document parsing orbiter 2008-03-26 19:51:05 +00:00
  • 7f9f639d20 - refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering - refactoring of word/phrase handling: word abstraction from condenser becomes part of index element handling - removed unused code parts from condenser orbiter 2008-03-26 15:37:49 +00:00
  • d6050b9ffb - separated the LURL data storage and Crawl result stack for process supervision. this is another step to enable multiple, concurrent fulltext-indexes - another try to make the yacy-httpc more stable orbiter 2008-03-26 14:13:05 +00:00
  • 8e889de50b *) Added Lotus' patch (http://forum.yacy-websuche.de/viewtopic.php?t=979), user will be taken back to last opened page after making changes in Advanced Settings. low012 2008-03-25 20:40:54 +00:00
  • 8d6a13bc07 refactoring of parsing-condensing-indexing process: - separated parts - removed storagePeer function next step will be parallelization of processes orbiter 2008-03-24 22:51:26 +00:00
  • d3b06913ec protection against seed-db failure during enumeration orbiter 2008-03-23 23:47:41 +00:00
  • 5aa96dbc36 fix for shutdown configuration orbiter 2008-03-23 13:14:57 +00:00
  • 93633abed8 - removed some debugging code from search process - should speed up now - added some profiling code to search event - more time details in PerformanceSearch_p.html orbiter 2008-03-23 00:55:04 +00:00
  • fba46c51d7 fixed non-termination bug in qsort orbiter 2008-03-22 23:15:28 +00:00
  • 541b817502 refactoring of switchboard queueing orbiter 2008-03-22 01:28:37 +00:00
  • fc94fbe224 another improvement to the collection sorting orbiter 2008-03-20 23:11:04 +00:00
  • 11270d450e better quicksort-pivot computation: 30% faster (measured with test program) orbiter 2008-03-20 22:01:12 +00:00
  • 3e44293f07 - fixed a problem with thread pools in row collection - added a line-viewing feature in threaddump orbiter 2008-03-20 14:21:58 +00:00
  • e43051b125 - fixed Threaddump output (html-escaped ie. <init>) - in EcoFS converted comments to javadoc danielr 2008-03-20 10:20:55 +00:00
  • 433ff855f7 - fixed another concurrency problem in collection sorting - fixed a typing problem that was introduced in svn 4579 and caused the crawl monitor to fail orbiter 2008-03-19 23:47:24 +00:00
  • 19286fa2d1 tried to fix seed2.old.db-problem orbiter 2008-03-19 22:35:19 +00:00
  • 5fde618337 changed display of y-marks apfelmaennchen 2008-03-19 19:54:35 +00:00
  • f3996e63b8 tried to fix more deadlocks: - changed connection modes in ftpc - replaced sort tread pool in row collections by new one using util.concurrent. the old pool had caused blockings orbiter 2008-03-19 11:23:43 +00:00
  • 26155cbb5e update daburna 2008-03-18 17:53:54 +00:00
  • 1618bd5505 added submenu daburna 2008-03-17 14:51:04 +00:00
  • 7008a218b3 avoid ConcurrentModificationException in plasmaCrawlerQueues danielr 2008-03-17 13:51:56 +00:00
  • 9d0af17c5b removed debug from startYACY.sh *ups* danielr 2008-03-17 13:02:39 +00:00
  • bf3e5807b0 fixed typo and remove redundant code from startYACY.sh danielr 2008-03-17 12:51:46 +00:00
  • ea59dbf4c2 fixed build problem see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=956&hilit= lulabad 2008-03-17 06:53:20 +00:00
  • 7150b463ff changed handling of default values and database paths: - the default files yacy.init and for the network definition is now moved to the path defaults - the httpProxy.conf is renamed to yacy.conf - the DATA/INDEX/PUBLIC is renamed to the actual network nickname, which should be freeworld or sciencenet more menu entries - added apfelmaennchens alternative search page to the menu - added the new thread dump page to the server log menu point as submenu modifications - modified the thread dump page: sorting by thread type orbiter 2008-03-16 22:31:54 +00:00
  • 54cb097ea4 added .trigger("update") after paging apfelmaennchen 2008-03-16 22:05:50 +00:00
  • 82f17ccee2 just an example sidebar apfelmaennchen 2008-03-16 21:44:51 +00:00
  • 3c710f22cd added server side driven pagination for search tabs apfelmaennchen 2008-03-16 21:44:20 +00:00
  • 25f5035f23 typo lulabad 2008-03-16 15:13:12 +00:00
  • 7fd094fcbe small bug in ftpc: did cot compile in Java 1.5 Please set compiler to Java 1.5-compliance orbiter 2008-03-16 13:41:49 +00:00
  • 0a593a4e4a small changes to Threaddump Servlet lulabad 2008-03-16 13:15:23 +00:00
  • 995b5a2b27 added Threaddump Servlet, see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=862 lulabad 2008-03-16 09:03:52 +00:00
  • f51bad8ae5 FTP: - report connection status (to break if no connection possible) - fixed isFolder() - additional error output - fixed paths with encoded symbols (ie. a%20file.txt) - refactoring danielr 2008-03-15 21:57:55 +00:00
  • 820641938e ftpc: fixed date parsing, some refactoring danielr 2008-03-15 10:56:47 +00:00
  • 4c584dff87 disabled soLinger to prevent that too many connections stay open (it's a TEST!) orbiter 2008-03-15 10:46:55 +00:00
  • d1b9f50b94 added "delete Release" to Update Page lulabad 2008-03-15 10:37:20 +00:00
  • 3d122c125c added "current installed release" to Update Page lulabad 2008-03-15 09:22:07 +00:00
  • 9c989fe5f7 fixed deadlock orbiter 2008-03-15 00:49:16 +00:00
  • c565906050 FTP: - added maxFileSize-check - added timeout for download - fixed dirlist (when all filenames have spaces, change to absolute links) - enhanced isFolder() - make sure data connection is closed, so a new can be opened - refactoring danielr 2008-03-14 16:28:27 +00:00
  • 368b8735b5 added 'close tab' function apfelmaennchen 2008-03-14 15:59:56 +00:00
  • 1a7870df0d FTP: source cleanup (added finals, indention for easier diffs) danielr 2008-03-14 12:35:53 +00:00
  • fa1090113d - next try to fix the networking problem: set the maximum transfer size to less than MTU=1500-52: buffer size <= 1448 - some refactoring of transfer methods (naming) orbiter 2008-03-14 00:16:04 +00:00
  • f63bd26268 fixed search performance / dynamic display of results apfelmaennchen 2008-03-12 21:24:53 +00:00
  • d87d295c68 one more try to fix the connection problem orbiter 2008-03-12 13:13:11 +00:00
  • a3dadcd89b preventing that peer which return a false search result are disconnected orbiter 2008-03-12 00:56:18 +00:00
  • ba622bb240 addendum to svn 4553 orbiter 2008-03-12 00:24:20 +00:00
  • 5530b8e1ca reverted changes to yacy protocol classes: they caused the sciencenet to loose connections a comparisment with the main release 0.57 had been made: this showed a stable network This is an emergency operation to ensure availability of the sciencenet network. orbiter 2008-03-12 00:05:18 +00:00
  • b664a53553 fix for NPE during search orbiter 2008-03-11 15:30:26 +00:00
  • b3d4571a5d fix for bad opensearch file format did you know that yacy has a built-in rss feed reader? try: http://localhost:8080/FeedReader_p.html?url=http://www.tagesthemen.de/xml/rss2 it can be used to fetch search results from other yacy peers: http://localhost:8080/FeedReader_p.html?url=http://sciencenet.fzk.de:8080/yacysearch.rss?search=yacy orbiter 2008-03-11 12:45:27 +00:00
  • b4ed937f1e - modified zone navigation (does still not work correctly) - added dht switch in network definition - 0.574 orbiter 2008-03-11 11:09:38 +00:00