Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-11-09 00:16:11 -05:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • c45117f81f fixed dates in metadata orbiter 2010-05-08 22:09:36 +00:00
  • 0a5fd15703 :-( orbiter 2010-05-06 22:06:31 +00:00
  • ac16f582aa fix for http://forum.yacy-websuche.de/viewtopic.php?p=20017#p20017 orbiter 2010-05-06 22:04:30 +00:00
  • a7d038bb7a The oai ListFriends source list becomes configurable: just write them into defaults/oaiListFriendsSource.xml orbiter 2010-05-06 10:01:37 +00:00
  • cf13c65bdd added another network definition file for the open access (decentral OAI) search network orbiter 2010-05-05 22:47:03 +00:00
  • 06ff0c5b06 fixes for metadata retrieval and presentation orbiter 2010-05-05 22:45:54 +00:00
  • 5c5e6accdb Fixes for (X)HTML compatibility. suessthomas 2010-05-05 21:12:58 +00:00
  • 5d87379be6 fix for yacy search urls in compare search orbiter 2010-05-05 15:44:40 +00:00
  • 31f30f0925 added compare search for OAIster and oai.yacy.net orbiter 2010-05-05 15:32:36 +00:00
  • 7ab207d93a better presentation of search result metadata and fixes to htcache loading orbiter 2010-05-04 20:57:09 +00:00
  • 5fbf866cae - fixed resumption token generation for oai-pmh import - relaxed dublin core parsing: the dc:reference tag may replace dc:identifier if this does not contain a valid url - parsing of completeRecords number and presentation in the download list of oai import orbiter 2010-05-02 22:20:24 +00:00
  • af13a02307 YaCySearchhtmlFF.html - fixes for XHTML compatibility. suessthomas 2010-05-02 21:03:40 +00:00
  • 5efc0dce0b fix for domain options in search box orbiter 2010-04-30 21:53:20 +00:00
  • fc5efcc05a enhanced and fixed OAI-PMH import - now importing OAI-PMH server list fron two sources - simultanous import from several servers (even > 2000) - check buttons on OAI-PMH server list to select multiple servers for import start - it is possible to select all servers at once for import - imported XML data is gzipped after import from surrogate reader orbiter 2010-04-30 14:03:51 +00:00
  • c2098f9399 close unused connections if there to many for DHT sixcooler 2010-04-29 23:38:50 +00:00
  • 455a763d7c performance hacks orbiter 2010-04-28 08:38:57 +00:00
  • 40a8d132d9 tried to fix 100% CPU when calling Balancer.top() see also: http://forum.yacy-websuche.de/viewtopic.php?p=19978#p19978 orbiter 2010-04-27 22:37:50 +00:00
  • b6cce08019 fixed a bug in rwi storage data size allocation orbiter 2010-04-27 22:22:16 +00:00
  • 90c3e5d6f6 - cleanup, removed unused imports - added crawling queue sizes to /api/status_p.xml, syntax same as in queues_p.html - fixed a bug in queue enumeration that caused a out of bounds exception orbiter 2010-04-27 21:47:41 +00:00
  • 3aad50d38e :-( orbiter 2010-04-26 15:26:08 +00:00
  • 9edd38fbc5 connectionCount limit too low? orbiter 2010-04-26 15:24:47 +00:00
  • 7a05db0fcb fixed to prevent that too many open connections exist - create less connections at maximum (smaller httpc connection pool size) - create less connections per host (2, standard required by RFC) - do not start DHT distributions if there are too many open connections - clear open/idle connections earlier; run cleaner more often orbiter 2010-04-25 23:08:36 +00:00
  • a9b9bf667b fix for http://forum.yacy-websuche.de/viewtopic.php?p=19910#p19910 orbiter 2010-04-25 21:48:30 +00:00
  • b18a7606a0 some performance hacks and fixed after reading dump in http://forum.yacy-websuche.de/viewtopic.php?p=19920#p19920 orbiter 2010-04-25 21:37:36 +00:00
  • 2bc3cba6f1 - fix for 'do not write to cache' rule. - do not read from cache if byte[] array is still filled from response object (will do less IO) orbiter 2010-04-24 08:22:45 +00:00
  • 4cd5418963 removed finalize methods because of a hint in http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/memleaks.html#gbyvh orbiter 2010-04-23 09:32:29 +00:00
  • bfa35d6d20 possible fix for ZURL.list counter orbiter 2010-04-23 08:46:47 +00:00
  • b7efbe55cf i guess these documents are here by mistake orbiter 2010-04-23 07:03:31 +00:00
  • cff8ed134f added index check to prevent blocking in synchronization orbiter 2010-04-22 22:16:38 +00:00
  • 65f383e70b some adjustments to the httpc after testing with a very slow httpd orbiter 2010-04-22 22:10:19 +00:00
  • 5ab5ac80fe fix for NPE in TextParser orbiter 2010-04-21 22:35:47 +00:00
  • b95ae2518b fix for assert orbiter 2010-04-21 17:59:22 +00:00
  • 3247f0e901 fix for deadlocks caused by self-blocking access to TreeMap in concurrent environments. The TreeMap was replaced by a ConcurrentHashMap and additional care that the strings are compared all in lowercase orbiter 2010-04-21 13:46:02 +00:00
  • 027b971bde fix for concurrent quicksort: catch jobs from ThreadPoolExecutor that had been rejected because of full processing queues. Non-catched jobs may have been the cause for blockings and freezes in case of overloading during strong processing orbiter 2010-04-21 13:44:59 +00:00
  • 8c40f1cb8e self-healing for broken table files (may cause other problems, but better than nothing) orbiter 2010-04-21 11:29:27 +00:00
  • 13f5b8e7ba fix for storing/getting bookmark-folders called by Quix0r sixcooler 2010-04-20 17:55:59 +00:00
  • 7b69d79727 enhanced remove() operation: in many cases it is not necessary to return the removed object to the called. for such cases the delete() operation was introduced which is sometimes much cheaper in operation since it does not need to create objects to hold the removed content and it does not need to read those objects. orbiter 2010-04-20 14:47:41 +00:00
  • 93ea0a4789 enhanced remove operation in search consequences (which are triggered when the snippet fetch proves that the word has disappeared from the page that was stored in the index) - no direct deletion of referenced during search (shifted to time after search) - bundling of all deletions for the references of a single word into one remove operation - enhanced remove operation by caring that the collection is stored sorted (experimental) - more String -> byte[] transition for search word lists - clean up of unused code - enhanced memory allocation of RowSet Objects (will use a little bit less memory which was wasted before) orbiter 2010-04-20 13:45:22 +00:00
  • 7a59012632 fix for NPE orbiter 2010-04-20 07:43:48 +00:00
  • 1a6c2f77b4 fix for NPE in statistic servlet orbiter 2010-04-20 00:08:43 +00:00
  • 64f29f990e a collection of performance hacks and code cleanup: - removed usage of URL-Caches which could have been a memory leak - removed unused classes and methods - removed not necessary synchronizations - added synchronization hacks where possible - fine-tuned crawling speed to prevent IO of balancer - fixed a bug in IODispatcher that may have caused that no merges were done - reduced number of parameters in very often called methods (compare methods) - reduced complexity of data structures of now massively used HandleSet class - reduction of new String() and getBytes() usage / new methods to support this transition orbiter 2010-04-19 16:42:37 +00:00
  • 8b8107b2a3 reduced IO-load and synchronization/blocking - enhanced the Balancer performance when building new domain stacks using a new Table buffer - added the new Table buffer BufferedObjectIndex class - changed order of access to LURL-read (prefereing segment over Crawl Queues) will reduced blocking time on balancer - fixed PPM setting in Crawler_p servlet (had doubled values) - reduced synchronization in IndexCell because it is not necessary: reduced blocking during indexing/merging/dumping - removed did-you-mean cache in IndexCell because that caused too much overhead and more memory usage but was not very useful. This reduced also deadlocks that could be causes when searched are performed during indexing. orbiter 2010-04-18 21:55:20 +00:00
  • 7417425e6a Added (b)log notice in header of table in Network.html Added some missing translations for German de.lng in Network.html page mikeworks 2010-04-17 08:16:42 +00:00
  • ed07046870 flush only when > 3000 RWIs present + code cleanup orbiter 2010-04-16 16:07:19 +00:00
  • 3a50b5aa04 enhanced object hash computation orbiter 2010-04-15 14:19:29 +00:00
  • 1a8a134e0c continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775 and continued in SVN 6790 The result should be a less usage of new String() and less memory usage (since a String-encapsulated byte[] has 40 bytes overhead) orbiter 2010-04-15 13:22:59 +00:00
  • dde394a977 - shifted some computation out of synchronization to allow more concurrency - removed synchronization where not necessary orbiter 2010-04-14 23:22:06 +00:00
  • f204076d25 removed usage of temporary files: causes too much IO orbiter 2010-04-14 22:17:18 +00:00
  • 48b9371735 changed balancer re-load counter. causes less blocking here doing intranet indexing. orbiter 2010-04-14 20:57:26 +00:00
  • 650be3599f added a time-out to the RWI cache to flush the cache if it has not been written for ten minutes. This additional dump criteria is necessary because some data sources repeat their vocabulary and may cause that the number of words in a RWI does not increase while the number of references in the RWI set increases. Now the RWI Buffer is flushed every 10 minutes or later if at that time already a dump is ongoing. orbiter 2010-04-14 20:30:34 +00:00
  • ff6cf24b80 replaced RowSetArray in ObjectIndexCache with RowSet to reduce complexity in MergeIterator. This complexity caused too much computing overhead when the RowSetArray had become very large. orbiter 2010-04-14 19:26:51 +00:00
  • 0d04ab1422 new access tracking data type strategy; previous data types may have caused deadlocks of httpd when performing statistic cleanups orbiter 2010-04-14 16:18:04 +00:00
  • fc43f3028e *) hopefully fixing NPE issue introduced in r6797 low012 2010-04-13 23:33:50 +00:00
  • 55d8e686ea performance hacks orbiter 2010-04-13 23:29:55 +00:00
  • 2f181d0027 introduced concurrency in HTCACHE storage compression orbiter 2010-04-13 16:22:09 +00:00
  • 2e26744f4e more concurrency when normalizing RWI entries + cleanup orbiter 2010-04-13 14:47:57 +00:00
  • 555b333041 fix for wrong count of server processes. may fix non-access problems in some cases orbiter 2010-04-13 14:34:16 +00:00
  • 5d8238680f better thread dump statistic evaluation in threaddump servlet when creating multiple dumps for profiling: no distinguishing between thread states orbiter 2010-04-13 12:19:09 +00:00
  • aa083fc45c try to get a fix for OOM problem in case that there is no real problem with missing memory. See also http://forum.yacy-websuche.de/viewtopic.php?p=19835#p19835 orbiter 2010-04-13 11:39:54 +00:00
  • 70e6222978 more concurrency during search requests orbiter 2010-04-13 11:12:36 +00:00
  • 4917f96729 fixes for some changes in SVN 6797 that caused NPEs when the bookmarks initialized orbiter 2010-04-13 10:14:08 +00:00
  • dff660441a *) changes for better code readability low012 2010-04-13 01:31:16 +00:00
  • 15d9ea8375 *) changes for better code readability low012 2010-04-13 01:25:15 +00:00
  • 2bc459252e *) changes for better code readability low012 2010-04-13 01:16:09 +00:00
  • dc93cec3a8 *) Java 1.5 compatibility (see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=2764) low012 2010-04-13 00:25:46 +00:00
  • 67ec58d8e7 search performance enhancement orbiter 2010-04-12 07:31:43 +00:00
  • 4ec0092677 more null == proxy fixes hermens 2010-04-10 18:31:12 +00:00
  • 2f90f0ad56 Remove asserts blocking proxy use cases hermens 2010-04-10 15:12:39 +00:00
  • ef467a0303 Another workaround for the second part of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770 This should prevent URLs with bad referrer entries from being dropped by transferURL or even crashing the whole Transmission$Chunk hermens 2010-04-10 13:57:46 +00:00
  • eb2a4bb555 workaround(?) for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2770&start=0&st=0&sk=t&sd=a&hilit=DefaultCharsetStringPart sixcooler 2010-04-10 00:21:07 +00:00
  • 25aef069a6 continuing String-hash - to - byte[]-hash redesign that was started in SVN 6775 orbiter 2010-04-08 00:11:32 +00:00
  • aad0a3045a Fixed CSS file yacyui-portalsearch.cs border-width: small; --> border-width: thin; margin-botton --> margin-bottom Forum Thread: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2767 mikeworks 2010-04-06 00:04:01 +00:00
  • 6ef4900ec5 Changed function name status() -> statuscheck() in yacyui-portalsearch.js fixing the "status is not a function" bug visible in Firefox error console: http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2625 mikeworks 2010-04-05 12:44:35 +00:00
  • b97ad0f380 *) some minor changes for better code readability *) added more SVN properties low012 2010-04-05 12:37:33 +00:00
  • 7a3c19846f Updated German translation de.lng: added new Table_RobotsTxt_p.html and some other changes Changed 'Sprache' -> 'Language' in yacydoc.html and added translation in de.lng mikeworks 2010-03-29 00:56:51 +00:00
  • ba51d140e1 added more info in assert in balancer orbiter 2010-03-28 22:59:19 +00:00
  • 000dad034b *) better usage of JS timers (intervals) *) added messages when status is checked low012 2010-03-28 22:17:36 +00:00
  • a85c5bb8a7 added support for multiple (fail-over) network definition locations when http-locations are given. multiple locations can be given with a comma-separated list of urls pointing to the network definition file orbiter 2010-03-27 23:15:15 +00:00
  • f83b1b91b9 increased dht busy sleep time to 10 seconds to reduce TCP/IP traffic for default settings. 2 seconds had been too much traffic for home-use routers. Please try to set your dht busy sleep time in existing installations also to 10 seconds. orbiter 2010-03-27 23:04:00 +00:00
  • 9b3840cb66 performance hacks for the template engine + cleanup orbiter 2010-03-27 22:52:48 +00:00
  • 5c10f8bc5f enhanced latest hack orbiter 2010-03-27 07:19:49 +00:00
  • b3238bec83 performance hack for httpd orbiter 2010-03-27 07:09:55 +00:00
  • 1e8e79b9ef redesign of reference hash (URL-hash) parameter hand-over: pass value as byte[], not as String. This should cause that less byte[] <-> String conversions are made during time-critical tasks. This redesign is not yet complete, more to come .. orbiter 2010-03-26 18:33:20 +00:00
  • 72d8e9897b removed unnecessary cache flush call in backend of BufferedRecords orbiter 2010-03-26 12:44:13 +00:00
  • 749ffbd642 - added another catch case for the index dump and index merge process that should cause non-blocking behavior in case that index dump and/or index merge caused any unexpected exception. - reverted SVN 6766, this is too dangerous (may cause unexpected memory usage) and should not be necessary orbiter 2010-03-26 10:46:40 +00:00
  • 9ddb8e4a43 set an option for the java-internal image parser that prevents that the image is cached using the file-system in a temporary file. This should speed up image parsing during image indexing dramatically and should also cause better performance when showing the yacy banner and OSM tiles. orbiter 2010-03-26 10:43:31 +00:00
  • 312ca5d917 removed flush at end of every rwi entry since this reduces the write performance. This should speed up RWI cache dump and RWI merge operations and should cause less blocking time during these processes for the indexer. orbiter 2010-03-26 10:41:20 +00:00
  • 0018163c07 moved table row/column matching method from front-end to back-end orbiter 2010-03-26 10:01:27 +00:00
  • e12f1fd821 - added setting of access rights for executable scripts after auto-installation The correct access right was missing expecially for bin/apicall.sh orbiter 2010-03-25 09:51:01 +00:00
  • 21fcbcc35f added sorting function in network table, reverting SVN 6736 (not removing new sorttable) orbiter 2010-03-25 07:47:41 +00:00
  • 31e29a8831 - removed synchronization during index dump and index cleaning - added semaphores to synchronize index dump and index cleaning for each process separately orbiter 2010-03-25 07:09:53 +00:00
  • 95f31da8da increase dump cache queue length from 1 to 2 orbiter 2010-03-24 20:36:35 +00:00
  • fad3abb524 Tables_p.html servlet can now show tables with selected rows using a search field orbiter 2010-03-24 10:52:37 +00:00
  • 4c6dc396d8 *) more beautyful (IMO) code, no functional changes low012 2010-03-23 21:05:26 +00:00
  • 6c093d6aed - enhanced domain navigator computation - fixed domain navigator content in case that a mustmatch constraint was given orbiter 2010-03-23 13:41:41 +00:00
  • bb63c5d075 using a Pattern object with precompiled regular expressions to apply must-match constraints to search results: should speed up pre-sorting of search results and should cause richer search result sets orbiter 2010-03-23 10:17:28 +00:00
  • 5233177a7f A small typo fixed suessthomas 2010-03-23 08:44:34 +00:00
  • e0da0a84b0 performance fix in http parser orbiter 2010-03-22 09:12:52 +00:00
  • 90dd197ae7 - no latency for local crawls - catch interrupted exception during 'fast' crawls in workflow processor orbiter 2010-03-22 09:12:18 +00:00
  • ea69300857 fix bad floating navigators on little results lotus 2010-03-20 22:09:55 +00:00