Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-11-09 00:16:11 -05:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • b6a8887ff5 better handling of running sessions without explicit hashtable orbiter 2009-11-04 11:59:15 +00:00
  • 1dc7ea986a added a dynamic keep-alive time-out for http server sessions: if there are many concurrent server sessions, the timout is decreased. This should avoid a situation where the clean-up thread is too late to stop running http sessions that should be terminated before the maximum number of server sessions is reached. orbiter 2009-11-04 11:01:09 +00:00
  • e77c906673 *) minor changes mainly in comments *) added svn:keyword settings for several files low012 2009-11-03 22:47:53 +00:00
  • f1740edbf8 *) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration) *) minor change in Digest class, added option in main method, might not be optimal yet low012 2009-11-03 22:28:29 +00:00
  • 11f7da06ed - fixes to csv parser - automatic OAI-PMH import by just clicking on one link from the provided resource list orbiter 2009-11-03 21:18:19 +00:00
  • 9b6762ec2e - added a csv "comma separated values" parser to parse OAI-PMH sources from http://roar.eprints.org/index.php?action=csv - integrated the csv parser into the crawlers parser list - added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser orbiter 2009-11-03 20:10:59 +00:00
  • 0f63de8236 - it is now possible to start several OAI imports concurrently (still not possible to start them with one single request, that will be next) - added a monitor for all running and finished OAI imports (with a little bit of animation..) orbiter 2009-11-03 16:15:22 +00:00
  • 176e334aa4 fixes orbiter 2009-11-02 19:23:05 +00:00
  • 2fa6bf440b workflow update to OAI-PMH importer orbiter 2009-11-02 18:19:30 +00:00
  • b0b7a4f9a5 - added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records - added monitoring for retrieved records orbiter 2009-11-02 11:53:14 +00:00
  • 350d13e153 very first working version of oai-pmh importer: if given the right url, the importer can read and index listRecord xml files and calculate the right resumptionURL which is then given as next default start point for the importer url input. no automatic harvesting by now, this will be done later orbiter 2009-11-02 00:14:14 +00:00
  • 58616d99e4 patch for yacy disk usage detection on lvm host by Michael S. lotus 2009-11-01 08:54:16 +00:00
  • 79251e6f60 configurable disk space hardlimit for dht lotus 2009-10-31 19:12:53 +00:00
  • a0e891c63d - some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports - extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data - some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization - added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior. orbiter 2009-10-31 11:58:06 +00:00
  • 8a1046feaa less maximum file size, too many problems with larger size orbiter 2009-10-30 20:21:45 +00:00
  • 4240785f20 added anti-alias function for line drawing orbiter 2009-10-30 15:58:36 +00:00
  • 30f108f97d added stub of oai-pmh importer (not working yet) orbiter 2009-10-30 15:58:04 +00:00
  • 77c99e500f added more control over memory allocation should avoid some of the OOMs orbiter 2009-10-27 15:25:48 +00:00
  • 52470d0de4 - fix for xls parser - fix for image parser - temporary integration of images as document types in the crawler and indexer for testing of the image parser orbiter 2009-10-22 22:38:04 +00:00
  • 5e8038ac4d - refactoring of blacklists - refactoring of event origin encoding orbiter 2009-10-21 20:14:30 +00:00
  • 26fafd85a5 - more refactoring - fixed problem with parsers orbiter 2009-10-21 15:12:34 +00:00
  • e48f3dfb1e added documentation for new yacy package structure orbiter 2009-10-20 12:05:36 +00:00
  • 3528b970d6 - refactoring - added new experimental (not-yet-working) image parser - added new test image orbiter 2009-10-19 22:34:44 +00:00
  • 6414ac9ecf fix for debian int script http://forum.yacy-websuche.de/viewtopic.php?t=2418 lotus 2009-10-19 17:05:15 +00:00
  • 63e489c5f7 removed win9x scripts because the latest jre has v1.3 for these systems http://www.java.com/en/download/help/win95.xml lotus 2009-10-18 09:45:34 +00:00
  • cde1611919 updated junit orbiter 2009-10-18 02:52:09 +00:00
  • a8ce192f63 - shifted main classes to new package net.yacy - fixed some bugs in last commit orbiter 2009-10-18 01:38:07 +00:00
  • b79f4f062f refactoring of yacy documents and parsers: they depend now only on the kelondro classes orbiter 2009-10-18 00:53:43 +00:00
  • 0fd9540866 Configuration of HTTPDProxyHandler logging hermens 2009-10-17 14:04:18 +00:00
  • 519c3619ff *) minor changes low012 2009-10-17 00:32:07 +00:00
  • f5656b2ae1 *) Made sure that only files with appropriate file endings are listed as skin or language files. *) Introduced protection against directory traversal attacks in configuration servlets for skin and language configuration. Files can only be deleted if they are contained in a list of files which has been read by the servlet first. low012 2009-10-17 00:26:14 +00:00
  • 3434ca381f *) grrr low012 2009-10-16 22:17:21 +00:00
  • ae42c51cf7 *) Skin names and language names are displayed in alphabetical order in dropdown menu now. low012 2009-10-16 22:16:36 +00:00
  • 56a5bd090d Small fixes to header.template for more XHTML compatibility. suessthomas 2009-10-16 20:31:06 +00:00
  • 34c71b22e8 fix and enable parser unit tests (tested with eclipse) f1ori 2009-10-16 09:33:18 +00:00
  • 99683f5f11 small changes to green color and round corners orbiter 2009-10-15 19:39:11 +00:00
  • 76bca8cffd show interactive search without menu orbiter 2009-10-15 13:26:14 +00:00
  • 3d5eeb842a new default skin 'pdblue' The old default skin named 'default' is renamed to 'classic-blue'. All users will keep their current default skin named default, but YaCy will copy the classic-blue also to the skin folder. For all new peers, the new skin pdblue is used. orbiter 2009-10-15 12:59:44 +00:00
  • cee7a05ff2 - de-serialized the pdf parser - added fail callback for file indexer orbiter 2009-10-15 10:47:29 +00:00
  • 9db928ce53 replaced fontbox 0.7.3 with fontbox 0.8.0 orbiter 2009-10-15 09:51:16 +00:00
  • c2272785c7 - fix for xlsx and pptx parsing - less exception logging for swf parser orbiter 2009-10-14 19:15:38 +00:00
  • afae2a0bee Small changes to the Yacy Skins. suessthomas 2009-10-14 19:11:52 +00:00
  • 0975b1b493 update for apache poi library possible solves http://forum.yacy-websuche.de/viewtopic.php?p=17736#p17736 lotus 2009-10-14 15:24:53 +00:00
  • c864901087 - moved httpd.mime to defaults path - some documentation fixes - adopted a default setting for the search window: moves css setting to base.css - some enhancements for the DocumentIndex class orbiter 2009-10-14 13:29:09 +00:00
  • 8829ec5f18 *) made sure that   is replaced with a space and not just deleted in CharacterCoding.java *) added annotations and made minor changes to serverObjects.java *) set subversion properties for several files low012 2009-10-13 20:57:56 +00:00
  • 6c347a37eb more options for DocumentIndex orbiter 2009-10-13 08:43:02 +00:00
  • 6192205533 more final modifier orbiter 2009-10-12 21:59:39 +00:00
  • 0f6b011e1a fix for new index location and better way to use own classes by reflection orbiter 2009-10-12 21:12:42 +00:00
  • 7a3bbd950f :-( orbiter 2009-10-12 20:29:03 +00:00
  • b953f04f90 one more reflection fix orbiter 2009-10-12 17:45:42 +00:00
  • 77d6604856 fix for npe, see http://forum.yacy-websuche.de/viewtopic.php?p=17727#p17727 orbiter 2009-10-12 17:41:16 +00:00
  • 2a7fe35f92 performance tuning using more final modifiers in the kelondro core orbiter 2009-10-12 17:37:12 +00:00
  • cb4de9ceee fixed a bug in table iterator (did not recognize elements in write buffer) orbiter 2009-10-12 08:06:35 +00:00
  • 5841ee83d3 refactoring orbiter 2009-10-11 21:29:18 +00:00
  • e7f18ba24b refactoring orbiter 2009-10-11 00:24:42 +00:00
  • ce8dc575ca refactoring orbiter 2009-10-11 00:12:19 +00:00
  • bea3b99aff moved table and util classes orbiter 2009-10-10 01:14:19 +00:00
  • bd876eb4b7 moved io classes orbiter 2009-10-10 01:00:49 +00:00
  • c0e0e1f422 moved blob classes orbiter 2009-10-10 00:43:25 +00:00
  • 1e4f8b56ed accumulated classes from different packages into the new rwi package orbiter 2009-10-10 00:39:15 +00:00
  • 194da25a2f moved kelondro index orbiter 2009-10-09 23:32:08 +00:00
  • 4446acc8cd moved kelondro order orbiter 2009-10-09 23:22:22 +00:00
  • f677d534b1 start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root - moved here the logging classes as part of the new net.yacy.kelondro package orbiter 2009-10-09 23:13:30 +00:00
  • ea473e32b8 refactoring orbiter 2009-10-09 22:27:50 +00:00
  • 735e2737e3 * added index segments This is a major change in the organization of indexes. Please consider a back-up of your data before you run this update. All existing index files will be moved and renamed to a new position. With this change, it will be possible to maintain different indexes for different purposes and it will be possible to have a distinction between DHT-in and DHT-out specific indexes. Tenants may also have their own index, and it may be possible to have histories and back-ups of indexes. This is just the beginning, many servlets must be adopted after this change, but all functions that had been there should still work. orbiter 2009-10-09 14:44:20 +00:00
  • 09de5da74a once again a performance hack orbiter 2009-10-08 18:26:54 +00:00
  • 2f6d88403e uä orbiter 2009-10-08 18:10:56 +00:00
  • d2615ea5a8 increased memory for scraper buffer to enhance parsing speed orbiter 2009-10-08 15:27:13 +00:00
  • 4bbbb74ec4 removed not necessary synchronization orbiter 2009-10-08 15:26:28 +00:00
  • 67e5464cc2 Fix for SVN6380: x[] Arrays are unsuitable Keys for Maps without using a proper Comparator. hermens 2009-10-08 12:55:36 +00:00
  • 5f72d2b19f update to jre6u16 lotus 2009-10-08 10:48:33 +00:00
  • aeab8c7917 Prevent failed DHT attemps from overwriting newer peer info hermens 2009-10-08 00:17:29 +00:00
  • 9324b5b6c5 Enhancements to DHT - speed up deletion of containers when selscted from whole index - correctly eliminate all references to unavailable URLs, not just the first encountered hermens 2009-10-08 00:03:16 +00:00
  • e49e2d75fe Limit the time Transmission.Chunks stay in the transmissionCloud by using a Map that iterates entires in insertion order. hermens 2009-10-07 23:41:25 +00:00
  • 92db7c5d07 increased timeout for index retrieval orbiter 2009-10-07 13:03:13 +00:00
  • 386b9f35f6 activated resource observer for windows 7 lotus 2009-10-07 06:20:24 +00:00
  • 6e0dc39a7d - some fixes to prevent blocking situations - better logging for the crawler - better default values for the crawler orbiter 2009-10-06 21:52:55 +00:00
  • 51f2bbf04b possible fix for problem in http://forum.yacy-websuche.de/viewtopic.php?p=17655#p17655 orbiter 2009-10-06 09:56:14 +00:00
  • f8371707e5 - possibly better termination for SplitTable - better abstraction in DidYouMean orbiter 2009-10-05 22:09:58 +00:00
  • 87780f2562 produce did-you-mean also for queries with more than one word orbiter 2009-10-05 21:51:02 +00:00
  • 04a548a1e3 - temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class - fixes for numerous other problems - removed dead code - resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs - another fix for the deadlock problem in SplitTable orbiter 2009-10-05 20:11:41 +00:00
  • ea427df944 fixed a worst case situation of the condenser which may cause a temporary full CPU load because of a bad data structure usage orbiter 2009-10-05 08:26:55 +00:00
  • f1bde59c50 logger config cleanup lotus 2009-10-02 18:03:14 +00:00
  • 3e38035389 fix for interrupted thread during has() property check orbiter 2009-10-02 10:55:40 +00:00
  • 5bd1c1d205 just added some comments that had been produced to learn about OAI-PMH orbiter 2009-10-01 22:56:22 +00:00
  • d52cf19835 small changes to de.lng (parser settings) suessthomas 2009-10-01 20:27:05 +00:00
  • 6aa474f529 - better logging for web cache access and fail reasons - better Exception handling for web cache access - distinction between access of web cache for proxy and crawler orbiter 2009-10-01 13:08:19 +00:00
  • 3671c37989 added experimental oai-pmh reader and integrated it with the existing dublin core parser orbiter 2009-09-30 22:11:00 +00:00
  • 0c17b600c6 remote search by default off orbiter 2009-09-30 15:06:29 +00:00
  • 58a00205d5 re-activated the emergency close when too many server connections exist orbiter 2009-09-30 14:29:43 +00:00
  • c57d2070e6 more logging orbiter 2009-09-30 13:25:08 +00:00
  • a995b95367 tried a fix for the httpd access bug (too many unclosed sessions) orbiter 2009-09-30 13:18:02 +00:00
  • e1fba41cad better logging orbiter 2009-09-29 21:52:17 +00:00
  • 2275f885a8 possible fix for concurrency problem orbiter 2009-09-29 21:40:50 +00:00
  • a6a3090c3d *) blacklist cleaner supports usage of regular expressions now *) refacored BlacklistCleaner_p.java for better readability *) moved check of validity of patterns to the Balcklist implementation since patterns might be valid in one implementation, but not in another *) added method to check validity to Blacklist interface *) fixed some minor issues like typos or wrong whitespaces *) set subversion properties for a whole bunch of files low012 2009-09-29 21:28:49 +00:00
  • 5a93807781 improved web cache speed: - removed one computation out of a synchronization - removed one not necessary has() call orbiter 2009-09-29 08:41:05 +00:00
  • 2e8b2867ff double performance of store method because it avoids one 'has' orbiter 2009-09-29 08:23:44 +00:00
  • afda5b1adc new join method for indexes (not yet used) orbiter 2009-09-29 08:16:24 +00:00
  • 65b66c2c18 better handling of array files of length 0 orbiter 2009-09-29 08:13:44 +00:00
  • 1957b5797a fix for seed generation orbiter 2009-09-29 08:05:36 +00:00