Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-11-06 23:56:13 -05:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 550312ac85 added new command script to do a auto-Update from command line. this will make it easy to do mass-auto-updates in private yacy clusters orbiter 2009-06-11 11:31:26 +00:00
  • 0fc1168554 - reduced time-out for socket-connection communication from 20 seconds to 5 seconds. This is a test to find out if the time-out was a cause for problems in metager environments - turned a fine log entry in case of rejected connections on the server socket into a warning. (look for 'exceeding limit') orbiter 2009-06-11 10:20:31 +00:00
  • 28b86385cd patch for bad behaving swf parser orbiter 2009-06-11 09:54:48 +00:00
  • d58b395993 fix for http://forum.yacy-websuche.de/viewtopic.php?p=15693#p15693 orbiter 2009-06-11 09:38:25 +00:00
  • cffef67dc5 added a short info line about the latency monitor orbiter 2009-06-10 23:03:29 +00:00
  • 733385cdd7 enahnced database access times by removal of unnecessary synchronization. added also more hacks that resulted from high-volum query testing orbiter 2009-06-10 23:02:42 +00:00
  • 5a7dec880e - some improvements for: http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1904#p15668 - portalsearch: introduced yconf.load_js and yconf.load_css - yacysearch.html still having problems with focus after sidebar is loaded - yacysearchtrailer.json seems not to be valid json for ?nav=all apfelmaennchen 2009-06-10 22:11:31 +00:00
  • 5d7045387b added more word lists and a multi-access search test tool for high-performance query testing: run searchtestmulti.sh; then 10 concurrent processes fire 1000 requests each to the local peer. orbiter 2009-06-10 22:01:48 +00:00
  • 398e210fef removed synchronization in logging that causes deadlocks in high-performance environments orbiter 2009-06-10 19:17:30 +00:00
  • db3a06dd81 removed cookie handling in httpc: - no need to do cookie handling in proxy, this was switched off so far - no need for cookies in crawler, this was switched on (by mistake) This fix was needed for a case where a web server flooded the crawler with cookies and caused a complete blocking of the httpc. orbiter 2009-06-10 16:11:09 +00:00
  • 1c54ae4a63 some small changes in HandleMap Testing orbiter 2009-06-10 15:02:52 +00:00
  • b21e9149f5 another fix for navigation results, the json result format and searches with yacyinteractive orbiter 2009-06-10 12:41:15 +00:00
  • 15c5406b9c fixed yacyinteractive orbiter 2009-06-10 07:24:45 +00:00
  • 2c5554c912 small enhancements in search result computation speed orbiter 2009-06-09 15:22:23 +00:00
  • e0b3984805 added navigation keys for site and author facets to remote search interface orbiter 2009-06-09 09:07:52 +00:00
  • 27fa6a66ad - completed the author navigation - removed some unused variables orbiter 2009-06-08 23:30:12 +00:00
  • a9a8b8d161 - added display of author navigation (usage of that navigator not yet implemented - added a synchronization in pdf parser which should help to avoid deadlocks that occur when displaying several search results pointing to pdf sources - fixed smaller bugs in navigation orbiter 2009-06-08 22:01:26 +00:00
  • c879783008 added steering of navigator computation: - by default the navigator computation if off for servlet yacysearch.html, but: - the servlet is called by default with a option to switch navigator results on this will prevent that metasearch users will get slow results that are caused by unnecessary computations orbiter 2009-06-07 22:51:15 +00:00
  • c079b18ee7 - refactoring of IntegerHandleIndex and LongHandleIndex: both classes had been merged into the new HandleMap class, which handles (key<byte[]>,n-byte-long) pairs with arbitraty key and value length. This will be useful to get a memory-enhanced/minimized database table indexing. - added a analysis method that counts bytes that could be saved in case the new HandleMap can be applied in the most efficient way. Look for the log messages beginning with "HeapReader saturation": in most cases we could save about 30% RAM! - removed the old FlexTable database structure. It was not used any more. - removed memory statistics in PerformanceMemory about flex tables and node caches (node caches were used by Tree Tables, which are also not used any more) - add a stub for a steering of navigation functions. That should help to switch off naviagtion computation in cases where it is not demanded by a client orbiter 2009-06-07 21:48:01 +00:00
  • bead0006da replaced tmp file extensions by prt orbiter 2009-06-06 18:09:58 +00:00
  • 3189f9cd39 fixed problem with DCEntry initialization orbiter 2009-06-06 18:00:50 +00:00
  • a704d82280 patch for problem with digest orbiter 2009-06-06 16:53:16 +00:00
  • 3029ef6eb3 fixed a bug that was recently inserted which caused that no idx and gap files were written. orbiter 2009-06-06 16:43:58 +00:00
  • b6e274f211 omit most of forced crawl delays by using a separat delay table which flushes delayed URLs at the correct time orbiter 2009-06-06 16:20:27 +00:00
  • d50be59088 - added a automatic re-construction of the domain stack after 10 minutes. this includes then urls to the domain stack that were left over in case of stack size limitations when the domain stack was created the last time - changed the busy sleep time for the crawl thread to 30 millisecons. This is sufficient to crawl with 2000 PPM. orbiter 2009-06-06 09:34:44 +00:00
  • 5fdba0fa51 - fixed a not working selection rule in balancer - more security about crawl-delay, be more fail-save - better logging in case of long forced crawl-delays orbiter 2009-06-06 08:46:59 +00:00
  • f5602404d5 another speed boost for the balancer orbiter 2009-06-06 02:37:04 +00:00
  • 95e8cbd1c3 new fully redesigned balancer and bugfixes regarding lost profile handles and killed crawls orbiter 2009-06-06 01:56:31 +00:00
  • c062385552 fix for http://forum.yacy-websuche.de/viewtopic.php?p=15555#p15555 orbiter 2009-06-05 18:18:16 +00:00
  • 42ae40b9f6 some bugfixes to database close() methods orbiter 2009-06-04 22:43:46 +00:00
  • a0c53abbe1 - wait until local results are computed during search, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2167&hilit=&p=15521#p15521 - show only x+1 pages in page navigator orbiter 2009-06-04 20:58:47 +00:00
  • 94f3d90af2 added a hint about regular expressions in crawl start orbiter 2009-06-04 20:03:26 +00:00
  • 9bfd22f65d fix for http://forum.yacy-websuche.de/viewtopic.php?p=15523#p15523 orbiter 2009-06-04 19:57:25 +00:00
  • 1c77db670f re-designed response format for navigation: - changed json and rss response templates orbiter 2009-06-04 10:54:49 +00:00
  • 15fad767c0 some refactoring of topic generation orbiter 2009-06-03 23:49:06 +00:00
  • f28f62fb21 added servlet for easy wiki content and search window integration orbiter 2009-06-03 22:22:20 +00:00
  • efe97f446a better proxy configuration in case of remote proxy lotus 2009-06-03 19:03:03 +00:00
  • cc49aedf12 - fixed problem with remote search NPE - more abstraction for search requests orbiter 2009-06-03 08:49:54 +00:00
  • 9e18abc2ac * fix charset detection, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2137 * why has this been uncommented??? f1ori 2009-06-02 20:54:13 +00:00
  • c38c852090 modified access method to get index entries out of a array of BLOBs: iterate them, then merge; not collect them and merge then. This should use less memory and may behave better in an environment with many queries. To ensure that too many queries will not cause total blocking, a time-out of one second was also added. After the time-out the index data that was collected so far is returned. orbiter 2009-06-02 16:53:45 +00:00
  • 55ff919b5d - yacysearchtrailer.html ... just an idea for a timeline apfelmaennchen 2009-06-02 16:47:39 +00:00
  • ab06a6edd2 renamed topwords to topics and enhanced computation methods of topics topics will now only be computed using the document title, not the document url, because the host navigator is now responsible for statistical effects of urls. orbiter 2009-06-02 15:20:10 +00:00
  • 61d9e131b4 better/new proxy auto config lotus 2009-06-02 12:18:29 +00:00
  • 0d44a6d503 - yacy portalsearch experiments with navigation in sidebar (topwords & domains) - not yet functional ... apfelmaennchen 2009-06-02 11:02:36 +00:00
  • 9f9a1b4ad8 - yacysearchtrailer.html small temporary work around for jquery-css display bug apfelmaennchen 2009-06-02 09:08:34 +00:00
  • b0e2d854e0 - fixed sidebar for yacysearch.html & yacysearchtrailer.html - @orbiter: please do not use <h2> or <h1> tags in the context of the sidebar!!! apfelmaennchen 2009-06-02 07:32:45 +00:00
  • a5d481eab1 enhanced navigation - fixed too early computation of navigation - moved navigation rendering to yacysearchtrailer - added more asserts orbiter 2009-06-01 22:45:28 +00:00
  • 3ca1f109c4 added more jquery themes orbiter 2009-06-01 21:49:18 +00:00
  • 3ea399ec91 fix for absolute paths for repository path orbiter 2009-06-01 10:54:41 +00:00
  • 6b92155eb6 corrected spelling lotus 2009-06-01 09:48:08 +00:00
  • 5eac607166 fixed configuration of repository path orbiter 2009-06-01 00:13:23 +00:00
  • daee735ad7 - fix for yacysearch.html - navigation/sidebar JavaScript is now also triggerd by #(navigation)# instead of display=3 apfelmaennchen 2009-05-31 07:29:24 +00:00
  • 8fe69da2bb - some fixes for prev. post - better resizing and dragging apfelmaennchen 2009-05-30 17:06:47 +00:00
  • 0eb3bffe97 - added 'drawer' (sidebar) for future navigational items to yacyui-portalsearch.js - http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1904#p15311 apfelmaennchen 2009-05-30 14:32:29 +00:00
  • 7639ec2f38 - fixed letter case bug for dc record creation - dc parser is now lazy against letter cases orbiter 2009-05-29 15:09:37 +00:00
  • 34af8b4877 - yacysearch.html compromise for positioning the sidebar - position is now fixed on the right top side - should scale down to window width 800px, smaller windows will cause distortions - see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1904#p15293 apfelmaennchen 2009-05-29 14:57:09 +00:00
  • 4522c13ee7 added option for a table prefix when importing phpbb3 orbiter 2009-05-29 14:29:02 +00:00
  • 1c69d9b8b6 more refactoring of the index classes orbiter 2009-05-29 14:16:41 +00:00
  • 3d5f2ff544 - added new servlets to support search portal administrators for the integration of yacy search fields in their web pages - moved some servlets from here to there.. - changed menu structure - removed yacyui-portaltest.html which contained an example for the live search which is now integrated on all pages in yacy. The code snippet example from that page is integrated into the ConfigLiveSearch.html servlet orbiter 2009-05-29 14:16:03 +00:00
  • 4d4315687f fix for problem with concurrency in host navigator, bug reported by wsb orbiter 2009-05-29 10:52:50 +00:00
  • 88426912ad more refactoring to make the segment object easier to use and to be prepared to integrate author navigation orbiter 2009-05-29 10:03:35 +00:00
  • d813fd26ed reset sent/received counters on index delete lotus 2009-05-28 15:49:42 +00:00
  • 99bf0b8e41 refactoring of plasmaWordIndex: divided that class into three parts: - the peers object is now hosted by the plasmaSwitchboard - the crawler elements are now in a new class, crawler.CrawlerSwitchboard - the index elements are core of the new segment data structure, which is a bundle of different indexes for the full text and (in the future) navigation indexes and the metadata store. The new class is now in kelondro.text.Segment orbiter 2009-05-28 14:26:05 +00:00
  • 876746602d catch problems of file hash computation, see also: http://forum.yacy-websuche.de/viewtopic.php?p=15245#p15245 orbiter 2009-05-28 10:08:36 +00:00
  • fec6f9054f some refactoring of search methods orbiter 2009-05-27 23:51:34 +00:00
  • d793bb0d76 the mysql lib was not in releases included; moved library from libx to lib orbiter 2009-05-27 20:23:28 +00:00
  • 3d4b826ca5 migration of all databases that use the deprecated BLOBTree format into the BLOBHeap format. Old databases are migrated automatically. This removes the last very IO-intensive data structures which were still used for Wiki, Blog and Bookmarks. Old database files will still remain in the DATA subdirectory but can be deleted manually if no major bugs appear during migration. There is no need for any user action, all migration is done automatically. orbiter 2009-05-27 15:04:04 +00:00
  • 4b4bddca00 added new submenu to crawler menu: import of phpbb3 forum postings from mysql - yacy can import phpbb3 posts without crawling - all data is written as surrogate - indexed surrogate files can be re-used orbiter 2009-05-27 14:53:23 +00:00
  • f1a9253baa see: http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1904#p15198 apfelmaennchen 2009-05-27 08:56:55 +00:00
  • 369bb78a7a - added "visibility: hidden;" for yacysearch.html sidebar - this only works with javascript....sorry apfelmaennchen 2009-05-27 07:30:59 +00:00
  • 16a226b016 removed debug-alert() from last commit apfelmaennchen 2009-05-27 07:13:45 +00:00
  • 01334686bd small fixes for yacysearch.html sidebar (display=3) apfelmaennchen 2009-05-27 07:06:13 +00:00
  • d8284046b0 enhanced speed of site navigation computation orbiter 2009-05-26 22:30:20 +00:00
  • c72a5cf326 added stub for PHPBB3 extraction code using direct access to mySQL orbiter 2009-05-26 15:58:49 +00:00
  • e735d3a69f fix for http://forum.yacy-websuche.de/viewtopic.php?p=15175#p15175 orbiter 2009-05-26 15:03:50 +00:00
  • 63a0255166 - refactoring: added new content package, which will contain connector classes for different types of data sources to import texts into the YaCy index - refactoring: migrated data objects for the new connector classes - added a DAO interface class to specify an abstract interface for database retrieval connector methods orbiter 2009-05-26 07:44:22 +00:00
  • f246928c20 first attempt to add 'real' Navigation to yacy search results: host navigation - after a search is started, it is analysed how many hits are in each site - this can be done really efficient, because the navigation information is hidden in the url hash and can be computed very fast - the search result shows a column on the right with the hosts and the hits per host - after a click on a host the search is modified using the efficient site: - operator orbiter 2009-05-25 22:27:34 +00:00
  • 54b9e99c01 - more information about peer tags - peer tag is by default '*' orbiter 2009-05-25 21:43:33 +00:00
  • 26a46b5521 increased default maximum file size for database files to 2GB Other file sizes can now be configured with the attributes filesize.max.win and filesize.max.other the default maximum file size for non-windows OS is now 32GB orbiter 2009-05-25 06:59:21 +00:00
  • eb36c9a092 hint for Java in start script this can usually only be seen if it isn't installed lotus 2009-05-24 17:06:37 +00:00
  • addecdb18c simplified code, removed one unused method in all implementing classes orbiter 2009-05-21 23:53:01 +00:00
  • 47fce9020c small change (Orbiter's wish) borg-0300 2009-05-21 17:51:52 +00:00
  • e07b14e5d7 finally a working fix for 5960 borg-0300 2009-05-21 16:07:04 +00:00
  • 3ebb904d2c fix for 5960, http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2119 borg-0300 2009-05-21 11:47:57 +00:00
  • 734680dc70 initialize the ResourceObsever in own thread lotus 2009-05-21 08:30:34 +00:00
  • e005cfea37 fix for bug in -incell option of URLAnalysis orbiter 2009-05-21 06:57:03 +00:00
  • a7e392f31b The collection index will not be supported any more. Existing indexes based on the old index collections must be migrated with YaCy 0.8 - removed index collection classes and all migration tools - added a 'incell' reference collection feature in URL analysis orbiter 2009-05-20 14:51:26 +00:00
  • a2f48863fc - added prototype for navigation index - refactoring of word index prototype (no functional changes so far) orbiter 2009-05-20 09:00:24 +00:00
  • 47fd226bdb proper parsing of sentences does not affect tokens/words lotus 2009-05-19 16:41:27 +00:00
  • 27eb8d62cb - new development cycle - removed temporary configuration with safe setting for indexer threads (=1) and replaced it with best value computed during performance tests (1/2 of number of processors) orbiter 2009-05-18 21:20:06 +00:00
  • fdb87a0616 release 0.8 orbiter 2009-05-17 22:23:36 +00:00
  • 41dd31cad2 replaced new navigation icons with same images but smaller resolution (16x16 instead of 128x128) orbiter 2009-05-17 22:22:29 +00:00
  • b7457d3807 patch for http://forum.yacy-websuche.de/viewtopic.php?p=14720#p14720 orbiter 2009-05-17 21:44:02 +00:00
  • bffbe43e09 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14522#p14522 fix for http://forum.yacy-websuche.de/viewtopic.php?p=14955#p14955 orbiter 2009-05-17 21:15:06 +00:00
  • f133d6065c fix for http://forum.yacy-websuche.de/viewtopic.php?p=14955#p14955 orbiter 2009-05-17 18:28:33 +00:00
  • 16d28496ac *) YaCy will load status page once it is up again after restart or update if Javascript is enabled. low012 2009-05-17 16:28:39 +00:00
  • 82af994041 added missing loglevel lotus 2009-05-16 08:21:51 +00:00
  • ad9762746d no exception in case of uniq() time-out, see also http://forum.yacy-websuche.de/viewtopic.php?p=13177#p13177 orbiter 2009-05-15 23:07:10 +00:00
  • 1efe686e3f fix for http://forum.yacy-websuche.de/viewtopic.php?p=13960#p13960 orbiter 2009-05-15 22:51:15 +00:00
  • 13fb84ab81 you can define your default number of search results displayed by search.items this applies only to requests through the classic-style page lotus 2009-05-15 14:48:34 +00:00