Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-11-09 00:16:11 -05:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • b6d67507db * implement proxy Florian Richter 2011-04-09 11:48:44 +02:00
  • 78d6d6ca06 refactoring for ymarks apfelmaennchen 2011-04-08 21:15:10 +00:00
  • 399d7d6878 * fix permissions of bin/-folder in debian package f1ori 2011-04-07 07:31:17 +00:00
  • 9ac02caf00 different initialization of empty variables in alternative constructor. This leads to wrong interpretation of user credentials, resulting in unnecessary "@" in front of host, and different urlhash values. cominch 2011-04-06 10:59:31 +00:00
  • a47bdc405b better logging for robinson selection according to peer tag orbiter 2011-04-05 08:04:25 +00:00
  • cafcb1f9ed removed the DNS resolving for web structure computation from the indexing queue and placed it in a concurrent computation queue that does not block the crawler. Makes crawling faster and less DNS-speed-dependent orbiter 2011-04-04 22:01:07 +00:00
  • 57ce1fb491 reverted synchronization from SVN 7641 orbiter 2011-04-04 20:31:02 +00:00
  • 17530ca7b5 fix for bug http://bugs.yacy.net/view.php?id=10 orbiter 2011-04-04 12:20:20 +00:00
  • 7c8e764201 removed synchronization again... orbiter 2011-04-04 10:13:30 +00:00
  • 96c32e87b0 fixes to crawler and new user-agent crawl-delay handling orbiter 2011-04-04 09:47:18 +00:00
  • b2fe4b7b1a added a handling of appearances of yacy bot entries in robots.txt if this entry addresses the yacy peer (directly or indirectly) and it grants a crawl-delay of 0. Then all forced pause mechanisms in YaCy are switched off and the domain is crawled at full speed. crawl delay values can be assigned to either - all yacy peers using the user-agent yacybot - a specific peer with peer name <peer-name>.yacy or - a specific peer with peer hash <peer-hash>.yacyh orbiter 2011-04-03 23:39:45 +00:00
  • 21fe5e6c6a * add bin-folder to debian package f1ori 2011-04-02 10:58:56 +00:00
  • e25c1f2ea3 *) preventing whitespace keys in config file low012 2011-04-02 09:25:07 +00:00
  • cb6f709a16 - enhancements in surrogate reading - better display of map in location search orbiter 2011-04-02 00:11:37 +00:00
  • 1ff9947f91 *) added new user right: extended search right (allows to define users who can query more results than anonymous users) *) cleaned up code a little bit low012 2011-04-01 23:32:40 +00:00
  • 564184909a enhanced the surrogate parser: better reading of UTF-8 characters orbiter 2011-04-01 11:05:42 +00:00
  • 156cf02703 - added an index constraint 'has location' to the condenser - added evaluation of the 'has location' constraint to search using the /location operator orbiter 2011-03-31 09:41:30 +00:00
  • 41b8d7f655 fix for url normalization (no backpath resolving in post parameters) orbiter 2011-03-31 09:40:01 +00:00
  • 0430a94eaa the location search shows now not re-evaluated locations but only such locations that are attached as metadata to web pages - added parser for in-text appearing geo-locations - added geo-locations to rss search result - added evaluation of metadata-attached geo-locations in yacysearch_location to show search results within a map orbiter 2011-03-30 23:26:36 +00:00
  • 8412f8787d fix for http://bugs.yacy.net/view.php?id=8 orbiter 2011-03-30 08:17:25 +00:00
  • 9b25d07295 - added geo information parsing to html parser - extended metadata information in index with geolocalisation - added display of location in yacydoc and ViewFile orbiter 2011-03-30 00:49:47 +00:00
  • efcf37a953 * show info in log, if robots.txt is rejected due to wrong mime-type f1ori 2011-03-28 19:55:15 +00:00
  • cbf87fe72f write PID to yacy.running lotus 2011-03-26 15:11:29 +00:00
  • 351d264a48 * yacy domain handler for jetty * rewrite from / to /index.html Florian Richter 2011-03-26 00:18:48 +01:00
  • 06afa94f9d hups lotus 2011-03-24 06:24:37 +00:00
  • a9a9db98c8 better rename modified version lotus 2011-03-23 20:22:08 +00:00
  • e19ca27004 do not autocomplete on mouseover. this has resulted in unwanted autocomplete. fixes bug #3 lotus 2011-03-23 20:13:43 +00:00
  • 16cd919795 *) fixed Exceptions which caused 500 error when entering invalid URL mask or invalid prefer mask, invalid masks are ignored, error message is displayed on yacysearch.html (what about yacysearch.rss and yacysearch.json?) *) fixed "more options" link on yacysearch.html low012 2011-03-23 00:48:19 +00:00
  • 1a24917cea *) fixed NPE which occured when empty String was entered as search word low012 2011-03-23 00:44:38 +00:00
  • 01b968d836 better concurrency in ViewImage icon cache and OOM protection for too large icon caches orbiter 2011-03-22 11:00:55 +00:00
  • b1a8d0c020 enhancements to web cache and less strict caching rules orbiter 2011-03-22 10:35:26 +00:00
  • f3baaca920 - enhancements to DNS IP caching and crawler speed - bugfixes (NPEs) orbiter 2011-03-22 09:34:10 +00:00
  • e7860b1239 *) <mode="Homer">D'oh!</Homer> low012 2011-03-21 22:23:20 +00:00
  • 82f1580a60 *) trying to fix ConcurrentModificationException low012 2011-03-21 22:20:19 +00:00
  • df71776929 * fix bug #7 * log requires poison to finish, so Base64Order main-function doesn't finish, when called from debian configure script f1ori 2011-03-21 19:42:22 +00:00
  • 9f0286b380 *) fixed potential "java.lang.IllegalArgumentException: Illegal group reference" which occured if special characters which are also used as metacharacters in regular expression were used inside of <pre>...</pre> (see: http://veerasundar.com/blog/2010/01/java-lang-illegalargumentexception-illegal-group-reference-in-string-replaceall/) low012 2011-03-21 18:02:09 +00:00
  • 78d4c45d09 enhancement during search process: fast fail of search in case that all index feeder have terminated. This change should affect filtering and navigators and should cause that search navigation gets faster orbiter 2011-03-21 13:05:51 +00:00
  • ba03ca8620 added more configuration options for search: - removed configuration button for 'search only for admin' from index.html and added this to ConfigPortal - added configuration of link verification options (iffresh, cacheonly, nocache, ifexist) to ConfigPortal - added configuration of navigation options to ConfigPortal - added an option to switch off automatic index cleaning in case that a link verification method fails orbiter 2011-03-21 07:50:34 +00:00
  • e0c7d490f9 * fix bug #6 * exclude signature files from auto-deletion of unknown files in DATA/RELEASE f1ori 2011-03-20 17:59:58 +00:00
  • 18ec7fe53c added a clearall.sh script that deletes the complete index and everything else that belongs to crawling orbiter 2011-03-20 08:36:29 +00:00
  • d98884f1d5 added script for importmediawiki.sh in build.xml orbiter 2011-03-19 23:58:11 +00:00
  • a50f28e6e7 - fixed missing save operation for peer name change - fixed import of mediawiki dump files - added script to add mediawiki dump files orbiter 2011-03-19 23:52:09 +00:00
  • 2b5f8585bf performance hack for Balancer and ip address parsing orbiter 2011-03-17 21:09:18 +00:00
  • 43e1660512 fix/enhancement in Crawler: do not generate domain match pattern if crawl depth is 0 orbiter 2011-03-17 21:07:44 +00:00
  • b1d133b69f another anhancement to the ThreadDump function: better multiple dumps and filtering out of not interesting dump parts orbiter 2011-03-17 20:48:39 +00:00
  • f25cc4407d * authentication complete (using old credentials from config file) Florian Richter 2011-03-17 20:40:05 +01:00
  • a35d513bd8 fix for not-deleted .gap and .idx files see also: http://forum.yacy-websuche.de/viewtopic.php?p=22128#p22128 orbiter 2011-03-17 17:09:19 +00:00
  • 7cfd3762d9 * authentication implemented with own securityhandler Florian Richter 2011-03-16 17:39:31 +01:00
  • a6935e7dc8 fix for active dns resolving: do not resolve in case that the dns server is not available (offline mode) orbiter 2011-03-16 07:05:10 +00:00
  • 859c99886c fix for multiple thread dump orbiter 2011-03-15 23:05:51 +00:00
  • 61acf55da4 avoided using a synchronized(this) for the hash computation to prevent that the lock on the object is (accidently) stolen by another thread and replaced this synchronization using the protocol object. Made also the protocol object final. orbiter 2011-03-15 09:52:39 +00:00
  • c2a968c23f fix for bug in formatting in ThreadDump and added hint for linux/Mac users that they may use the LOCKED feature using the start option -l orbiter 2011-03-15 08:39:05 +00:00
  • 2861d0888a *) simplified code\n*) fixed potential NumberFormatExceptions low012 2011-03-15 01:03:35 +00:00
  • 078ecacf61 avoid synchronization in DigestURI hash requests orbiter 2011-03-15 00:47:30 +00:00
  • 68ca0fbb2e * add copyright info * implement basic authentication * update jetty to 7.3.0 Florian Richter 2011-03-15 00:33:36 +01:00
  • 1989ebc24b removed more warnings orbiter 2011-03-14 22:52:30 +00:00
  • 0324de1467 removed debug line orbiter 2011-03-14 21:34:42 +00:00
  • 1aba7869bf patch for Windows: do not use the thread lock feature from previous commit if used on Windows orbiter 2011-03-14 21:33:36 +00:00
  • 0a11727374 added new feature for Thread dump: "THREADS WITH STATES: LOCK FOR OTHERS" will show only such threads that lock other threads. This is the 'opposite part' of the blocked threads. Because that this uses a thread dump that is produced with a kill -3 on the PID of the process and such thread dumps are written by the Java core outside of System.out and Sytem.err it is necessary to read the dump from a log in the file system. Such a log is only written if YaCy is started with startYACY.sh on a linux system. That means: this feature is only available on linux and Mac OS X if YaCy is started with ./startYACY.sh -l orbiter 2011-03-14 21:32:20 +00:00
  • b62b79675b removed type cast warnings orbiter 2011-03-14 21:08:18 +00:00
  • a07a1a8b1e removed type cast warnings orbiter 2011-03-14 21:07:15 +00:00
  • 8edaccfedf removed unused variables orbiter 2011-03-14 21:03:37 +00:00
  • e6c3507b17 disabled some of the previous changes (did not work in openjdk) orbiter 2011-03-14 20:48:36 +00:00
  • f9e5c21083 update to thread dump logs orbiter 2011-03-14 20:46:04 +00:00
  • ed3bcfaf71 * SSI work with jetty, it's pretty usable now Florian Richter 2011-03-14 21:17:01 +01:00
  • 4a237bfa5d Windows Installer: - add support for Windows Firewall on Win XP (SP2/SP3), Vista and Win 7 (open port 8090) - this should cover almost every Windows installation at home pca 2011-03-14 19:32:08 +00:00
  • 9199b9e3c6 also putting jcifs-1.3.15 into classpath (let me me build YaCy again :-) sixcooler 2011-03-13 22:44:50 +00:00
  • 9956dc9dce Update jcifs-library to Version 1.3.15. Small Changes, read: http://jcifs.samba.org/ - "Minor adjustments have been applied to DcerpcHandle locking routines in the SID class to fix sporadic occurances of "All pipe instances are busy" errors under high load." suessthomas 2011-03-13 20:09:52 +00:00
  • 8f11d3a5bb redesigned the ScoreMap classes: - new concurrent score map using atom operation from java concurrency classes - redesigned difference beween StaticScore and Dynamic Score into ScoreMap and ReversibleScoreMap allowed that many classes can now use simple ScoreMap Objects which can be used better in concurrent environments using the ConcurrentScoreMap - switched from DynamicScore to ConcurrentScoreMap usage wherever possible orbiter 2011-03-13 01:41:44 +00:00
  • a564230c48 more enhancements against blocked threads occurred in seed age evaluation (blocks httpd in some cases) orbiter 2011-03-12 22:54:41 +00:00
  • 2d2df64870 * jetty works for many pages now Florian Richter 2011-03-11 13:15:56 +01:00
  • dc0db3550e avoid string conversion orbiter 2011-03-11 00:59:27 +00:00
  • 694fa3a2a5 - replaced more direct string-based UTF-8 conversions by predefined UTF-8 conversion - changed menu structure slightly orbiter 2011-03-10 23:25:07 +00:00
  • bbb7aea8f3 fix basic config change in portal mode lotus 2011-03-10 20:04:15 +00:00
  • df68bf6001 Windows Installer: - check Windows-Version on startup, support only Windows 2000 and newer (necessary for Sun-JRE and as preparation for firewall section) - little changes in JRE section handling pca 2011-03-10 19:30:05 +00:00
  • 30aed9824a moved getBytes() to UTF8.getBytes() to use a default String encoding orbiter 2011-03-10 12:35:32 +00:00
  • cb6d307bba adding extension for parser lotus 2011-03-09 20:36:01 +00:00
  • 61a64bdbef Windows Installer: - detect JRE at startup, showing install-option depends on result - hide window for external call "attrib" - some cleanup and restructure for readability pca 2011-03-09 20:19:34 +00:00
  • 4d733608fb fix for broken JSON, see: http://forum.yacy-websuche.de/viewtopic.php?p=22162#p22162 orbiter 2011-03-09 20:08:20 +00:00
  • 1214615185 fix for 'invisible entry', see http://forum.yacy-websuche.de/viewtopic.php?p=22133#p22133 orbiter 2011-03-09 17:04:34 +00:00
  • 3820525464 more memory protection: auto-flush of caches in case of memory shortage orbiter 2011-03-09 16:32:34 +00:00
  • 7962d35425 - removed file upload function in crawl start and replaced it with an input field for a file path where the crawl start file is loaded. This was necessary to support the API steering for file crawl starts, for two reasons: 1) if the file is changed for a re-crawl this is not reflected in the steering because it would take the previously uploaded crawl start file 2) browsers do not submit the full path of the selected file even if this path is shown in the input field because of security reasons. There is no work-around or hack to make the submission of the full path possible orbiter 2011-03-09 12:50:39 +00:00
  • 1989ba64c0 * jetty Florian Richter 2011-01-17 12:06:17 +01:00
  • adbdabb5e7 Merge branch 'master' of github.com:f1ori/yacy Florian Richter 2011-03-09 12:17:45 +01:00
  • 96bb33ed9b added default size to StringBuffer in logger (and it is not possible to replace the StringBuffer with a StringBuilder...) orbiter 2011-03-09 09:53:57 +00:00
  • e1b6916423 always try to guess the size of a StringBuilder to prevent too many memory re-allocations orbiter 2011-03-09 09:29:05 +00:00
  • bea8137997 *) minor changes *) fixed potential NPE in suggest.java low012 2011-03-08 23:27:41 +00:00
  • 3e03963b1c *) minor changes low012 2011-03-08 22:37:17 +00:00
  • 8b94e33a33 Cleanup for last commit (forget some unnecessary parts) pca 2011-03-08 19:11:49 +00:00
  • b49f8894e1 Add check for hard drive and free space to Windows-Installer pca 2011-03-08 19:07:22 +00:00
  • 31a8ebe02a Merge branch 'master' of github.com:f1ori/yacy Florian Richter 2011-03-08 15:21:20 +01:00
  • 3b40b98256 *) set SVN properties *) minor changes low012 2011-03-08 01:51:51 +00:00
  • 2af8e33773 better performance computing search targets with index abstracts orbiter 2011-03-07 23:32:01 +00:00
  • 619b561a4a enhanced secondary search: index abstracts decompression is now much faster and does not cause strong CPU load after several searches with more than one word orbiter 2011-03-07 23:12:39 +00:00
  • bf27a72d53 *) set SVN properties low012 2011-03-07 23:05:23 +00:00
  • b649ce2dd7 *) minor changes low012 2011-03-07 22:59:19 +00:00
  • 0da3b6489e *) added the only changes from r7557 which actualy made sense *) caught potential exception (occured when user entered a string which did not contain digits only for the maximum number of lines) *) use prop.putHTML to avoid potential XSS attack in case an attacker manages to cause something to end up in the logs which contains a string which was defined by the attacker low012 2011-03-07 21:44:58 +00:00
  • 27ecdb5444 use less peers for remote search orbiter 2011-03-07 21:24:46 +00:00
  • 29acd2f108 reverted also changes in ViewLog from SVN 7557 because the ThreadDump submenu was not visible any more. orbiter 2011-03-07 21:19:47 +00:00
  • 70a996a06c reverted SVN 7557 because these classes are called using reflection. The class declaration is in the log configuration. Without these classes you get errors during runtime and a non-formatted log output, i.e.: orbiter 2011-03-07 20:42:19 +00:00