Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-07-19 08:44:42 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 2c58af6874 - added a short memory status simulation mode - added a button in PerformanceMemory_p.html to set the simulated short memory status - bugfix: added a missing lowercase in KeyList - better concurrency in loader dispatcher orbiter 2011-08-17 22:24:17 +00:00
  • c64faf41e2 addon to svn 7880 orbiter 2011-08-15 11:07:03 +00:00
  • 7b7a196243 ignore cookies in httpclient per default disable cookiestore,cause the default one caused segfaults on my peers this does not harm use of cookies via YaCy as proxy sixcooler 2011-08-14 12:28:28 +00:00
  • 06408a9428 since many POST-requests come as gzip they report a contentlength of -1 request memory of -1 * 3 look useless to me so I added some megs to it - even correct report of contentlength should not be harmed by this sixcooler 2011-08-13 01:04:37 +00:00
  • 411ed159f8 do some extra sleep while running low on memory (1 sec. per outofmemoryCycle) sixcooler 2011-08-13 00:59:59 +00:00
  • 6361f1d875 select the search window on focus so its easy to type in another query orbiter 2011-08-12 18:07:46 +00:00
  • 9ab0ba41e2 using GzipDecompressingEntity from httpclient instead of our own (was just fixed there in httpclient-4.1.2 and does a proper job) sixcooler 2011-08-12 17:51:30 +00:00
  • 52b477cf6f bump to httpclient-4.1.2, httpcore-4.1.3 - bugfixrelease sixcooler 2011-08-12 17:42:32 +00:00
  • ca09081341 better interaction orbiter 2011-08-12 17:13:34 +00:00
  • 3f0349e362 added a 'loading...' message orbiter 2011-08-12 16:09:40 +00:00
  • feac494f26 switch off real-time search if index is large orbiter 2011-08-12 14:42:20 +00:00
  • 07f5954570 try better handling of corrupt blobs @developer: please revert if I'm wrong see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=3334 sixcooler 2011-08-12 13:27:11 +00:00
  • f970670a7c - bugfix in ServerScannerList - speed up of generation of scanner list avoiding forced dns lookup orbiter 2011-08-12 13:21:18 +00:00
  • 8e03b8ee8b better integration of server list in interactive search orbiter 2011-08-12 12:25:45 +00:00
  • 606c5a9b40 added a serlvet that shows all scanned servers inside of the yacyinteractive search page orbiter 2011-08-12 10:31:01 +00:00
  • 0a3ab7da1b do not sort concrrently the same array orbiter 2011-08-12 08:06:21 +00:00
  • 77a9af99f1 same values for Xmx and Xms: memory extension may be difficult if the OS has not the remaining memory available and may kill the jvm. If the memory is reserved at the start but never used the OS may handle that as well and leave non-used space in swap area (and never swap) orbiter 2011-08-11 21:54:27 +00:00
  • 594d8f546a #cccamp11 maintenance fix: anons may find up to 1000 items in interactive search (was: 100) orbiter 2011-08-11 21:37:35 +00:00
  • eb14111200 encapsulate potential expensive objects in TextSnippet to allow GC them asap this reduces chance of OOMs at massive search & snippet-fetching sixcooler 2011-08-11 21:07:52 +00:00
  • 3aa6528ed0 the form value was not correctly interpreted cominch 2011-08-11 07:31:35 +00:00
  • 0d33cf352b removed synchronization in DNS resolve (solves a problem when loading snippets but in the past concurrent dns requests also caused deadlocks. but this is many years ago and we will give it another try) orbiter 2011-08-03 19:42:18 +00:00
  • e3fc1efbef performance hack and ensuring termination in serverAccessTracker. cause: "Session_:53600#0_POST /yacy/hello.html HTTP/1.1" prio=10 tid=0x2322b000 nid=0x3ba7 runnable [0x03d3e000] java.lang.Thread.State: RUNNABLE at java.lang.Long.valueOf(Long.java:557) at de.anomic.server.serverAccessTracker.clearTooOldAccess(serverAccessTracker.java:113) at de.anomic.server.serverAccessTracker.cleanupAccessTracker(serverAccessTracker.java:75) - locked <0x3bda2ae8> (a de.anomic.server.serverAccessTracker) at de.anomic.server.serverAccessTracker.track(serverAccessTracker.java:125) at de.anomic.server.serverSwitch.track(serverSwitch.java:542) at de.anomic.http.server.HTTPDemon.parseRequestLine(HTTPDemon.java:641) at de.anomic.http.server.HTTPDemon.POST(HTTPDemon.java:491) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at de.anomic.server.serverCore$Session.listen(serverCore.java:757) at de.anomic.server.serverCore$Session.run(serverCore.java:651) orbiter 2011-08-03 18:47:43 +00:00
  • 44d74f8f89 performance hacks for seed generation (because thread dumps showed multiple occurrences at these code points) orbiter 2011-08-03 18:32:11 +00:00
  • 5cd07d7f84 early freeing resources on deleting index reference if search-verification fails (aka Switchboard.cleanupJob) sixcooler 2011-08-02 15:52:33 +00:00
  • a311596881 finishing up my commits (7855-7858) which could be helpful for sixcooler 2011-08-01 23:35:24 +00:00
  • 9170a434ed throwing an exception again in FileUtils.copy(reader, writer) OOMs could occour here and should not be ignored sixcooler 2011-08-01 23:32:58 +00:00
  • c0caca57e3 stoping thread for fetching searchresults if running short on memory - in most cases at least one thread stays alive for getting the results - fewer threads should do the work with less resouces, but much slower then sixcooler 2011-08-01 23:32:29 +00:00
  • ce248cc8dd less byte-arrays of response-content, less byte-array <-> stream conversation sixcooler 2011-08-01 23:31:08 +00:00
  • 59b767eebd stop loading via http at defined maximum of bytes - even size is unknown before loading sixcooler 2011-08-01 23:28:23 +00:00
  • e131bf804d correct value @ low memory warning see: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3317 sixcooler 2011-07-26 01:05:28 +00:00
  • 48560a44a9 bump to httpcore-4.1.2: a bugfixrelease sixcooler 2011-07-25 00:48:29 +00:00
  • 916d79111e Runtime.maxMemory() DOES change @ runtime: I wondered getting Total-ram > Max-ram and MemoryControl.available() < 0 MemoryControl.available() < 0 causes some errors where its value is used for dimension of buffers for eg. sixcooler 2011-07-19 12:48:50 +00:00
  • 3a5fa73008 * revert parts of previous commit, because it breaks the trickle-feature f1ori 2011-07-19 12:04:40 +00:00
  • 6e79675ff3 * use gzip-encoding in more cases * send Expire-Header for static content * should improve webserver-performance for slow connections * fixes #37 f1ori 2011-07-19 11:47:53 +00:00
  • 299af4943c added another memory protection hack orbiter 2011-07-17 17:55:08 +00:00
  • 1f300217f8 more protection for the cleanup thread orbiter 2011-07-17 08:39:39 +00:00
  • d13103a0a7 changed the way how the index cache is flushed: do not flush when a put was made because that could cause that many put calls synchronize for a long time when the dump or a merge is performed. Instead a watchdog thread is doing the dump and therefore puts cannot block any more which is good when a put happens during a search result preparation. orbiter 2011-07-17 00:02:42 +00:00
  • b06faab9d3 do not allocate a StringBuilder object in case that there is not enough memory for that orbiter 2011-07-16 23:17:19 +00:00
  • 6a6f27eaf3 do not sort arrays again if arrays are already sorted orbiter 2011-07-16 19:21:39 +00:00
  • 3d043ce9d6 - refactoring - do not start worker threads in Array class if concurrency is not used orbiter 2011-07-16 19:13:30 +00:00
  • 48b78e9ff4 disabling concurrency in new sort since that is not working yet correctly orbiter 2011-07-16 11:54:47 +00:00
  • 62ac73a108 fixed bugs and deadlocks in core database indexing structures: - added new Array class that contains an abstraction of the java Arrrays class which replaces the home-brew quicksort algorithm. - the new class is about four times slower than the old one, but it works correct (the old one had errors) - fixed a synchronization problem orbiter 2011-07-16 10:08:43 +00:00
  • aff875baef smaler ping-entry @ ProfilingGraph sixcooler 2011-07-15 09:14:21 +00:00
  • 1912d0cccc changed handling of RowSet element retrieval: until today all elements had been copied from the underlying byte[] arrays into a new Entry object that again had a copy of a portion of that byte[] in its own bye[]. There was an option to just refer to the underlying byte[] with a pointer but that was almost never used. This commit now changes an interface to the Row class where it is now necessary to tell if a copy is always required. Fortunately the copy is only needed in very rare cases. That means that this change should cause much less memory allocation; it is expected that this happens especially during search situations. orbiter 2011-07-15 08:38:10 +00:00
  • bb8e3f8523 code cleanup orbiter 2011-07-14 21:42:30 +00:00
  • be15874be1 added request line in http which can support better debugging orbiter 2011-07-14 11:00:38 +00:00
  • 11dc653de3 added a visualization of peer pings to the performance graphic orbiter 2011-07-14 07:07:06 +00:00
  • 3a191cdf14 because newbies are scared about the memory consumption in the performance graph and arguments about high memory consumption according to bad knowledge about java garbage collection techniques, the memory display had been removed from the performance graph shown on the Status.html page. The memory graph can still be seen on the Performance page where the memory graph is just like it was. orbiter 2011-07-14 03:25:57 +00:00
  • 09bb7a390c do not replace malformed or invalid URLs in urlproxy cominch 2011-07-12 07:44:23 +00:00
  • c0d9474b31 update to eclipse class path environmen orbiter 2011-07-06 14:29:17 +00:00
  • 52d799e7c8 fix for solr auth orbiter 2011-07-05 09:21:30 +00:00
  • 9eb8e9acd9 no error message about missing browser in headless environments orbiter 2011-07-05 06:54:05 +00:00
  • d3c89b90ce temporary adding the old httpclient-3.1 again because the solrj classes need them. should be removed as soon solrj supports httpclient-4 orbiter 2011-07-04 17:04:49 +00:00
  • bd99969758 fixed bad query orbiter 2011-07-04 16:53:18 +00:00
  • 768c59740c - replaced solrj 3.1 with solrj 3.3 - updated also slf4j - added authentication for solrj orbiter 2011-07-04 16:35:30 +00:00
  • e7c7598923 docfix orbiter 2011-07-04 10:48:01 +00:00
  • c7b95e8c81 *) Invalid crawl profiles (containing invalid mustmatch/mustnotmatch filters) will be moved from active crawls to invalid crawls (new file: DATA/INDEX/freeworld/QUEUES/crawlProfilesInvalid.heap). This file can not be edited yet, but it shoudl be easy to extend the CrawlProfileEditor accordingly. *) Corrupt crawlProfilesPassive.heap would cause crawlProfilesActive.heap to be deleted. Don't know if this ever happend, but will not happen anymore. *) Cleaned up a little bit. *) Added some comments. low012 2011-07-03 23:55:55 +00:00
  • b84089ff04 fix for solr scheme list definition orbiter 2011-07-03 22:59:43 +00:00
  • fd02d6d9f8 fixed solr scheme table view orbiter 2011-07-03 22:55:36 +00:00
  • 4f730a711b same for debian as for latest commit orbiter 2011-07-03 21:40:12 +00:00
  • 60ee245486 setting startup options: -Xss256k and -XX:ReservedCodeCacheSize=1024m after appearance of a malloc error together with a crash of the jvm which stated at the end of the log: orbiter 2011-07-03 21:33:24 +00:00
  • 6d2e252bcf fix for: java.lang.NullPointerException at net.yacy.kelondro.index.RowCollection.<init>(RowCollection.java:97) at net.yacy.kelondro.index.RowSet.<init>(RowSet.java:48) at net.yacy.kelondro.rwi.ReferenceContainer.<init>(ReferenceContainer.java:58) at net.yacy.kelondro.rwi.ReferenceIterator.next(ReferenceIterator.java:69) at net.yacy.kelondro.rwi.ReferenceIterator.next(ReferenceIterator.java:43) at net.yacy.kelondro.blob.ArrayStack.merge(ArrayStack.java:1023) at net.yacy.kelondro.blob.ArrayStack.mergeWorker(ArrayStack.java:922) at net.yacy.kelondro.blob.ArrayStack.mergeMount(ArrayStack.java:869) at net.yacy.kelondro.rwi.IODispatcher$MergeJob.merge(IODispatcher.java:267) at net.yacy.kelondro.rwi.IODispatcher$MergeJob.access$300(IODispatcher.java:239) at net.yacy.kelondro.rwi.IODispatcher.run(IODispatcher.java:180) orbiter 2011-07-03 20:44:33 +00:00
  • 719777b2a7 replaced method to call getUsableSpace using reflection with direct call since we now use java 1.6 orbiter 2011-07-03 18:13:37 +00:00
  • 2d4bb139d3 - added counting of links with noindex tag for solr index - bugfixes for solr index orbiter 2011-07-03 06:40:05 +00:00
  • 528b59e078 replaced xerces.jar library that was originally added 2005 with SVN 126 to the libx directory and that was moved to lib in SVN 5781 the new replacement is taken from http://xerces.apache.org and has the version 2.11.0 and was inside the file Xerces-J-bin.2.11.0.tar.gz and consists of two files named xercesImpl.jar and xml-apis.jar The original purpose of that library was to support: - content parsers - optional seed uploader - SOAP API (which will be committed later) Since the SOAP API does not exist any more the purpose is to support content parser and an optional seed uploader orbiter 2011-07-02 22:33:35 +00:00
  • e7e1a0f328 replaced commons-io v1.4 with v2.0.1 orbiter 2011-07-02 21:10:13 +00:00
  • 5092a14bcb replaced fontbox, jempbox, pdfbox v 1.5 with v1.6 orbiter 2011-07-02 20:52:33 +00:00
  • 68681a9576 hint for proxy scraping lotus 2011-07-02 17:23:37 +00:00
  • fa6f2c2b44 use proxy accounts by default for more security http://bugs.yacy.net/view.php?id=45 lotus 2011-07-02 17:16:00 +00:00
  • 892caccdca added default configuration in ConfigurationSet in case of new values orbiter 2011-07-02 00:09:49 +00:00
  • 7bf39c8bcf added XX:MaxPermSize to debian and mac start scripts orbiter 2011-07-01 22:50:46 +00:00
  • bda3eec0ff added parsing of canonical link element to html parser orbiter 2011-07-01 16:38:01 +00:00
  • b6f09a475d - added an index profile editor in the /indexFederated_p.html servlet for solr indexes orbiter 2011-06-30 15:49:21 +00:00
  • 214ea005cf added "-XX:MaxPermSize=256m" to start script orbiter 2011-06-30 15:44:06 +00:00
  • b666a929e7 fixed Semaphore handling in case of interruptions orbiter 2011-06-30 15:37:14 +00:00
  • de7a054d77 added parser for such files like the new solr.key.list it parses text files with the following syntax: - all lines beginning with '##' are comments - all non-empty lines not beginning with '#' are keyword lines - all lines beginning with '#' and where the second character is not '#' are commented-out keyword lines orbiter 2011-06-29 15:35:45 +00:00
  • 6deef60bc0 added keyword list for solr index attributes orbiter 2011-06-29 15:33:27 +00:00
  • a17351dcfe * navigation bar for filetype constraints f1ori 2011-06-29 15:30:24 +00:00
  • 96957375cc * fix url proxy for relative links and chromium f1ori 2011-06-29 09:32:02 +00:00
  • fdc84d8319 small pi link on index page to administration pages f1ori 2011-06-29 09:32:00 +00:00
  • 9ebc75db4b fix for channel authorization orbiter 2011-06-26 23:14:02 +00:00
  • 267290a821 removed the semaphores from the cache dump process because I believe some of the semaphores may be lost somewhere which then causes that the cache is never flushed and then the peer dies from a OOM. The re-introduced synchronization may not be the best solution but should ensure that the caches are flushed. orbiter 2011-06-26 21:45:04 +00:00
  • 6d9e5865ee faster appearance of search result page (but complete search time is the same) this was inspired by http://bugs.yacy.net/view.php?id=37 orbiter 2011-06-26 21:17:02 +00:00
  • f7ca84cfc0 enhanced template engine orbiter 2011-06-26 21:15:13 +00:00
  • 4fe1329de2 *) trying to at least fix symptoms of http://forum.yacy-websuche.de/viewtopic.php?f=5&t=3293#p22791 low012 2011-06-25 10:15:42 +00:00
  • d8072d1866 added more info to DNS cache in /PerformanceMemory_p.html orbiter 2011-06-24 08:27:36 +00:00
  • f803da8aae code cleanup orbiter 2011-06-24 00:24:00 +00:00
  • 4999740790 added new navigation to search trailer json and xml files which causes that these navigation is also available in the search widget orbiter 2011-06-24 00:22:57 +00:00
  • 84c9658644 added a file type navigator added a protocol navigator orbiter 2011-06-23 15:39:52 +00:00
  • 31283ecd07 - added a search option to filter only specific network protocols. i.e. get only results from ftp servers. Just add '/ftp' to your search. for example search for "passwd /ftp". This can also be done with /http /https and /smb - fixed some search throttling processes that should protect your peer against search DoS or strong search load orbiter 2011-06-23 11:57:17 +00:00
  • 4b425ffdd2 fix for http://bugs.yacy.net/view.php?id=41 added another RSS channel "PROXY". the rss feed for peer news filters this channel if there is not an authorized access on that channel orbiter 2011-06-22 10:19:32 +00:00
  • a65ecffef6 fix for http://bugs.yacy.net/view.php?id=42 orbiter 2011-06-22 10:04:30 +00:00
  • 7db208c992 performance hacks: more pre-allocated StringBuilder orbiter 2011-06-21 23:10:50 +00:00
  • 87bd559c42 fixed warning orbiter 2011-06-20 22:53:43 +00:00
  • 07e89a7ae5 added @Deprecated orbiter 2011-06-20 22:33:45 +00:00
  • 9706fc55aa enhanced content scraper (should discover urls much faster in case of very large plain texts) orbiter 2011-06-20 22:29:45 +00:00
  • 996f0a8764 disabled assert in Base64Order which eats away too much performance during testing with -l orbiter 2011-06-19 13:34:55 +00:00
  • f667b9c289 enhanced identificator: using AtomicInteger for counter orbiter 2011-06-19 13:31:10 +00:00
  • 16327d1cbe unwrapping of call depth (one call less for UTF8.String) orbiter 2011-06-19 13:15:01 +00:00
  • f30d36b101 enhanced template engine orbiter 2011-06-19 13:02:06 +00:00