Mirrors/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2025-06-22 04:16:09 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 9cc8468b30 added tools to visualize image generation (i.e. during testing) Michael Peter Christen 2013-09-09 12:58:26 +02:00
  • 105cf8f593 changes to adjust jetty to recent code changes reger 2013-09-09 02:37:29 +02:00
  • aafef72a8a merged current rc1/master into jetty branch to allow further development with latest version ServerSideIncludes and servlet return values need further work (for working jetty integration) - TODO: added nasty quickfix to allow SSI - needs further work - TODO: YaCy servlet return values/parameters are not handled reger 2013-09-09 02:36:06 +02:00
  • dbef8ccfcb forced deletion of ZURL entries for a specific host for each host that appears in the crawl url list Michael Peter Christen 2013-09-05 13:22:16 +02:00
  • e137ff4171 refactoring (im preparation for new removeHost method) Michael Peter Christen 2013-09-05 09:59:41 +02:00
  • 7a5574cd51 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git Michael Peter Christen 2013-09-04 23:12:04 +02:00
  • 85456f46b2 added two new fields, exact_signature_copycount_i and fuzzy_signature_copycount_i, which count the number of copies of non-unique documents and assigns this to each document. Thus, each document there is a number assigned which shows how many copies of this document exists. These fields are disabled by default. Michael Peter Christen 2013-09-04 23:11:53 +02:00
  • 26366596d9 fix for a problem which ocurres when a site is crawled where the start url is redirected. orbiter 2013-09-04 16:00:47 +02:00
  • a2511b5600 turned images_alt_txt back to images_alt_sxt because it is not necessary to index the alt text. Indexed image Text is in images_text_t Michael Peter Christen 2013-09-04 10:47:18 +02:00
  • 85b1922244 activated image type navigation for image search Michael Peter Christen 2013-09-03 13:34:01 +02:00
  • 9e12fdff23 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git Michael Peter Christen 2013-09-03 12:22:57 +02:00
  • ab1201fdfd fixed wrong facet count Michael Peter Christen 2013-09-03 12:22:29 +02:00
  • 049c3b3f2e added an option to exclude image search results from text search. This is on by default. Michael Peter Christen 2013-09-03 11:14:23 +02:00
  • 69f85265e1 added an option to put image links to the crawl queue and handle these like normal documents. Using this option (by default on at this moment; this might change soon) it is possible to get the exif data into the search index to be used in image search. Michael Peter Christen 2013-09-03 11:13:45 +02:00
  • e8e558a9b7 fix for content domain classification in URIMetadataNode Michael Peter Christen 2013-09-03 10:49:09 +02:00
  • a8c5bfcf58 avoid to create unnecessary objects Michael Peter Christen 2013-09-03 09:48:05 +02:00
  • 5a0de1b77d moving image description text to image text field Michael Peter Christen 2013-09-03 09:47:27 +02:00
  • dc179bd61f fix for catchall query goal for image search Michael Peter Christen 2013-09-03 07:55:21 +02:00
  • 5d71a4c8bc fix for dc:description field Michael Peter Christen 2013-09-03 07:54:49 +02:00
  • 392174de8c remove all_words, all_strings lists from QueryGoal - only used for text highlighting in parser text (ViewFile.html) which can be done with include_strings only reger 2013-09-02 23:09:43 +02:00
  • 169ef8963d one more fix for image search Michael Peter Christen 2013-09-02 20:02:26 +02:00
  • cb85b22725 redesign of the image search process (with much better results, unfortunately the index schema has changed and p2p image search will not be muchmuch better until many people update) Michael Peter Christen 2013-09-02 18:55:38 +02:00
  • 6184fd9d9a fix for solr/gsa result logging Michael Peter Christen 2013-09-02 08:05:42 +02:00
  • 29967102a2 optimized QueryGoal (reducing mem and computation by removing all_hashes) - all_hashes used for text highlighting and word distance computation which can be done with include_hashes only reger 2013-09-02 04:19:53 +02:00
  • f106345eef link strings should not be tokenized orbiter 2013-09-01 14:35:36 +02:00
  • deadeb406e image alt tag strings should be tokenized orbiter 2013-09-01 13:48:10 +02:00
  • 5b14bdfffd npe fix orbiter 2013-09-01 13:28:37 +02:00
  • 3e5f8e29e2 next development release step to reflect the extension of the solr api with javabin format capability orbiter 2013-09-01 13:12:36 +02:00
  • 1ca4b9612c added special handling of the BinaryResponseWriter in the solr interface which makes it possible to use solrj with the javabin format which is much better (compressed, no xml overhead, java object streams) and faster. Furthermore, this enables the 'shards' option in the solr interface which connects one solr (YaCy) to another solr (YaCy) ad-hoc. orbiter 2013-09-01 13:11:40 +02:00
  • d0e78082d1 return field names in index instead of in schema for SolrServerConnector.getFields reger 2013-08-31 06:25:12 +02:00
  • 1a3e42eca4 index migration to lucene 4.4 Michael Peter Christen 2013-08-26 12:49:39 +02:00
  • a88a62f7aa added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url. Michael Peter Christen 2013-08-25 00:13:48 +02:00
  • 3c5abedabf NPE during shutdown fix Michael Peter Christen 2013-08-24 23:36:50 +02:00
  • e4cbe9232d fixed a crawler bug where a double-occurring url was not re-crawled because the double-check error was written to the error-db and never deleted. No the error-db is cleared on every start and these double-messages are not written to the error-db any more. Michael Peter Christen 2013-08-22 15:56:09 +02:00
  • 765943a4b7 Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent. Michael Peter Christen 2013-08-22 14:23:47 +02:00
  • 0f3d8890db removed an assert which causes a shortcut call circuit Michael Peter Christen 2013-08-22 10:12:25 +02:00
  • 6d5fefe060 added missing files :( Michael Peter Christen 2013-08-20 16:31:34 +02:00
  • 554c0351dd fix for http://bugs.yacy.net/view.php?id=286 Michael Peter Christen 2013-08-20 16:10:26 +02:00
  • 47b1c81d08 - refactoring - generalized writing of url attributes to solr documents - added more url attributes to error documents Michael Peter Christen 2013-08-20 15:46:04 +02:00
  • e6b423c4d9 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git Michael Peter Christen 2013-08-19 22:02:41 +02:00
  • 94bec24d14 add back menu to Surftips page (currently no menu is displayed) reger 2013-08-19 17:53:37 +02:00
  • 1f299b0d42 removed link.gif as link button because this image is now shown automatically for expernal links Michael Peter Christen 2013-08-19 10:54:23 +02:00
  • 1c62fa7698 fix for bad snippets in gsa api Michael Peter Christen 2013-08-18 10:37:25 +02:00
  • 48ddd50a6c html fix Michael Peter Christen 2013-08-17 09:32:24 +02:00
  • 697613170d less logging for postprocessing (this was a debugging logging with high CPU load) Michael Peter Christen 2013-08-17 09:25:32 +02:00
  • 96ae332427 revert del _blank (last commit) in template reger 2013-08-15 00:15:01 +02:00
  • 43348a98a9 add some href target=_blank to ext. links with external icon reger 2013-08-15 00:05:32 +02:00
  • b4016ff324 - remove possible double initialization of rdfa parser - use ordered list to use preferred parser for mime/extension first (relates to html, rdfa, argument parser) - harmonize xhtml extension config for the 3 html base parsers reger 2013-08-14 21:12:10 +02:00
  • 82d81a57bd info msg if no embedded Solr http://bugs.yacy.net/view.php?id=279 reger 2013-08-14 20:59:46 +02:00
  • f0575bd44b FieldReIndex: omit active vocabulary fields from reindex detection reger 2013-08-14 00:00:30 +02:00
  • a5019bc470 make Vocabulary Navigator tags a hard result entry filter by checking vocabulary tags also for rwi results (currently a filter is applied to the solr query) reger 2013-08-13 03:07:25 +02:00
  • a67a4b7d86 improve tld: query modifier filter pattern (to prevent tld:net accepting www.abcinet.org) reger 2013-08-12 21:20:23 +02:00
  • 02fe8b43ba Field Re-Indexing: display list of fields in reindex queue change servlet to display statistic on 1st click (instead after refresh) reger 2013-08-11 04:51:29 +02:00
  • 7f501b7c38 clear some caches before reporting low Memory do not break lines in Network-table-rows sixcooler 2013-08-08 14:38:26 +02:00
  • b355dd52c6 Index Administration - Field Re-Indexing: exclude internal Solr _version_ field from obsolete field check reger 2013-08-08 00:55:21 +02:00
  • 1bc6003057 rise autoCommit maxTime to 3 Minutes to reduce IO lower mergeFactor again (5) for less segments sixcooler 2013-08-06 03:58:53 +02:00
  • 5189620026 add branch to packet-name if not build from master sixcooler 2013-08-06 03:48:29 +02:00
  • 070bf85b33 css fix for IE10 showing border on all img within <a /> tag since introduction of external link icon (commit 112836dcc9) reger 2013-08-04 05:37:20 +02:00
  • 8a96140f92 fix / workaround for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4750 + Seed.hash should be final sixcooler 2013-08-01 16:40:58 +02:00
  • c0ff91b9a8 bugfix release 1.62 Michael Peter Christen 2013-08-01 12:36:59 +02:00
  • 2674d28ef4 protection against self-ping (may be cause by fraud attempts) Michael Peter Christen 2013-08-01 12:35:44 +02:00
  • 944ae5686c added donation plea to the about box as default (you can replace this in your peer!) orbiter 2013-08-01 12:11:56 +02:00
  • f3d001c7ab more space in the about section orbiter 2013-08-01 11:49:07 +02:00
  • e879b97b0a added line to enhance debugging Michael Peter Christen 2013-07-31 13:33:05 +02:00
  • 2857499467 fix to collection schema; bug appeared for _txt fields with empty String as content Michael Peter Christen 2013-07-31 13:32:05 +02:00
  • dbfa865700 added a stub of a class for crawler redesign Michael Peter Christen 2013-07-31 13:16:32 +02:00
  • 76afcccaaf fix for default boolean post values: the default value MUST NOT be TRUE, because it's normal that a boolean value is missing in the post argument if a checkbox is not selected. Added also some style enhancements to IndexFederated, removed the Solr attachment manual and replaced it with a link to the wiki which explains this in more detail. Michael Peter Christen 2013-07-31 10:49:26 +02:00
  • 252c525709 fixed feed api servlet and and enhanced RSSReader class orbiter 2013-07-31 06:18:30 +02:00
  • d38c3c14d8 fix for CGI test orbiter 2013-07-31 05:43:58 +02:00
  • 112836dcc9 Improved external links. Marc Nause 2013-07-30 21:40:37 +02:00
  • d64a094f0e External links in HTML interface are marked as external with small icon. Marc Nause 2013-07-30 20:46:51 +02:00
  • 31902f54df fix for NPE which happens within solr code at MultiMapSolrParams.java, line 52 in case that the array arr.length == 0 Michael Peter Christen 2013-07-30 14:32:59 +02:00
  • 5b7c0d0745 update to pdfbox 1.8.2 Michael Peter Christen 2013-07-30 14:14:16 +02:00
  • f13df9dbb6 migration to solr 4.4.0 Michael Peter Christen 2013-07-30 14:01:16 +02:00
  • dc1002e511 cleaned sourcepaths from eclipse classpath Michael Peter Christen 2013-07-30 13:05:32 +02:00
  • 1b09362949 next development cycle Michael Peter Christen 2013-07-30 12:51:00 +02:00
  • 58fe986cca Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git Michael Peter Christen 2013-07-30 12:49:14 +02:00
  • cf12835f20 replaced the single-text description solr field with a multi-value description_txt text field Michael Peter Christen 2013-07-30 12:48:57 +02:00
  • 7d53ac86a3 fix for Blacklist (-Administration) sixcooler 2013-07-29 19:09:28 +02:00
  • f2d99053ed Field Re-Indexing: prevent endless error loop in ReindexSolrBusyThread on Solr exception (by skipping query causing the exception) (occured during testing while working on q=store:[* TO *]) reger 2013-07-29 01:32:02 +02:00
  • 92d3f71b16 htmlParser: closes input stream -> changed it to leave it open for a reset (used by AugmentParser - even if this is practically not used), note: stream.close is done by caller (Textparser.parseSource) - removed unnecessary reset in AugmentParser - added stream.mark in tdfatripleimpl. to make stream.reset work here reger 2013-07-28 03:41:09 +02:00
  • f117ea0492 reverted start script options - yacy on windows did not start with the given values orbiter 2013-07-27 15:36:46 +02:00
  • 87cfeaa4f3 fix for npe orbiter 2013-07-27 15:20:09 +02:00
  • 268a36aaff emergency fix for crawler: this will otherwise cause loss of complete crawl queue if latency of remote system is too low orbiter 2013-07-27 11:59:07 +02:00
  • 743e4878a8 Release 1.6 Release_1.6 orbiter 2013-07-27 11:26:14 +02:00
  • e7fcb81cea we should not do too much greedylearning at this time as we don't have enough experience with it. set greedylearning.limit.doccount to a much lower limit. orbiter 2013-07-27 11:22:40 +02:00
  • d05e0c5368 wait a bit longer before doing the first peer ping orbiter 2013-07-27 11:00:35 +02:00
  • f425b2c61c re-try to fetch url after a soft commit orbiter 2013-07-27 10:56:02 +02:00
  • b8f57f7703 don't be noisy when doing background tasks that may be allowed to fail orbiter 2013-07-27 10:51:58 +02:00
  • bf0ad04e1b apply load limitation also to dht-in orbiter 2013-07-27 10:42:38 +02:00
  • 0343f0668c Fix for NPE: E 2013/07/26 20:29:29 BUSYTHREAD Runtime Error in serverInstantThread.job, thread 'net.yacy.search.Switchboard.cleanupJob': null; target exception: null java.lang.NullPointerException at net.yacy.search.schema.CollectionConfiguration.convergenceStep(CollectionConfiguration.java:1116) at net.yacy.search.schema.CollectionConfiguration.postprocessing(CollectionConfiguration.java:897) at net.yacy.search.Switchboard.cleanupJob(Switchboard.java:2296) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:107) at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:165) Roland Haeder 2013-07-27 10:19:46 +02:00
  • b58ca8622d Some cleanups: - added SKINS_PATH_DEFAULT as same as LISTS_PATH_DEFAULT was added - Added 'final' keyword to a string Roland Haeder 2013-07-26 19:51:34 +02:00
  • e2ee412160 Use SwitchboardConstants.LISTS_PATH_DEFAULT instead of 'DATA/LISTS' Roland Haeder 2013-07-27 10:12:58 +02:00
  • ae19401af0 Removed another duplicate occurance of Blacklist.BLACKLIST_FILENAME_FILTER Roland Haeder 2013-07-26 19:17:31 +02:00
  • 59225487ea Fix for blacklist export, also applied the filename filter here Roland Haeder 2013-07-26 19:09:38 +02:00
  • 952fc0e7bd Removed superfluous check for files ending '.black' as the previous commit already excluded all other files (e.g. .ser dumps), added logging in catch-all block Roland Haeder 2013-07-26 19:09:00 +02:00
  • 060fec1577 Reuse Blacklist.BLACKLIST_FILENAME_FILTER Roland Haeder 2013-07-26 19:07:42 +02:00
  • 29049c71f5 Possible fix for ticket http://bugs.yacy.net/view.php?id=270, the filter for only including *.black must be applied Roland Haeder 2013-07-26 18:32:04 +02:00
  • 7263bb82fb Fix for NPE on shutdown: java.lang.NullPointerException at net.yacy.search.Switchboard.storeDocumentIndex(Switchboard.java:2732) at net.yacy.search.Switchboard.access00(Switchboard.java:207) at net.yacy.search.Switchboard.run(Switchboard.java:3049) Roland Haeder 2013-07-25 12:02:11 +02:00
  • 13433d41a1 Log this exception better Roland Haeder 2013-07-27 09:54:51 +02:00