yacy_search_server

Mirrors/yacy_search_server

Fork 0

mirror of https://github.com/yacy/yacy_search_server.git synced 2025-11-03 23:26:12 -05:00

Commit Graph

Select branches

Hide Pull Requests

master

#1

#10

#100

#100

#101

#101

#104

#105

#108

#110

#114

#12

#122

#127

#127

#128

#135

#137

#137

#14

#144

#149

#151

#154

#155

#157

#159

#167

#167

#18

#183

#186

#188

#2

#21

#218

#219

#220

#226

#23

#238

#239

#240

#250

#258

#26

#260

#261

#27

#278

#28

#283

#29

#292

#3

#303

#32

#325

#327

#332

#333

#334

#338

#34

#355

#356

#357

#37

#38

#389

#39

#390

#4

#40

#402

#403

#404

#405

#408

#41

#41

#419

#42

#420

#436

#437

#439

#44

#440

#442

#443

#447

#448

#45

#45

#46

#461

#462

#463

#464

#47

#472

#478

#484

#485

#488

#489

#490

#495

#5

#5

#51

#52

#53

#533

#535

#54

#540

#541

#55

#553

#554

#556

#559

#56

#560

#562

#564

#567

#568

#575

#576

#58

#580

#583

#584

#585

#588

#589

#594

#596

#597

#598

#60

#600

#603

#605

#607

#61

#610

#612

#614

#615

#616

#617

#619

#62

#621

#628

#633

#642

#649

#65

#650

#651

#657

#66

#661

#661

#664

#666

#67

#670

#672

#678

#68

#682

#686

#688

#689

#69

#690

#692

#693

#694

#695

#696

#697

#698

#699

#700

#704

#71

#710

#710

#715

#718

#72

#721

#722

#730

#734

#737

#76

#77

#78

#8

#88

#9

#9

#91

#93

#93

#97

#97

#98

0.99

1.0

Release_1.0

Release_1.01

Release_1.02

Release_1.03

Release_1.04

Release_1.1

Release_1.2

Release_1.3

Release_1.4

Release_1.5

Release_1.6

Release_1.68

Release_1.7

Release_1.72

Release_1.80

Release_1.82

Release_1.90

Release_1.92

f57df061da upgraded commons-io to 2.17.0 Michael Peter Christen 2024-09-21 21:56:12 +02:00
c88c30a5c5 added an option to ViewFile to see all solr fields which contain texts Michael Peter Christen 2024-09-21 21:51:19 +02:00
c6127e4f2e feat(nix): add a nix flake Cody Mikol 2024-08-30 20:41:09 -04:00
6db374fdcf upgraded metadata-extractor to 2.19.0 Michael Peter Christen 2024-08-26 23:49:27 +02:00
3944984840 added snippet extraction with synonym matching Michael Peter Christen 2024-08-26 23:44:42 +02:00
d181b9e89b added deleted files from commit 254f12d60b which are still needed and had been linked outside of yacy/ui Michael Peter Christen 2024-07-24 15:57:51 +02:00
910a496c9f replaced http links with https Michael Peter Christen 2024-07-21 18:02:58 +02:00
fd45ccf76e added sponsoring images Michael Peter Christen 2024-07-21 17:25:22 +02:00
833d720989 upgraded ppt parser by migration of org.apache,poi from 3.17 to 5.3.0 This also fixes the security waning https://github.com/yacy/yacy_search_server/security/dependabot/37 Michael Peter Christen 2024-07-21 15:28:13 +02:00
687820788d this assert does not work because of the 9_0_0 solr version format. An 9_0 is expected but it does not work this way with this version. Michael Peter Christen 2024-07-21 13:33:47 +02:00
accf4e424b Merge pull request #649 from virginOne/master Michael Christen 2024-07-10 16:36:54 +02:00
2f5f3f8853 Merge pull request #650 from zutto/master Michael Christen 2024-07-10 16:35:39 +02:00
326b5f6e6e Merge pull request #651 from okybaca/removeui Michael Christen 2024-07-10 16:34:07 +02:00
254f12d60b removed yacy/ui as obsolete okybaca 2024-07-09 16:26:50 +02:00
5268ae2ce9 check the document protocol & host values before proceeding to form final url. zutto 2024-06-29 10:11:58 +03:00
962aaec0c0 Improve the clarity of deep crawl feature UI text on AutoCrawler zutto 2024-06-29 09:37:05 +03:00
d958d1c0c4 ensure that returned SolrDocument is not null zutto 2024-06-29 09:33:06 +03:00
f3cc818305 Merge branch 'yacy:master' into master virginOne 2024-06-23 19:52:54 +08:00
66cf7d4ca5 disables autowarm of filtercache, corrects luceneMatchVersion sgaebel 2024-06-12 12:59:34 +02:00
89c07f0900 Fix the issue of not being able to import the JSON format export of Solr index due to the inconsistency in time format between the exported JSON format and the Solr time format. Virgo 2024-06-19 21:14:45 +08:00
70454654f3 by default open the https url for a given host, not the http url (http does almost not exist any more) Michael Peter Christen 2024-05-27 00:53:18 +02:00
71a6074cc5 added setting of cache configuration for solr according to recommendation from https://community.searchlab.eu/t/yacy-support-gpt-chatgpt-assistant/1622 However it is not clear if this configuration actually works (has an effect at all) or is the solution for performance issues. Michael Peter Christen 2024-05-26 12:59:59 +02:00
b8479430b6 400 is too small Michael Peter Christen 2024-05-25 01:21:06 +02:00
5f4ea9ac5d reduced memory amount for network image reduced also the number of memory allocation for image storage Michael Peter Christen 2024-05-25 01:10:23 +02:00
fe4c0aa890 refactoring of RAG reverse proxy: extracted code for ollama code to their own classes Michael Peter Christen 2024-05-21 00:06:19 +02:00
8b65a4d14e fixed build clean Michael Peter Christen 2024-05-19 18:02:49 +02:00
f1c70dce33 Merge branch 'master' of github.com:yacy/yacy_search_server Michael Peter Christen 2024-05-19 17:35:24 +02:00
8eb0d490aa migrated solr to 9.0 This is a major step because solr removed support for embedded solr instances in 9.0 and we want to keep it because we want to ship YaCy with an embedded solr. It was necessary to add parts of solr code into YaCy to make this migration possible. Further on with Solr 9.1 they removed even more parts which are required for embedded operation, therefore we cannot migrate yet further without big changes. If you are running a YaCy instance with Solr 8.x, the migration should be done automatically. If not you require to first migrate to a YaCy version 1.93 with Solr 8.x to migrate to Solr 8 data. Michael Peter Christen 2024-05-19 17:34:57 +02:00
b8417e5619 removed Mac specific code which is not working any more on recent Macs Michael Peter Christen 2024-05-19 17:29:16 +02:00
13fbff0bff Added a RAG Proxy for AI Chat with YaCy Michael Peter Christen 2024-05-19 17:19:09 +02:00
59c0cb0f30 fixed aarch64 dockerfile Michael Peter Christen 2024-05-13 02:10:24 +02:00
0405ec8ad2 Merge pull request #642 from HeliosLHC/reduce_image_size Michael Christen 2024-05-12 23:50:24 +02:00
c2ad1950e8 updated jetty to 9.4.54.v20240208 Michael Peter Christen 2024-05-10 15:41:20 +02:00
b295e38969 fine-tuned the import process of jsonl files which had been missing to actually be able to make searches and browse the index with the host browser Michael Peter Christen 2024-05-10 12:13:44 +02:00
262b23532d reduce image size HeliosLHC 2024-05-05 23:23:34 +02:00
de941c6fee Merge pull request #633 from LillySchramm/fix/632/remove-sayat-link Michael Christen 2024-04-05 17:43:55 +08:00
160e346d4c fix: Remove SayAt.Me Link EPS-DEV 2024-04-04 15:05:41 +00:00
656d47c1ac Merge pull request #628 from frankenstein91/vagrant Michael Christen 2024-02-27 14:18:24 +01:00
9a406d310c switch user Frank Tornack 2024-02-04 19:36:48 +01:00
6864486196 add Yacy vagrant Frank Tornack 2024-02-04 19:29:16 +01:00
331e0a24fc Merge pull request #621 from OFA54/patch-1 Michael Christen 2023-12-20 23:34:35 +01:00
d825a85a01 Merge pull request #619 from pr0vieh/initrecrawl Michael Christen 2023-12-09 14:47:13 +01:00
35620762ac bring defaults for recrawlindex to init config pr0vieh 2023-12-09 01:32:31 +01:00
d097a642c2 Merge pull request #615 from okybaca/logging2 Michael Christen 2023-12-03 16:40:21 +01:00
6d5e9ff53f Merge pull request #616 from okybaca/logging3 Michael Christen 2023-12-03 16:39:29 +01:00
d5d4e8fe3a Merge pull request #617 from pr0vieh/master Michael Christen 2023-12-03 16:38:46 +01:00
dfb2b79609 Add setting for DHT receive loadprereq insted of hardcoded load < 2.0 pr0vieh 2023-12-03 01:27:36 +01:00
5dee8dbcbd changed the log entry REJECTED to CRAWLER * REJECTED, loglevel fine okybaca 2023-12-02 12:24:36 +01:00
4c603e23f0 Merge pull request #610 from okybaca/cr-text Michael Christen 2023-11-27 12:17:05 +01:00
040cd8be6d Merge pull request #612 from okybaca/sitemap-fix Michael Christen 2023-11-27 12:16:43 +01:00
0233ecd481 Merge pull request #614 from okybaca/logging Michael Christen 2023-11-27 12:15:34 +01:00
7831f294a9 changed regular peerping messages to level fine okybaca 2023-11-27 08:12:03 +01:00
553c859703 logging: moved some log-cluttering DHT messages to level 'fine' okybaca 2023-11-27 07:51:42 +01:00
1c5fca9a58 changed network operation log category from YACY to NETWORK okybaca 2023-11-26 12:24:09 +01:00
2f44fc0257 added some logging prefixes to yacy.logging okybaca 2023-11-25 18:39:08 +01:00
89c2a92cfb tr.lng OFA 2023-11-18 01:03:28 +03:00
3d3bdb0f5f added zim importer rule for mdwiki Michael Peter Christen 2023-11-16 23:11:57 +01:00
4a611ac6a3 another possible fix for https://github.com/yacy/yacy_search_server/issues/500 Michael Peter Christen 2023-11-15 23:45:53 +01:00
9c59c6814b updated apache libs okybaca 2023-11-15 10:22:00 +01:00
d72cd7916c Merge branch 'master' of https://github.com/yacy/yacy_search_server sgaebel 2023-11-14 20:43:42 +01:00
0663ae3c99 adds synchornized dumplog sgaebel 2020-12-01 22:34:30 +01:00
cba84632ee UI: added a more descriptive message, CitationRank instead of cr okybaca 2023-11-14 00:05:23 +01:00
cff0991d85 test if this is helpful for https://github.com/yacy/yacy_search_server/issues/500 Michael Peter Christen 2023-11-13 16:41:19 +01:00
ceb07a5218 fixed problem with zim importer which crashed when non-valid urls appeared Michael Peter Christen 2023-11-13 11:12:10 +01:00
656b3e3e77 updated guava to latest and added missing library for failureaccess Michael Peter Christen 2023-11-13 10:59:49 +01:00
3268a93019 added a 'minified' option to YaCy dumps Michael Peter Christen 2023-11-13 10:27:50 +01:00
c20c4b8a21 modified export: added maximum number of docs per chunk The export file can now be many files, called chunks. By default still only one chunk is exported. This function is required in case that the exported files shall be imported to an elasticsearch/opensearch index. The bulk import function of elasticsearch/opensearch is limited to 100MB. To make it possible to import YaCy files, those must be splitted into chunks. Right now we cannot estimate the chunk size as bytes, only as number of documents. The user must do experiments to find out the optimum chunk max size, like 50000 docs per chunk. Try this as first attempt. Michael Peter Christen 2023-11-12 22:11:55 +01:00
655d8db802 detailed directions in index export to explain how the export can be imported again using elasticsearch/opensearch Michael Peter Christen 2023-11-12 15:26:18 +01:00
24011dcbcc more file name extensions for json list surrogate files Michael Peter Christen 2023-11-06 22:44:18 +01:00
34a9fc1a07 bugfixes to zim reader: Michael Peter Christen 2023-11-05 12:46:37 +01:00
7db0534d8a Added a zim parser to the surrogate import option. You can now import zim files into YaCy by simply moving them to the DATA/SURROGATE/IN folder. They will be fetched and after parsing moved to DATA/SURROGATE/OUT. There are exceptions where the parser is not able to identify the original URL of the documents in the zim file. In that case the file is simply ignored. This commit also carries an important fix to the pdf parser and an increase of the maximum parsing speed to 60000 PPM which should make it possible to index up to 1000 files in one second. Michael Peter Christen 2023-11-05 02:16:40 +01:00
70e29937ef added a check in zim importer which tests if import URLs actually exist Michael Peter Christen 2023-11-04 19:07:50 +01:00
496f768c44 modified cache strategy for zim clusters Michael Peter Christen 2023-11-03 18:20:10 +01:00
fdc6311dc7 added parsing rules for wikibooks and wikinews in zim reader Michael Peter Christen 2023-11-02 00:27:24 +01:00
2ea54b3503 fixed blob iterator in zim cluster definition Michael Peter Christen 2023-11-01 23:43:27 +01:00
54fa5d3c2e added a cluster cache but it requires more testing Michael Peter Christen 2023-11-01 19:52:44 +01:00
53b01dbf2e Merge branch 'master' of https://github.com/yacy/yacy_search_server.git Michael Peter Christen 2023-11-01 18:57:04 +01:00
41856e9f34 added an optimized zim file entry iterator Michael Peter Christen 2023-11-01 18:50:28 +01:00
1c0df28bfb added a zim importer that can be used for surrogate imports. Can not be used yet because it requires some security additions to verify that the given urls actually work. Michael Peter Christen 2023-11-01 18:48:40 +01:00
b9912ff50d repaired dockerfiles for aarch64 and armv7 Michael Peter Christen 2023-10-29 22:09:24 +00:00
33b6878ded Merge branch 'master' of https://github.com/yacy/yacy_search_server.git Michael Peter Christen 2023-10-29 14:58:47 +01:00
68554cea07 Merge pull request #605 from okybaca/readme-docker-link Michael Christen 2023-10-29 14:56:26 +01:00
06bfd5802f Merge pull request #603 from okybaca/dark-green-css Michael Christen 2023-10-29 14:55:58 +01:00
43d5cd101e Merge pull request #607 from okybaca/wikilinks Michael Christen 2023-10-29 14:55:26 +01:00
4add1f6bc7 replaced all the links to legacy legacy wiki to legacy wiki okybaca 2023-10-29 13:12:24 +01:00
e2c86a8eba added a ZIM cluster pointer cache Michael Peter Christen 2023-10-29 12:49:08 +01:00
4a54b24703 fix for "negative seek offset" error during extension of heap files. This would have always happend when a heap file exceeds 2GB. should fix https://github.com/yacy/yacy_search_server/issues/372 Michael Peter Christen 2023-10-29 09:32:21 +01:00
69db75ce45 added a link to docker build guide okybaca 2023-10-29 02:35:57 +01:00
9c8fb97985 introduced url list and title list caching and enhanced input stream performance in ZIM reader Michael Peter Christen 2023-10-29 00:43:12 +02:00
b0ae660790 added Zstandard compressed data decompression for ZIM files type 5 also: more generalization and performance enhancements Michael Peter Christen 2023-10-28 12:24:29 +02:00
ad8ee3a0b6 fixed typo in class name Michael Peter Christen 2023-10-28 08:57:42 +02:00
c4082c4ff2 refactoring of ZIM reader, simplification, removed unnecessary code Michael Peter Christen 2023-10-28 08:56:58 +02:00
c2b6b6e7b9 Fixed a large number of problems in the ZIM reader. This library was not prepared for large data because it was missing long data types for pointers. I had to modify the code-base in a fundamental way: - Proof-Reading, - unclustering, - refactoring, - naming adoption to https://wiki.openzim.org/wiki/ZIM_file_format, - change of Exception handling, - extension to more attributes as defined in spec (bugfix for mime type loading) - bugfix to long parsing (prevented reading of large files) The code is furthermore very inefficient and requires more attention. However the format is very useful for YaCy as there are numerous data sources for ZIM-Files. Michael Peter Christen 2023-10-27 15:49:23 +02:00
5ba5fb5d23 upgraded pdfbox to 3.0.0 Michael Peter Christen 2023-10-27 12:05:24 +02:00
c10944bd4a updated bcmail-jdk15on 1.75 to bcmail-jdk18on 1.67 Michael Peter Christen 2023-10-27 11:08:19 +02:00
1fefae9baf integrated the source code of a openzim file format reader. These are the raw format reader files with no integration in YaCy yet, which will maybe follow as a next step. The zim file format is documented in https://openzim.org and the reader code was taken from the archived, non-maintained repository at https://github.com/openzim/zimreader-java Michael Peter Christen 2023-10-27 10:59:06 +02:00
ec2d14e973 fine tuning the dark-green color scheme okybaca 2023-10-26 12:35:22 +02:00
4308aa5415 removed concept of empty passwords as "no passwords used", because we now start YaCy with a default password (yacy). This has impact of all function that check the current state of password-protection that included the empty password situation, including the warnings to set a password in case that none is set (which cannot be the case any more). Michael Peter Christen 2023-10-25 22:56:06 +02:00
2c60ff14bb fixed default pw comparison Michael Peter Christen 2023-10-25 13:59:02 +02:00
4da320bebf added a warning message in ConfigBasic in case that the default password was not changed. Michael Peter Christen 2023-10-24 23:36:26 +02:00