Ivan Skytte Jørgensen 43ee356e10 tokenizer compilation fix

2018-07-26 17:31:36 +02:00

..

.gitignore

tokenizer: first shot at somethign that appears to work

2018-03-09 16:24:39 +01:00

ligature_decomposition.cpp

work-in-progress: new tokenizer

2018-03-01 16:38:19 +01:00

ligature_decomposition.h

work-in-progress: new tokenizer

2018-03-01 16:38:19 +01:00

Makefile

Use same optimization in unicode+tokenizer as in the main executable

2018-03-26 16:58:25 +02:00

tokenizer2.cpp

bugfix tokenizer posessive-s fix was refering to possible reallocated memory

2018-07-26 15:12:57 +02:00

tokenizer3.cpp

tokenizer: first shot at somethign that appears to work

2018-03-09 16:24:39 +01:00

tokenizer4.cpp

tokenizer compilation fix

2018-07-26 17:31:36 +02:00

tokenizer5.cpp

Updated superscript/supscript html handling

2018-03-06 11:38:00 +01:00

tokenizer_unittest.cpp

tokenizer: combining mark removal for Italian

2018-06-13 16:56:59 +02:00

tokenizer_util.cpp

Updated superscript/supscript html handling

2018-03-06 11:38:00 +01:00

tokenizer_util.h

Updated superscript/supscript html handling

2018-03-06 11:38:00 +01:00

tokenizer.cpp

tokenizer: keep track of whether a token is from phase 1 or phase 2

2018-03-20 14:37:56 +01:00

tokenizer.h

tokenizer: Handle slash-abbrevations as a single token

2018-04-03 16:00:55 +02:00

xml_tokenizer_unittest.cpp

Fixed soft-hypen unittest in Xml tokenizer

2018-03-19 15:32:30 +01:00