28 Commits

Author SHA1 Message Date
0bb43ce451 Remove now empty LanguageIdentifier.h and LanguageIdentifier.cpp 2017-06-26 13:05:05 +02:00
3fcaf21a34 Move guessCountryTLD to CountryCode.h from LanguageIdentifier.h 2017-06-23 15:13:50 +02:00
d2643169de Add encoding statistics so we can keep track of what we detect as encoding, and what language we detect it as 2017-05-05 12:44:03 +02:00
ee2a212537 Remove unused include file 2016-09-23 22:29:38 +02:00
8a261ef6fb fixed possible out-of-bounds reads in CountryCode 2016-09-23 10:23:28 +02:00
0f8f528c52 Remove niceness from HashTableX 2016-09-02 10:43:24 +02:00
63762342bb constness on static 2016-08-21 17:26:09 +02:00
8e5da1cd13 constness 2016-08-19 11:40:55 +02:00
3c2773a928 Removed gbstrlen()
gbstrlen() just checked for NULL and called gbshutdownAbort(). Dereferncing NULL on mordern platforms cases a SIGSEGV which is cought by our signal handler and .... gbshutdownAbort() is called. So gbstrlen() was superfluous and complicated static analysis.
2016-07-28 17:04:35 +02:00
f3f5eefcb6 First batch of changes streamlining emergency shutdown code 2016-06-20 12:30:26 +02:00
b8dcb7f994 Fix conversion from string literal to 'char *' for CountryCode 2016-05-31 11:26:00 +02:00
be9840d1c0 Replaced Threads.* with a jobscheduler
Threads were being created and destroyed which can be expensive. The
thread-per-job model has been changed to a job scheduler that manages the job
queues and threads in pools. The submission of a job now specifies start/finish
routines, state, and as precisely what kind of job it is. The job scheduler then
takes care of the rest. it is hidden how many queues and pools there are.
2016-04-29 14:27:27 +02:00
96ab75be1d Moved MAX_NICENESS from Threads.h to a separate max_niceness.h file 2016-04-28 14:28:14 +02:00
642552d040 Cleanup in Mem.h
Removed unnecessary #includes. Turned out that includign Mem.h was mostly
equivalent to #include *.* because Mem.h would pull in Xml.h, File.h Hostdb.h
etc. A small step toward making header files and modules self-contained.
2016-03-14 00:37:18 +01:00
fd3506cab9 Fixed inclusion of unused header files 2016-03-12 17:59:38 +01:00
d90738ab4d Make include order more consistent
With glibc-2.18 it is possible to end up with int32_t typedef'ed as "int" in one
module and "unsigned int" in another, making the use of it nominally
inconsistent, which Flexelint complains about. But using flexelint's -fdc flag
reveals that the C++ template numeric_limits<> relies on "char", "signed char"
and "unsigned char" being three distinct types. glibc-2.18 contains at least
three different typedefs for the integer types, so it is a mess. Best option is
to make include order consistent in our source code.
2016-03-07 22:44:50 +01:00
a4d8f6b70c Remove unused catcountry files 2016-03-05 10:52:34 +01:00
f8e9dac6de Removed AutoBan, dmoz & related code (Categories and related Msg/Db), scraping code. 2015-12-11 21:56:05 +01:00
6184e376a4 Cleanup related to languages/Compiler warnings 2015-11-23 14:45:18 +01:00
87285ba3cd use gbmemcpy not memcpy so we can get profiler working again
since memcpy can't be interrupted and backtrace() called.
2015-01-13 12:25:42 -07:00
96b8197ad3 now it compiles with -m32 2014-11-10 14:45:11 -08:00
b13f3d24d7 replaced unsigned long long with uint64_t 2014-10-30 13:30:39 -06:00
060e887f08 misc/various bug fixes.
fix canonical redir url bug with iframes.
2014-08-28 18:07:22 -07:00
628fe2336f make code compile cleaner. 2014-06-07 14:11:12 -07:00
bc35b7d0ec fix pagecrawlbot.cpp to support &c=token-name.
cleanup mem at process exit better.
2014-01-22 23:40:38 -08:00
660f43cec7 fix bugs of pthreads junk not being async safe.
we were calling fprintf from a signal handler
(interrupt) while fprintf was currently in progress
and the pthread junk did not like that.
2013-12-15 11:41:41 -07:00
44ae7c4de6 mem labelling fixes.
fixed bad alloc when generating gigabits.
2013-12-09 14:05:02 -07:00
f6e560c1f4 Initial file population. 2013-08-02 13:12:24 -07:00