Forks/open-source-search-engine
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,832 Commits 7 Branches 0 Tags
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Matt Wells f9d607665b forgot comma.
2017-02-24 10:01:26 -08:00
antiword-dir
Initial file population.
2013-08-02 13:12:24 -07:00
diffbot-widget
widget updates
2014-04-21 09:21:28 -07:00
doxygen
put in place doxygen stuffs
2015-05-15 14:47:47 -07:00
html
updated dmoz docs
2016-01-23 08:54:35 -07:00
openssl
we already include our own 32-bit
2013-09-15 18:25:49 -06:00
script
Increase time to mark item as stale in warc injector.
2015-11-01 19:45:29 -07:00
ucdata
Initial file population.
2013-08-02 13:12:24 -07:00
.gitignore
added Codeblocks project file
2014-10-31 11:00:18 -07:00
Abbreviations.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Abbreviations.h
replace long long with int64_t
2014-10-30 13:36:39 -06:00
Accessdb.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Accessdb.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Address.cpp
fixed langid based query stop words.
2015-03-08 15:44:23 -07:00
Address.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
addtest.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Ads.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Ads.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
AdultBit.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
AdultBit.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
animate.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
antiword
fix ulimit and antiword bugs
2014-06-18 04:06:20 -07:00
AutoBan.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
AutoBan.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
badcattable.dat
Initial file population.
2013-08-02 13:12:24 -07:00
BigFile.cpp
added FIXBUG code to fix seg fault from
2015-12-08 10:30:16 -08:00
BigFile.h
all files made are now group writable.
2015-09-21 11:19:34 -06:00
Bits.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Bits.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
blaster2.cpp
fix right
2015-10-08 13:42:42 -07:00
Blaster.cpp
bring back max mem control into master controls.
2015-08-14 12:58:54 -06:00
Blaster.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
bmptopnm
Initial file population.
2013-08-02 13:12:24 -07:00
Cachedb.cpp
fix compiler warnings
2015-09-10 13:24:59 -06:00
Cachedb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
camsort.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
catcountry.dat
Initial file population.
2013-08-02 13:12:24 -07:00
Catdb.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Catdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Categories.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Categories.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
CatRec.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
CatRec.h
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
character-sets
Initial file population.
2013-08-02 13:12:24 -07:00
check_unicode.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
Clusterdb.cpp
fix compiler warnings
2015-09-10 13:24:59 -06:00
Clusterdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Collectiondb.cpp
tell malloc to trim 100MB at a time to prevent kernel
2016-04-17 09:16:03 -07:00
Collectiondb.h
Add input validation to regexs before crawlbot collections are created.
2016-05-11 14:03:45 -06:00
Conf.cpp
fix permissions bug when creating directories,
2015-10-07 08:26:27 -06:00
Conf.h
fix the source of lots of corruption in spiderdb and titledb.
2016-03-15 15:54:12 -07:00
control.deb
package bldg updates
2014-06-16 21:50:32 -06:00
convert.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
copyright.head
package bldg updates
2014-06-16 21:50:32 -06:00
copyright.tail
package bldg updates
2014-06-16 21:50:32 -06:00
CountryCode.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
CountryCode.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
create_ucd_tables.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
DailyMerge.cpp
fix so we can generate posdb map for
2015-11-01 14:56:39 -08:00
DailyMerge.h
move CollectionRec stuff into Collectiondb files
2013-12-10 15:28:04 -08:00
DataFeed.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
DataFeed.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Datedb.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Datedb.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Dates.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Dates.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Diff.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Diff.h
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Dir.cpp
try to fix core dumps. not sure how
2015-08-22 08:52:28 -07:00
Dir.h
replace long long with int64_t
2014-10-30 13:36:39 -06:00
DiskPageCache.cpp
re-disbale page cache. wtf?
2015-09-09 22:06:00 -07:00
DiskPageCache.h
the new disk page cache. temporarily disabled.
2015-08-14 15:52:24 -06:00
dlstubs.c
Initial file population.
2013-08-02 13:12:24 -07:00
dmozparse.cpp
fix make dmozparse
2015-09-13 13:21:36 -07:00
Dns.cpp
More fixes to prevent spider traffic from hitting hosts with nospider
2015-11-13 15:03:02 -07:00
Dns.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
DnsProtocol.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
dnstest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Domains.cpp
Fixes for new tlds.
2016-05-12 16:18:53 -06:00
Domains.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
dumpcore.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
Entities.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Entities.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Errno.cpp
added 4 more diffbot errors so hopefully
2016-01-11 16:12:33 -08:00
Errno.h
Add input validation to regexs before crawlbot collections are created.
2016-05-11 14:03:45 -06:00
errnotest.cpp
errnotest.cpp fix
2015-08-24 16:22:11 -06:00
Events.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Facebook.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Facebook.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
fastIndexTable.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
fctypes.cpp
Merge branch 'ia' into ia-zak
2015-10-10 14:07:15 -06:00
fctypes.h
Merge branch 'ia' into testing
2015-10-12 10:40:16 -06:00
File.cpp
use ./cleanexit file to ensure gb doesn't restart
2016-03-16 14:57:19 -07:00
File.h
use ./cleanexit file to ensure gb doesn't restart
2016-03-16 14:57:19 -07:00
filterquerylogs.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Flags.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Flags.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
gb-1.0.spec
make it so we don't need --nodeps with
2014-05-25 22:08:46 -04:00
gb-include.h
replace memcpy_ass with bcopy
2015-01-14 14:12:55 -08:00
gb.deb.rules
if netpbm pkg already installed use it.
2014-07-06 09:54:28 -07:00
gb.pem
so we have spider https sites add
2013-10-13 00:15:39 -07:00
gbfilter.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
gbtitletest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
geneaology.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
generateSuperMergeCode.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
geo_ip_table.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
geo_ip_table.h
Initial file population.
2013-08-02 13:12:24 -07:00
GeoIP_internal.h
Initial file population.
2013-08-02 13:12:24 -07:00
GeoIP.c
Initial file population.
2013-08-02 13:12:24 -07:00
GeoIP.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
GeoIPCity.c
Initial file population.
2013-08-02 13:12:24 -07:00
GeoIPCity.h
Initial file population.
2013-08-02 13:12:24 -07:00
getsample.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
giftopnm
Initial file population.
2013-08-02 13:12:24 -07:00
gigablast.cbp
added Codeblocks project file
2014-10-31 11:00:18 -07:00
gigablast.layout
added Codeblocks project file
2014-10-31 11:00:18 -07:00
hash.cpp
fix more possible unicode errors
2015-07-19 12:05:09 -06:00
hash.h
fix more possible unicode errors
2015-07-19 12:05:09 -06:00
HashTable.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
HashTable.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
HashTableT.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
HashTableT.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
HashTableX.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
HashTableX.h
quite a few bug fixes.
2015-07-02 17:42:05 -06:00
hashtest2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
hashtest3.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
hashtest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Highlight.cpp
fix pesky memory leak finally
2015-07-13 17:47:34 -06:00
Highlight.h
allow up to 3000 query terms. really we can allow
2015-07-10 19:02:30 -06:00
Hostdb.cpp
fix getLeastLoadedInShard() to only return
2015-11-16 09:53:40 -07:00
Hostdb.h
Fix host selection for downloading when nospider directives are present.
2015-11-29 21:36:19 -07:00
hosts.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
HttpMime.cpp
fix gap.com redirects that require us
2016-02-09 13:38:59 -08:00
HttpMime.h
fix gap.com redirects that require us
2016-02-09 13:38:59 -08:00
HttpRequest.cpp
added httprequest debug line
2016-03-21 14:46:10 -07:00
HttpRequest.h
added support for supplying basic proxy authorization
2015-02-02 13:23:38 -08:00
HttpServer.cpp
fix gap.com redirects that require us
2016-02-09 13:38:59 -08:00
HttpServer.h
use http/1.0 since we dont support chunked transfer encoding
2016-02-09 12:04:05 -07:00
iana_charset.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
iana_charset.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
iconv.h
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Images.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
Images.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Indexdb.cpp
fix compiler warnings
2015-09-10 13:24:59 -06:00
Indexdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
IndexList.cpp
cleanup all warning when not using -m32
2014-11-12 14:11:27 -08:00
IndexList.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
IndexReadInfo.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
IndexReadInfo.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
IndexTable2.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
IndexTable2.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
IndexTable.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
IndexTable.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
init.gb.conf
minor make install changes
2014-05-22 18:46:38 -07:00
injectme3
added injectme3 file and documentation into compare.html
2013-08-17 11:02:26 -06:00
injectmedemo
fix sections.cpp to not set root title section
2014-12-11 19:54:33 -08:00
injector.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
iostream.h
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
ip.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
ip.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
ipconfig.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Iso8859.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
Iso8859.h
Initial file population.
2013-08-02 13:12:24 -07:00
jointest.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
jpegtopnm
Initial file population.
2013-08-02 13:12:24 -07:00
Json.cpp
Add gbcapturedate to individual doc's metadata when injecting warcs.
2015-10-04 01:53:54 -06:00
Json.h
Add gbcapturedate to individual doc's metadata when injecting warcs.
2015-10-04 01:53:54 -06:00
keepalive.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
Lang.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Lang.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
LangList.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
LangList.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Language.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
Language.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
LanguageIdentifier.cpp
Add gbcapturedate to individual doc's metadata when injecting warcs.
2015-10-04 01:53:54 -06:00
LanguageIdentifier.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
LanguagePages.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
LanguagePages.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
libc.a
Initial file population.
2013-08-02 13:12:24 -07:00
libcrypto.a
turn off hearbeats when compiling openssl libs
2014-04-22 16:39:40 -07:00
libgcc.a
Initial file population.
2013-08-02 13:12:24 -07:00
libiconv64.a
added 64 bit libiconv64.a
2014-11-14 17:34:11 -08:00
libiconv.a
Initial file population.
2013-08-02 13:12:24 -07:00
libiconv.la
Initial file population.
2013-08-02 13:12:24 -07:00
libjpeg.so.62
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
libm.a
Initial file population.
2013-08-02 13:12:24 -07:00
libnetpbm.so.10
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
libpng12.so.0
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
libpthread.a
Initial file population.
2013-08-02 13:12:24 -07:00
libssl.a
turn off hearbeats when compiling openssl libs
2014-04-22 16:39:40 -07:00
libstdc++.a
Initial file population.
2013-08-02 13:12:24 -07:00
libtiff.so.4
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
libz64.a
add libz64.a
2014-11-17 14:53:59 -08:00
libz.a
Initial file population.
2013-08-02 13:12:24 -07:00
libz.so.1
thumbnail generation support back in.
2014-04-24 10:13:45 -07:00
LICENSE
license fix
2014-06-16 13:52:51 -07:00
Linkdb.cpp
fix to allow us to gather ip-only url outlinks again
2016-03-14 10:56:33 -06:00
Linkdb.h
Revert "hash the normalized outlinks in the diffbot reply"
2015-12-02 13:04:56 -07:00
LinkedList.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
linkspam.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
linkspam.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Log.cpp
rotate log file at 1GB.
2016-09-14 09:37:23 -07:00
Log.h
make new logfile when current logfile hits 1GB.
2015-01-05 11:29:49 -08:00
Loop.cpp
added some more quickpolls.
2015-12-04 09:02:03 -08:00
Loop.h
Fix load balance of msg22s to use the udp slots in pinginfo.
2015-11-03 11:51:19 -07:00
looptest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
main.cpp
import final hung crawl fix into old gb.
2016-11-07 09:11:27 -08:00
Make.depend
force gb to recompile version every time
2014-09-19 12:23:40 -07:00
Makefile
makefile optimizations
2016-03-14 16:34:24 -07:00
malloc.c
Initial file population.
2013-08-02 13:12:24 -07:00
matches2.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
matches2.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Matches.cpp
more core dump fixes concerning -1 being returned
2016-04-17 18:50:23 -07:00
Matches.h
Fix anomalous link text detector to take into consideration the total
2015-11-20 10:42:46 -07:00
Mem.cpp
Fix: possible double free
2016-02-05 16:11:53 +03:00
Mem.h
fixes for umsg00 electric fence.
2015-08-24 11:35:33 -06:00
membustest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MemPool.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MemPool.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MemPoolTree.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
MemPoolTree.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
memtest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
mergetest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MetaContainer.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MetaContainer.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Mime.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Mime.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
mixfile.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
mmseg.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
monitor.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Monitordb.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Monitordb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Msg0.cpp
prevent core when injecting when not in sync with host #0
2015-04-28 15:29:26 -07:00
Msg0.h
try to handle those quick tagdb lookups first.
2015-01-29 20:55:02 -07:00
Msg1.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg1.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Msg1f.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
Msg1f.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg2.cpp
fix pesky memory leak finally
2015-07-13 17:47:34 -06:00
Msg2.h
allow up to 3000 query terms. really we can allow
2015-07-10 19:02:30 -06:00
Msg2a.cpp
working with -m32 for basic testing.
2014-11-12 11:38:37 -08:00
Msg2a.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg2b.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg2b.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg3.cpp
added some more quickpolls.
2015-12-04 09:02:03 -08:00
Msg3.h
added cache validation logic
2015-09-10 13:56:38 -06:00
Msg3a.cpp
do not report edocunchanged for bulk jobs ever.
2016-01-30 11:14:12 -08:00
Msg3a.h
allow up to 3000 query terms. really we can allow
2015-07-10 19:02:30 -06:00
Msg3e.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg3e.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg4.cpp
thanks for the bug fix, ivan!
2016-02-09 10:38:46 -07:00
Msg4.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg5.cpp
do not hit file cache when merging files on disk.
2015-09-11 11:09:15 -07:00
Msg5.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg6b.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg6b.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg8b.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg8b.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg9b.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg9b.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg13.cpp
increase regular page download timeout from 30
2016-05-18 10:05:02 -07:00
Msg13.h
in the sockets table page,
2015-08-25 09:34:45 -07:00
Msg17.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg17.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg20.cpp
Merge branch 'testing' into diffbot-testing
2015-12-09 23:11:37 -07:00
Msg20.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Msg22.cpp
if old title rec was corrupted we would get a random docid
2016-03-15 23:26:57 -07:00
Msg22.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg24.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg28.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg28.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg30.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msg30.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg35.cpp
working with -m32 for basic testing.
2014-11-12 11:38:37 -08:00
Msg35.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg36.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Msg36.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg37.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg37.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg39.cpp
fix cores on gi #0
2015-09-25 08:09:05 -07:00
Msg39.h
fix some mem leaks from allowing really big queries.
2015-07-13 23:17:53 -06:00
Msg40.cpp
fix core from a federated query and null msg20
2016-02-18 10:53:20 -08:00
Msg40.h
Fix double call of gotSummary when computing facets in msg40. Fixes
2015-10-20 17:21:37 -06:00
Msg40Cache.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg40Cache.h
Initial file population.
2013-08-02 13:12:24 -07:00
Msg42.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Msg42.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msg51.cpp
fix core
2014-11-27 14:33:04 -07:00
Msg51.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msgaa.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Msgaa.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
MsgC.cpp
fix dns lookup bug that was causing us
2016-05-17 11:57:21 -07:00
MsgC.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msge0.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Msge0.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Msge1.cpp
loop.cpp cleanups.
2015-02-13 12:07:10 -08:00
Msge1.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Multicast.cpp
Fix load balance of msg22s to use the udp slots in pinginfo.
2015-11-03 11:51:19 -07:00
Multicast.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
mysynonyms.txt
mysyn fixes
2015-04-22 08:34:29 -06:00
numwords.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
OldDiskPageCache.cpp
bring back max mem control into master controls.
2015-08-14 12:58:54 -06:00
OldDiskPageCache.h
undo #define thing
2015-08-14 13:08:11 -06:00
PageAddColl.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
PageAddUrl.cpp
do not consider .gz a 'media' url extension any more
2015-05-02 14:52:17 -07:00
PageBasic.cpp
fix core from adding a lot of sites
2015-03-07 20:57:17 -07:00
PageCatdb.cpp
return ENOPERM on certain pages if not
2015-01-29 09:46:48 -07:00
PageCrawlBot.cpp
import final hung crawl fix into old gb.
2016-11-07 09:11:27 -08:00
PageCrawlBot.h
more api updates
2014-07-13 09:35:44 -07:00
PageDirectory.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageEvents.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageGet.cpp
added trivial link on cached page to gb root page
2016-01-03 11:27:24 -08:00
PageHosts.cpp
change try agains recvd to try agains sent
2015-12-23 22:18:24 -07:00
PageIndexdb.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageInject.cpp
Merge branch 'ia' into testing
2015-11-09 11:14:00 -07:00
PageInject.h
show inject requests in the spider queue table now
2015-09-11 14:16:26 -06:00
PageLogView.cpp
More testing on nospider, noquery.
2015-08-31 10:47:19 -06:00
PageNetTest.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageNetTest.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PageOverview.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
PageParser.cpp
quite a few bug fixes from adding the new query
2014-12-11 18:24:28 -08:00
PageParser.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PagePerf.cpp
Fixes to injector script.
2015-08-13 23:29:20 -06:00
PageReindex.cpp
make query reindex (not query delete) distribute
2015-05-07 09:08:59 -07:00
PageReindex.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PageResults.cpp
hack on parentUrlDocId to the json object dump
2016-03-28 12:39:48 -06:00
PageResults.h
some debug statement to track down the socket snafu on host 0
2015-09-10 19:18:48 -07:00
PageRoot.cpp
fix add url on root page to set collnum properly.
2016-04-06 10:31:04 -06:00
Pages.cpp
Add input validation to regexs before crawlbot collections are created.
2016-05-11 14:03:45 -06:00
Pages.h
return ENOPERM on certain pages if not
2015-01-29 09:46:48 -07:00
PageSockets.cpp
fix bug of losing the line waiter header
2015-11-19 19:40:30 -07:00
PageSpam.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PageStats.cpp
fix empty winner tree bug.
2015-10-02 12:16:48 -07:00
PageStatsdb.cpp
Warc pipe fixes. Fix arcs not processing https. Fix nulls being left
2015-10-12 00:30:28 -06:00
PageSubmit.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageThesaurus.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PageThreads.cpp
undo some possible averse changes
2015-09-04 11:31:43 -07:00
PageTitledb.cpp
Merge branch 'diffbot-testing' into diffbot-matt
2014-11-20 16:53:07 -08:00
PageTurk.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PageTurk.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Parms.cpp
empty &seeds= fix to not reset seeds
2016-11-16 10:34:25 -08:00
Parms.h
Add input validation to regexs before crawlbot collections are created.
2016-05-11 14:03:45 -06:00
parse_iana_charsets.pl
move CollectionRec stuff into Collectiondb files
2013-12-10 15:28:04 -08:00
pdftohtml
fix rdbcache init core
2014-12-01 12:37:51 -08:00
Phrases.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Phrases.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
PingServer.cpp
fix core from sending a url alert, then customer deleting
2015-09-08 15:57:46 -07:00
PingServer.h
fix core from sending a url alert, then customer deleting
2015-09-08 15:57:46 -07:00
Placedb.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Placedb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
pngtopnm
Initial file population.
2013-08-02 13:12:24 -07:00
pnmscale
Initial file population.
2013-08-02 13:12:24 -07:00
Pops.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Pops.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
porter.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Pos.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Pos.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Posdb.cpp
fix core in posdbtable from docid of 0.
2016-02-09 22:43:09 -08:00
Posdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
postalCodes.txt
Initial file population.
2013-08-02 13:12:24 -07:00
PostQueryRerank.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
PostQueryRerank.h
get mike's super long query working
2015-07-13 14:59:44 -06:00
ppmtojpeg
Initial file population.
2013-08-02 13:12:24 -07:00
Process.cpp
do not save doledb on exit to prevent corruption
2016-10-24 14:23:13 -07:00
Process.h
more fixes for new spider updates
2015-02-11 21:54:36 -08:00
Profiler.cpp
Merge branch 'diffbot-testing' into testing
2015-11-09 11:13:42 -07:00
Profiler.h
turn off profiler automatically after 60 seconds.
2015-09-10 13:37:14 -06:00
Proxy.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Proxy.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
pstotext
Initial file population.
2013-08-02 13:12:24 -07:00
Punycode.cpp
Start to detect non-asci urls and encode them to ascii.
2015-09-12 15:47:33 -06:00
Punycode.h
Start to detect non-asci urls and encode them to ascii.
2015-09-12 15:47:33 -06:00
qa.cpp
complete merge of ia code into testing.
2015-11-09 12:46:06 -07:00
QAClient.cpp
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
QAClient.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
quarantine.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Query.cpp
Filter link text anomalies at query time.
2015-11-19 12:25:25 -07:00
Query.h
fix more cores from the dynamic query size changes.
2015-07-18 14:15:47 -06:00
Rdb.cpp
import final hung crawl fix into old gb.
2016-11-07 09:11:27 -08:00
Rdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
RdbBase.cpp
make trash subdir in case missing.
2016-05-30 19:37:03 -07:00
RdbBase.h
fix bug of dumping too many files to disk and not
2015-11-17 09:52:41 -08:00
RdbBuckets.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
RdbBuckets.h
added RdbBuckets::cleanBuckets() corresponding to
2015-03-21 22:28:34 -06:00
RdbCache.cpp
Merge branch 'diffbot-testing' into ia
2015-10-10 14:05:27 -06:00
RdbCache.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
RdbDump.cpp
fix dump core when collection deleted while dumping
2016-03-18 06:46:38 -07:00
RdbDump.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
RdbList.cpp
show corrupt collection numbers for spiderdb corrupted recs.
2016-08-15 12:55:22 -07:00
RdbList.h
fix churn bug in winnerlistcache in spider.cpp
2015-10-01 19:35:34 -07:00
RdbMap.cpp
fix so we can generate posdb map for
2015-11-01 14:56:39 -08:00
RdbMap.h
fix so we can generate posdb map for
2015-11-01 14:56:39 -08:00
RdbMem.cpp
fix dump core when collection deleted while dumping
2016-03-18 06:46:38 -07:00
RdbMem.h
after dump completes scan tree to ensure all nodes
2016-03-17 10:09:49 -07:00
RdbMerge.cpp
fix core when exiting while merging
2015-10-24 12:50:57 -07:00
RdbMerge.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
RdbScan.cpp
remove unnecessary line
2015-09-13 17:54:38 -07:00
RdbScan.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
rdbtest2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
rdbtest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
RdbTree.cpp
fix dump core when collection deleted while dumping
2016-03-18 06:46:38 -07:00
RdbTree.h
fixed bad deletenode call causing dups in
2015-02-12 16:12:23 -08:00
README.md
update README.md
2015-03-19 23:31:09 -06:00
readRec.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Rebalance.cpp
fix save prevention when coring in malloc/free.
2015-08-23 11:51:46 -07:00
Rebalance.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
reindex2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Repair.cpp
clean out rebuild trees/buckets too
2015-03-21 22:42:49 -06:00
Repair.h
Merge branch 'diffbot-testing' into diffbot-matt
2014-11-20 16:53:07 -08:00
RequestTable.cpp
cleanup all warning when not using -m32
2014-11-12 14:11:27 -08:00
RequestTable.h
cleanup all warning when not using -m32
2014-11-12 14:11:27 -08:00
rescue.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Revdb.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Revdb.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
rmbots.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
S99gb
added S99gb for loading at boot.
2014-06-23 07:32:38 -06:00
SafeBuf.cpp
Merge branch 'ia' into testing
2015-11-09 11:14:00 -07:00
SafeBuf.h
Merge branch 'ia' into testing
2015-11-09 11:14:00 -07:00
SafeList.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Sanity.h
Merge branch 'diffbot-testing' into diffbot-matt
2014-11-20 16:53:07 -08:00
Scores.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Scores.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Scraper.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Scraper.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
SearchInput.cpp
fix more cores from the dynamic query size changes.
2015-07-18 14:15:47 -06:00
SearchInput.h
added support for &nf=50 to limit to top 50 facets.
2015-01-29 10:34:22 -07:00
Sections.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Sections.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
seektest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
seo.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
SiteGetter.cpp
fix infinite loop bug from EBADRBDID
2015-07-31 08:56:26 -07:00
SiteGetter.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
sitelinks.txt
fixed missing sites in sitelinks.txt
2015-03-05 20:32:01 -08:00
sleepandlog.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
sort.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
sort.h
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Speller.cpp
fix file/dir creation permissions bugs
2015-09-21 12:44:41 -06:00
Speller.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Spider.cpp
forgot comma.
2017-02-24 10:01:26 -08:00
Spider.h
import final hung crawl fix into old gb.
2016-11-07 09:11:27 -08:00
SpiderProxy.cpp
fix some diffbot crawls.
2015-12-23 13:49:21 -08:00
SpiderProxy.h
spider proxy fixes for negative ports
2015-10-21 15:32:58 -07:00
Stats.cpp
allow up to 3000 query terms. really we can allow
2015-07-10 19:02:30 -06:00
Stats.h
allow up to 3000 query terms. really we can allow
2015-07-10 19:02:30 -06:00
Statsdb.cpp
Fix repeating label.
2015-09-24 01:33:51 -06:00
Statsdb.h
fix signed/unsigned bug
2014-12-10 11:04:37 -08:00
StopWords.cpp
fixed langid based query stop words.
2015-03-08 15:44:23 -07:00
StopWords.h
fixed langid based query stop words.
2015-03-08 15:44:23 -07:00
streambuf.h
good checkpoint. quite a few fixes.
2014-11-17 18:13:36 -08:00
Strings.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Strings.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Summary.cpp
fix add url on root page to set collnum properly.
2016-04-06 10:31:04 -06:00
Summary.h
nomenclature changes
2015-07-13 18:42:13 -06:00
superMergeTest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
supported_charsets.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
supported_charsets.txt
Initial file population.
2013-08-02 13:12:24 -07:00
Syncdb.cpp
try to fix core dumps. not sure how
2015-08-22 08:52:28 -07:00
Syncdb.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Synonyms.cpp
a lot of bug fixes thanks to isj.
2016-03-29 04:08:17 -06:00
Synonyms.h
nomenclature change
2014-12-04 11:02:54 -07:00
Tagdb.cpp
try to fix a couple more core dumps.
2016-02-19 08:54:48 -08:00
Tagdb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
TcpServer.cpp
added support for TLS SNI (Server name identification)
2015-12-23 13:30:49 -07:00
TcpServer.h
prevent double ./gb start calls from messing
2015-08-31 11:13:33 -06:00
TcpSocket.h
added proper write callback registration into
2015-02-16 14:48:39 -07:00
test2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
test_convert.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
test_hash.cpp
replace long long with int64_t
2014-10-30 13:36:39 -06:00
test_norm.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
test_parser2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
test_parser.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
test_unicode.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Test.cpp
now we add the spider status docs as json documents.
2015-03-19 16:17:36 -06:00
Test.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
testfloats.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Tfndb.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Tfndb.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Thesaurus.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Thesaurus.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Threads.cpp
join with threads when exiting -- to no avail
2015-12-17 10:15:39 -08:00
Threads.h
try to fix exiting w/ pthreads some more (part 2)
2015-12-17 08:38:12 -07:00
threadtest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
thunder.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
tifftopnm
Initial file population.
2013-08-02 13:12:24 -07:00
Timedb.cpp
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Timedb.h
text replacements for bad int32_t substitutions
2014-11-17 18:24:38 -08:00
Timer.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Title.cpp
tell malloc to trim 100MB at a time to prevent kernel
2016-04-17 09:16:03 -07:00
Title.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Titledb.cpp
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
Titledb.h
do not store cblock, etc. tags into tagdb to save
2015-09-10 12:46:00 -06:00
TopTree.cpp
fixed bad deletenode call causing dups in
2015-02-12 16:12:23 -08:00
TopTree.h
fix cores in top tree with last commit. this one
2014-12-08 09:29:21 -08:00
treetest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
TuringTest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
TuringTest.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Turkdb.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
types.h
fix keysize==8 bug in keycmp
2016-03-28 09:17:01 -06:00
UCNormalizer.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
UCNormalizer.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
UCPropTable.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
UCPropTable.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
UCWordIterator.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
UCWordIterator.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
UdpProtocol.h
no limit to tagdb lookups even if niceness 1
2015-01-29 21:38:10 -07:00
UdpServer.cpp
Merge branch 'ia-zak' of https://github.com/gigablast/open-source-search-engine into ia-zak
2015-09-10 21:32:36 -06:00
UdpServer.h
Add logic to limit number of msg7s to 100 per hosts, then we drop the
2015-09-03 22:17:16 -06:00
UdpSlot.cpp
change try agains recvd to try agains sent
2015-12-23 22:18:24 -07:00
UdpSlot.h
allow more docids to be downloaded/served in search results.
2016-03-22 15:24:33 -07:00
udptest.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Unicode.cpp
a lot of bug fixes thanks to isj.
2016-03-29 04:08:17 -06:00
Unicode.h
Optimize UTF-8 handling in getUtf8CharSize() by using logic instead of table lookup (memory fetch) for bytes<128
2015-09-07 13:32:36 +02:00
UnicodeProperties.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
UnicodeProperties.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
unifiedDict.txt
Initial file population.
2013-08-02 13:12:24 -07:00
uniq2.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Url.cpp
Fixes for new tlds.
2016-05-12 16:18:53 -06:00
Url.h
Show utf8 url in page results.
2015-09-21 16:44:40 -06:00
urlinfo.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Users.cpp
fix save prevention when coring in malloc/free.
2015-08-23 11:51:46 -07:00
Users.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
ValidPointer.cpp
Initial file population.
2013-08-02 13:12:24 -07:00
ValidPointer.h
Initial file population.
2013-08-02 13:12:24 -07:00
Vector.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Vector.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Version.cpp
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Version.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Weights.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Weights.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Wiki.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Wiki.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
wikititles.txt.part1
Initial file population.
2013-08-02 13:12:24 -07:00
wikititles.txt.part2
Initial file population.
2013-08-02 13:12:24 -07:00
wiktionary-buf.txt
when user searches for a word without the
2014-06-01 09:37:00 -07:00
wiktionary-lang.txt
when user searches for a word without the
2014-06-01 09:37:00 -07:00
wiktionary-syns.dat
when user searches for a word without the
2014-06-01 09:37:00 -07:00
Wiktionary.cpp
use gbmemcpy not memcpy so we can get profiler working again
2015-01-13 12:25:42 -07:00
Wiktionary.h
now it compiles with -m32
2014-11-10 14:45:11 -08:00
Words.cpp
quite a few bug fixes.
2015-07-02 17:42:05 -06:00
Words.h
query stop words now based on selected langid.
2015-03-08 15:16:24 -07:00
Xml.cpp
fix </script> tag detection stuff again.
2015-08-31 14:06:44 -06:00
Xml.h
fix links parser so it harvests outlinks from rss feeds'
2015-03-12 17:35:47 -07:00
XmlDoc.cpp
import final hung crawl fix into old gb.
2016-11-07 09:11:27 -08:00
XmlDoc.h
fix stack related bug.
2016-09-14 10:00:18 -07:00
XmlNode.cpp
sitemap.xml support for harvesting loc urls.
2015-03-17 14:26:16 -06:00
XmlNode.h
sitemap.xml support for harvesting loc urls.
2015-03-17 14:26:16 -06:00
zconf.h
updated to a new libz64.a. updated zconf.h and
2014-11-17 14:53:15 -08:00
zlib.h
updated to a new libz64.a. updated zconf.h and
2014-11-17 14:53:15 -08:00

README.md

open-source-search-engine

An open source web and enterprise search engine and spider/crawler. As can be seen on http://www.gigablast.com/ .

RUNNING GIGABLAST

See html/faq.html for all administrative documentation including the quick start instructions.

Alternatively, visit http://www.gigablast.com/faq.html

CODE ARCHITECTURE

See html/developer.html for all code documentation.

Alternatively, visit http://www.gigablast.com/developer.html

CONTACT

Contact me for feature requests or help in general. I will work for free for good use cases. mattdwells@hotmail.com.

Description
An open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. See the README.md file at the very bottom of this page for instructions.
search
Readme 226 MiB
Languages
C++ 96.7%
HTML 1.5%
C 1.2%
Python 0.5%