Ai Lin Chia
|
124811501b
|
Use angle brace instead of quote to include gtest.h
|
2016-09-23 11:05:22 +02:00 |
|
Ai Lin Chia
|
17d5004f0e
|
Add todo notes
|
2016-06-09 14:52:18 +02:00 |
|
Ai Lin Chia
|
3bf9f64109
|
Fix valgrind invalid read error on Robots parsing
|
2016-05-24 12:06:43 +02:00 |
|
Ai Lin Chia
|
2cb2f4e877
|
Normalize url. Don't encode character which are not suppose to be encoded.
|
2016-05-10 16:15:31 +02:00 |
|
Ai Lin Chia
|
3e12649824
|
Whitespace changes in comments
|
2016-05-10 16:15:31 +02:00 |
|
Ai Lin Chia
|
5f06305be0
|
Remove hack for pragma pack mess
|
2016-04-05 23:21:51 +02:00 |
|
Ai Lin Chia
|
13a8d2017f
|
Extend unit test scope logging
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
3aa1dec175
|
Quick hack to prevent the pragma pack mess to crash our code/unit test.
Add -fno-stack-protector for now until pragma pack stuff is fixed
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
0ae20dbfee
|
Add more unit test for RobotRule & unit test for real robots.txt found in the wild
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
dceebd6053
|
Add debug log. Remove logging from unit tests
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
1b8f66902b
|
Make Robots::print public
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
a8b5aa6d24
|
Move all unit test to use new code
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
41d1418518
|
Add code for wildcard searching. Additional unit test for wildcard searching
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
82a76baab0
|
Refactored Robots. Move RobotRule into separate file. Additional unit test.
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
2336613db4
|
Fix multi-line consecutive user-agent
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
86a708cae5
|
Add code to handle crawl-delay & unit test for it
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
3e0ce1fc0b
|
Add code to extract lines, field & value from robots.txt
|
2016-04-05 23:21:50 +02:00 |
|
Ai Lin Chia
|
91df665033
|
Add more robots.txt unit test
|
2016-04-05 23:21:49 +02:00 |
|
Ai Lin Chia
|
ed565cc095
|
Add robots.txt precedence test with robots.txt in reverse order (disallow -> allow)
|
2016-03-21 02:05:24 +01:00 |
|
Ai Lin Chia
|
7b5a85e109
|
Add placeholder for more robots.txt tests
|
2016-03-21 01:07:05 +01:00 |
|
Ai Lin Chia
|
7caaa20f47
|
Add more unit test for robots.txt
|
2016-03-21 00:24:21 +01:00 |
|
Ai Lin Chia
|
70a020c4a8
|
Add more test to cover robots.txt precedence
|
2016-03-20 20:34:39 +01:00 |
|
Ai Lin Chia
|
ead8322860
|
Add more unit test for robots.txt
|
2016-03-20 18:04:48 +01:00 |
|
Ai Lin Chia
|
d76518d556
|
Add unit test for robots.txt. Disable failing unit test for now
|
2016-03-18 17:46:25 +01:00 |
|