f01548ad8a
Some checks failed
CI / build (push) Has been cancelled
* add native-root HTTPS client for robots policy so HTTPS robots.txt fetches work under MITM * introduce `robots.missing_policy`, document all robots flags, and handle missing robots responses * add HTTPS MITM integration tests (self-signed CA, CONNECT tunnel verification) * document MITM CA generation/install steps and wire server listener to support upgrades