16abbd78e7
Add scripts/import_wpt_reftests.py to sparse-clone the upstream WPT repo and bulk-import qualifying CSS reftests (no JS, no external resources) as known_fail entries. 23 tests already pass and are promoted. The import script is idempotent and exposed via `just import-wpt`. CI now prints the WPT summary (pass=36 known_fail=2877 skip=1) on every run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
159 lines
4.2 KiB
Markdown
159 lines
4.2 KiB
Markdown
# External Conformance: WPT Harness (HTML/CSS/Layout)
|
|
|
|
This project includes a curated Web Platform Tests (WPT)-style subset under `tests/external/wpt`.
|
|
|
|
## Test source
|
|
|
|
Canonical upstream source:
|
|
|
|
- WPT repository: https://github.com/web-platform-tests/wpt
|
|
- WPT project site: https://web-platform-tests.org/
|
|
|
|
The files in `tests/external/wpt/fixtures` are local, pinned copies adapted from that upstream corpus for this engine's current no-JS scope.
|
|
When adding tests, record the originating WPT test path (or URL) in the PR description so updates remain traceable.
|
|
|
|
## Scope
|
|
|
|
Current scope intentionally excludes JavaScript-dependent tests. The harness focuses on deterministic HTML/CSS/layout behavior using the existing structural oracles:
|
|
|
|
- layout tree dump
|
|
- display list dump
|
|
|
|
## Files
|
|
|
|
- Manifest: `tests/external/wpt/wpt_manifest.toml`
|
|
- Fixtures: `tests/external/wpt/fixtures`
|
|
- Expected outputs: `tests/external/wpt/expected`
|
|
- Harness test target: `tests/wpt_harness.rs`
|
|
|
|
## Status policy
|
|
|
|
Each case is marked with one of:
|
|
|
|
- `pass`: must pass in CI
|
|
- `known_fail`: expected to fail for documented reasons
|
|
- `skip`: not executed in current phase (for example, JS-dependent)
|
|
|
|
`known_fail` and `skip` entries require non-empty reasons.
|
|
|
|
## Running
|
|
|
|
Run the suite:
|
|
|
|
```bash
|
|
cargo test -p rust_browser --test wpt_harness
|
|
```
|
|
|
|
Failure artifacts are written to `target/wpt_artifacts`.
|
|
|
|
Regenerate expected outputs for `pass` + `single` cases:
|
|
|
|
```bash
|
|
just regen-wpt
|
|
```
|
|
|
|
## Bulk-importing upstream WPT reftests
|
|
|
|
The `import-wpt` command sparse-clones the upstream WPT repo and imports qualifying CSS reftests as `known_fail`:
|
|
|
|
```bash
|
|
# See what's available without importing
|
|
just import-wpt --scan-only
|
|
|
|
# Import all qualifying reftests (idempotent)
|
|
just import-wpt
|
|
|
|
# Import from specific CSS modules only
|
|
just import-wpt --modules css-box,css-text
|
|
|
|
# Limit the number of tests imported
|
|
just import-wpt --max-tests 50
|
|
|
|
# Preview what would be imported
|
|
just import-wpt --dry-run
|
|
```
|
|
|
|
A test qualifies for import if it has `<link rel="match">`, no `<script>`, no external stylesheets, no `<img>`/`<video>`/`<canvas>`/`<iframe>`, and no `@import`/`url()` in CSS (except `data:`). The reference file must also pass the same checks.
|
|
|
|
The upstream repo is cloned to `tests/external/wpt/upstream/` (gitignored).
|
|
|
|
## Investigating and promoting known_fail tests
|
|
|
|
### 1. Pick a test
|
|
|
|
Browse the manifest for `known_fail` entries. The `reason` field records the upstream WPT path and detected CSS features:
|
|
|
|
```toml
|
|
[[case]]
|
|
id = "wpt-css-css-box-margin-trim-block-container-block-001"
|
|
status = "known_fail"
|
|
reason = "upstream: css/css-box/margin-trim/block-container-block-001.html"
|
|
flags = ["css-box"]
|
|
```
|
|
|
|
### 2. Read the test and reference fixtures
|
|
|
|
Open them side by side to understand what the test expects:
|
|
|
|
```
|
|
tests/external/wpt/fixtures/<id>-test.html
|
|
tests/external/wpt/fixtures/<id>-ref.html
|
|
```
|
|
|
|
The test passes when both files produce identical layout tree and display list output.
|
|
|
|
### 3. Check failure artifacts
|
|
|
|
Run the suite and inspect the diff:
|
|
|
|
```bash
|
|
cargo test -p rust_browser --test wpt_harness wpt_suite -- --nocapture
|
|
```
|
|
|
|
The harness writes artifacts to `target/wpt_artifacts/`:
|
|
|
|
- `<id>.actual.layout.txt` — layout tree for the test file
|
|
- `<id>.actual.dl.txt` — display list for the test file
|
|
|
|
Compare these against the reference file's output to identify the rendering gap.
|
|
|
|
### 4. Fix the engine
|
|
|
|
Fix the relevant crate (`css`, `layout`, `style`, etc.) and re-run:
|
|
|
|
```bash
|
|
cargo test -p rust_browser --test wpt_harness wpt_suite
|
|
```
|
|
|
|
If the test now passes, CI will fail with:
|
|
|
|
> case '...' is marked known_fail but now passes; promote it to pass in manifest
|
|
|
|
### 5. Promote in the manifest
|
|
|
|
In `wpt_manifest.toml`, change the entry:
|
|
|
|
```toml
|
|
# Before
|
|
status = "known_fail"
|
|
reason = "upstream: css/css-box/margin-trim/block-container-block-001.html"
|
|
|
|
# After
|
|
status = "pass"
|
|
# (remove the reason line)
|
|
```
|
|
|
|
### 6. Verify
|
|
|
|
```bash
|
|
just ci
|
|
```
|
|
|
|
## Adding a case manually
|
|
|
|
1. Add fixture HTML under `tests/external/wpt/fixtures`.
|
|
2. Add a `[[case]]` entry in `tests/external/wpt/wpt_manifest.toml`.
|
|
3. If `mode = "single"`, regenerate expected outputs.
|
|
4. If `mode = "reftest"`, provide `reference` fixture and ensure deterministic equivalence.
|
|
5. Use `known_fail` only with a concrete reason.
|