core(robots-txt): add License directive - #17160

Summary

RSL (Really Simple Licensing) 1.0 defines a License robots.txt directive, which points crawlers at a machine-readable licensing document describing how a site's content may be used. RSL explicitly extends RFC 9309, and an RSL-compliant robots.txt is required to carry one or more License directives — but Lighthouse currently flags it as an "Unknown directive", so a site that adopts the standard loses the robots-txt audit and roughly 8% of its SEO category score for doing so.

Google's own parser does not treat it as an error. Per Google's robots.txt documentation: "Rules other than allow, disallow, and user-agent are ignored by the robots.txt parser", and "Google ignores invalid lines in robots.txt files." An unrecognised directive invalidates neither the file nor the group.

This PR adds license to the robots.txt safelist and extends the existing valid-robots.txt test to cover it. Same shape and rationale as #16767, which added Content-Signal to the same list under the "not officially supported, but used in the wild" comment.

Resources:

• https://rslstandard.org/rsl — RSL 1.0 specification (the License directive)
• https://rslstandard.org/guide/robots-txt — adding RSL to robots.txt
• https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt — Google ignores unsupported rules
• Real-world example: https://rafeblandford.com/robots.txt

Related Issues/PRs

• core(robots-txt): add Content-Signal directive #16767 — added Content-Signal to the same safelist