Skip to content

fix: stop rejecting skills whose SKILL.md uses thematic breaks - #3297

Merged
Patrick-Erichsen merged 2 commits into
openclaw:mainfrom
Yigtwxx:fix/skill-quality-frontmatter-strip
Jul 30, 2026
Merged

fix: stop rejecting skills whose SKILL.md uses thematic breaks#3297
Patrick-Erichsen merged 2 commits into
openclaw:mainfrom
Yigtwxx:fix/skill-quality-frontmatter-strip

Conversation

@Yigtwxx

@Yigtwxx Yigtwxx commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes an issue where publishing a skill fails with

Skill content is too thin or templated. Add meaningful, specific documentation.

even though the SKILL.md is fully documented. The trigger is a SKILL.md that has
no YAML frontmatter and uses --- as an ordinary Markdown thematic break. The
publish is rejected outright, so the skill never reaches the catalog.

Newly created accounts are hit hardest: the reject floors are highest for the
low trust tier, which covers every account under 30 days old or with fewer
than 10 published skills.

Why This Change Was Made

The quality gate removes frontmatter before measuring the body:

raw.replace(/^---\s*\n[\s\S]*?\n---\s*\n?/m, "")

The m flag makes ^ match at every line start rather than only at the start of
the document. Frontmatter is optional on publish — publishSkillVersion takes the
display name from its own arguments, and parseFrontmatter returns an empty
record for a document that has none — so a SKILL.md may legitimately open with a
heading and use --- as a horizontal rule. When it does, the pattern latches onto
the first rule and deletes everything up to the second one.

Dropping the flag anchors the pattern to the start of the document, which is what
the rest of the repository already does. The canonical parser
parseFrontmatter in convex/lib/skills/index.ts guards with
normalized.startsWith("---"), and the three other frontmatter patterns
(convex/skillsShMirror.ts, packages/schema/src/clawPackage.ts,
packages/clawhub/src/schema/clawPackage.ts) carry no m flag. skillQuality
was the only one reading the same document differently from the publish path that
calls it.

The truncated body also fed toStructuralFingerprint, so template-spam
similarity was being compared over a fragment rather than the real document. That
is corrected by the same change.

Non-goals, kept out to hold the change to one concern:

  • Tuning the reject or quarantine thresholds. They are unchanged.
  • Sharing a single frontmatter helper across the four call sites.

User Impact

A skill whose SKILL.md uses --- as a thematic break now publishes normally
instead of being rejected as thin content. Spam detection also compares whole
documents rather than truncated fragments, so the fingerprint is no longer
skewed toward whatever a skill happens to place above its first horizontal rule.

Evidence

Base commit: 79ef4af1. The fix is not on mainstripFrontmatter there still
carries the m flag.

Signals and the gate decision read from computeQualitySignals and
evaluateQuality, for a 109-word SKILL.md with two headings, four bullets and two
--- thematic breaks, evaluated at the low trust tier with no similar recent
publishes:

before after
bodyChars 35 618
bodyWords 6 109
headingCount 1 3
bulletCount 0 4
score 30 100
decision reject pass
reason Skill content is too thin or templated. Add meaningful, specific documentation. Quality checks passed.

94% of the body was being discarded before measurement.

This module had no unit test. convex/lib/skillQuality.test.ts adds three cases;
two of them fail on the parent commit and pass with the fix:

$ VITE_CONVEX_URL=https://example.invalid bunx vitest run \
    convex/lib/skillQuality.test.ts        # parent commit 79ef4af1

 × computeQualitySignals > keeps the body of a frontmatter-less SKILL.md that uses thematic breaks
AssertionError: expected 1 to be 3 // Object.is equality

 × evaluateQuality > does not reject a documented frontmatter-less skill from a new account
AssertionError: expected 'reject' to be 'pass' // Object.is equality

 Test Files  1 failed (1)
      Tests  2 failed | 1 passed (3)
$ VITE_CONVEX_URL=https://example.invalid bunx vitest run \
    convex/lib/skillQuality.test.ts        # with the fix

 Test Files  1 passed (1)
      Tests  3 passed (3)

The third case (still strips real frontmatter) passes on both sides: it compares
a document carrying frontmatter against the same body without it and asserts the
measured bodyWords and bodyChars match, so the intended stripping is still
happening.

Consumers of the module, run together:

$ bunx vitest run convex/lib/skillPublish.test.ts convex/lib/skillQuality.test.ts \
    convex/lib/skills.test.ts
 Test Files  3 passed (3)
      Tests  61 passed (61)

Gates run locally on Windows:

  • bunx tsc --noEmit — clean
  • bun run lint — clean
  • bun run deadcode:ci — clean
  • bun run format:check — the only two files it reports are CLAUDE.md and
    .agents/skills/autoreview/CLAUDE.md; both report identically on an unmodified
    checkout of 79ef4af1, so they are pre-existing and untouched here.
  • bun run ci:unit — 5542 passed. The 24 failures are pre-existing on this
    platform: an unmodified checkout of 79ef4af1 fails the same 14 files with the
    same 24 tests (5539 passed there; the difference is the three tests added by
    this PR). They are the scripts/ worker, CLI and security-dataset suites that
    shell out to bun, plus convex/lib/githubAccount.test.ts and
    src/routes/-management.test.tsx. Linux CI is the authoritative signal.

The Vercel preview check will need OpenClaw Foundation team authorization, as with
other fork pull requests.

The quality gate stripped frontmatter with a regex carrying the `m` flag, so
`^---` matched at every line start rather than only at the start of the
document. Frontmatter is optional when publishing, so a SKILL.md that opens
with a heading and uses `---` as an ordinary Markdown thematic break had
everything between its first two rules deleted before the body was measured.

The truncated body then fell under the word and character floors and the
publish was rejected outright with "Skill content is too thin or templated".
The same truncation also fed the structural fingerprint used for template-spam
detection.

The three other frontmatter parsers in the repository are all anchored to the
start of the document; this one is now consistent with them.
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@Yigtwxx is attempting to deploy a commit to the OpenClaw Foundation Team on Vercel.

A member of the Team first needs to authorize it.

@Patrick-Erichsen

Copy link
Copy Markdown
Collaborator

Maintainer follow-up pushed at exact head cab5b600126476676ed816a67e6ce82136c0b3ec.

The quality gate now gets its body from the shared SKILL.md parser instead of maintaining a second frontmatter regex. The parser only strips a delimited block when it parses as a YAML object, normalizes LF/CRLF/CR line endings, and preserves leading Markdown thematic breaks plus malformed or unterminated frontmatter-like text.

Regression coverage now includes:

  • the originally reported frontmatter-less SKILL.md with mid-document --- rules
  • a frontmatter-less document beginning with ---
  • valid YAML frontmatter
  • malformed and missing closing delimiters
  • LF, CRLF, and CR line endings

Validation:

  • focused publish/parser tests: 69/69 passed
  • bun run ci:static: passed
  • VITE_CONVEX_URL=https://example.invalid bun run ci:unit: 5,646 passed, 2 skipped
  • bun run ci:types-build: passed
  • autoreview: no actionable findings
  • exact-head GitHub repository checks: passed, including static, unit, packages, types-build, e2e-http, Playwright, CodeQL, and secret scanning

The fork-only Vercel authorization status remains unrelated and non-required. The earlier long-running moderation browser lane passed on retry after the first runner experienced broad local Convex timeouts.

Maintainer verdict: merge.

@Patrick-Erichsen
Patrick-Erichsen merged commit 3979883 into openclaw:main Jul 30, 2026
49 of 52 checks passed
@Yigtwxx

Yigtwxx commented Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the follow-up — routing the gate through parseSkillMarkdown is the better shape than the flag fix it landed on top of: it deletes the fourth frontmatter regex instead of correcting it, and gating the strip on "parses as a YAML object" also covers the case an anchored pattern still gets wrong, a frontmatter-less document that legitimately opens with --- as a rule. The CR/CRLF and unterminated-delimiter cases are a good addition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants