feat: support "draft" as a first-class spec version target#255
feat: support "draft" as a first-class spec version target#255
Conversation
- Add LATEST_SPEC_VERSION and DATED_SPEC_VERSIONS constants so the next spec release is a one-line change in types.ts. - Accept "draft" as a valid protocolVersion in the initialize check and mock-server response. - --spec-version draft now selects latest-dated scenarios plus draft-tagged ones, so SEP authors can run the full suite against an SDK tracking the in-progress spec without retagging core scenarios. - Forward --spec-version to the client process via MCP_CONFORMANCE_SPEC_VERSION so SDK examples can pick the matching protocolVersion. Closes #253
commit: |
felixweinberger
left a comment
There was a problem hiding this comment.
LGTM, optional nit on consolidating a single constant instead of having 2 copies
| const VALID_VERSIONS = [ | ||
| '2025-06-18', | ||
| LATEST_SPEC_VERSION, | ||
| DRAFT_PROTOCOL_VERSION | ||
| ]; |
There was a problem hiding this comment.
could consider consolildating a VALID_VERSIONS constant in types.ts instead of having 2 copies here and in client.ts
| // could reuse the filter plumbing. It has no corresponding wire | ||
| // protocolVersion. Split it out of this type when moving to | ||
| // introducedIn/removedIn tagging. | ||
| if (version === 'extension') return undefined; |
There was a problem hiding this comment.
feels like something worth cleaning up, spec vs protocolVersion might get more confusing over time the longer we have both of these 🤔
There was a problem hiding this comment.
👍 we need a different way to refer to extensions, it doesn't super make sense to run "all extensions" as a selector. ties a bit into my comment on #114 about making it easier to selectively run scenarios via a config file
|
|
||
| // Mirrors LATEST_PROTOCOL_VERSION in the spec repo's schema/draft/schema.ts. | ||
| // Bump when that constant changes. | ||
| export const DRAFT_PROTOCOL_VERSION = 'DRAFT-2026-v1'; |
There was a problem hiding this comment.
I'm confused. What protocol version should we be using for conformance tests of draft features? This one or "draft"? If it is this one, then why the special casing of "draft" for the --spec-version parameter?
Closes #253.
Problem
SEP authors testing against the in-progress spec can't run the full suite: the
initializecheck rejects draftprotocolVersionstrings, and--spec-version draftonly selects the handful of explicitly draft-tagged scenarios (skippinginitializeand the rest of the core suite). Tagging core scenarios with'draft'is blocked by the spec-version isolation test.Raised by @mikekistler in #conformance-testing-wg while testing SEP-2243.
Naming:
'draft'vs'DRAFT-2026-v1'vs'2026-06-DRAFT'There are two distinct namespaces in play, and the spec repo treats them differently:
docs/specification/draft/— the folder is literallydraft, renamed to a date on releaseSpecVersion = ... | 'draft'(unchanged)protocolVersionschema/draft/schema.ts→LATEST_PROTOCOL_VERSIONhas cycledDRAFT-2025-v1→-v2→-v3→DRAFT-2026-v1(history)DRAFT_PROTOCOL_VERSION = 'DRAFT-2026-v1'So this PR keeps
'draft'as the conformance scenario tag (it's a folder alias, not a wire value, and survives release renames cleanly), and pins the exact current draft wire string asDRAFT_PROTOCOL_VERSION. We considered a permissive/^draft/iregex but rejected it: an SDK sending a staleDRAFT-2025-v3is testing against a draft that no longer exists, and that should fail. The constant needs bumping when the spec repo bumps theirs — same maintenance model asLATEST_SPEC_VERSION. A'2026-06-DRAFT'form was also rejected: the spec repo doesn't use it anywhere and the release date isn't known in advance.Changes
types.ts: addDATED_SPEC_VERSIONS,LATEST_SPEC_VERSION, andDRAFT_PROTOCOL_VERSIONconstants plusspecVersionToProtocolVersion(). Next spec release (or draft revision bump) is a one-line change here.checks/client.ts/scenarios/client/initialize.ts: acceptDRAFT_PROTOCOL_VERSIONas a validprotocolVersion(exact match — stale draft strings likeDRAFT-2025-v1are rejected). Mock server echoes it back.scenarios/index.ts: newmatchesSpecVersion()helper.--spec-version draftnow resolves to latest-dated ∪ draft-tagged — scenarios stay tagged draft-only, but selection treats draft as a superset of the current release.runner/client.ts/index.ts: forward the resolved wire version to the client process viaMCP_CONFORMANCE_PROTOCOL_VERSION(e.g.,--spec-version draft→DRAFT-2026-v1). Example clients can use this directly as theirprotocolVersionwithout maintaining their own mapping; SDKs that hard-code their version can ignore it.DRAFT_PROTOCOL_VERSION, negative tests reject'DRAFT-2025-v1'and bare'draft'.--spec-versionand the new env var.Behavioral note: tier-check
tier-checkcallslistScenariosForSpec(test-conformance-results.ts:208,264), so a tier-check run with--spec-version draftwill now score against latest ∪ draft instead of draft-only. This is internally consistent (filter and expected-list both go through the same function) but means draft tier scoring is meaningful only as "would this SDK pass tier N if draft shipped today."Testing
Typecheck and lint clean.
Follow-ups (not in this PR)
specVersions: [...]lists withintroducedIn/ optionalremovedIn. The superset model in this PR can't express a SEP that tightens or removes an existing requirement (e.g., SEP-986 narrowed valid tool-name format) — there's no subtraction arm. Every spec release also currently requires touching every carried-forward scenario.matchesSpecVersion()is the seam where range logic slots in; the migration is mechanical across ~40 scenario declarations. ('extension'would stay an orthogonal flag since it's off the dated timeline.)'2025-11-25'literals in server scenarios toLATEST_SPEC_VERSION.