Skip to content

feat: add getContents() for reading external markdown files into table descriptions#2138

Open
AmmanuelT wants to merge 8 commits intodataform-co:mainfrom
AmmanuelT:read_description_from_external_file
Open

feat: add getContents() for reading external markdown files into table descriptions#2138
AmmanuelT wants to merge 8 commits intodataform-co:mainfrom
AmmanuelT:read_description_from_external_file

Conversation

@AmmanuelT
Copy link
Copy Markdown
Contributor

Closes #1723

Allows users to source table descriptions from external markdown files, keeping .sqlx files smaller and enabling reusable, renderable descriptions across the project.

Usage:

config {
  type: "table",
  description: getContents('./my_table_description.md')
}
SELECT ...

Changes:

  • core/compilers.ts : .md files now compile to exports.contents = ...``
  • core/session.ts : new getContents(filePath) method; resolves the path relative to the calling file using the call stack, then builds an absolute path for nativeRequire
  • core/main.ts : getContents bound to the global scope
  • cli/vm/compile.ts + testing/run_core.ts: "md" added to sourceExtensions so vm2 applies the compiler to .md files
  • core/main_test.ts + core/compilers_test.ts : tests for happy path and missing file error

Notes:

  • The method is currently named getContents, inspired by the suggestion in the issue thread, but happy to rename before merge if the maintainers have a preference!
  • When getContents references a file that doesn't exist, the error surfaces as a VMError: Cannot find module '/absolute/path/to/file.md' compilation error. A cleaner message could be produced by intercepting in dataformCompile in main.ts also happy to add this if preferred, just wanted to flag the tradeoff before doing extra work.

@AmmanuelT AmmanuelT requested a review from a team as a code owner April 3, 2026 16:06
@AmmanuelT AmmanuelT requested review from krushangSk17 and removed request for a team April 3, 2026 16:06
@kolina
Copy link
Copy Markdown
Contributor

kolina commented Apr 11, 2026

/gcbrun

Copy link
Copy Markdown
Contributor

@kolina kolina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your contribution!

Overall looks good, I've left some not very big comments to address.

Currently tests are failing due to expired BQ credentials, can you please rebase your PR onto #2139 so I can re-run them?

Comment thread core/session.ts Outdated
Comment thread docs/reference/session.md Outdated
Comment thread core/session.ts
@AmmanuelT
Copy link
Copy Markdown
Contributor Author

Thanks for the comments! Working on the fixes!

@AmmanuelT AmmanuelT force-pushed the read_description_from_external_file branch from 8cf17c1 to cf359f2 Compare April 11, 2026 17:54
@AmmanuelT
Copy link
Copy Markdown
Contributor Author

@kolina ready for another check 🫡

Comment thread core/session.ts Outdated
const resolvedPath = Path.normalize(Path.join(callerDir,filePath));
const absolutePath = nodePath.join(this.rootDir, resolvedPath);

if (!absolutePath.startsWith(this.rootDir)){
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add separator to this.rootDir here (if it already doesn't end with it) to make sure that file a/somepath.txt won't be considered inside a/somepath/ folder

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't even considered that, good call! It's been added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read table description from external files

2 participants