Count tokens worker

[cte]

Context

There does seem to be some benefit to offloading tiktoken in a web worker, especially for thinking models on older hardware. This is a slightly different take on @samhvw8's original implementation in #2848. I used an existing library (workerpool) for simplicity.

Implementation

Screenshots

before	after

How to Test

Get in Touch

---

Important

Introduces worker-based token counting using workerpool, updates BaseProvider to utilize it, and adds tests for the tiktoken function.

• Behavior:
  • Introduces worker-based token counting using workerpool in countTokens.ts.
  • Updates BaseProvider in base-provider.ts to use countTokens with workers.
  • Adds tiktoken function in tiktoken.ts for token counting.
• Testing:
  • Adds tests for tiktoken in tiktoken.test.ts to verify token counting for text and image content.
• Dependencies:
  • Adds workerpool as a dependency in package.json.
• Configuration:
  • Updates esbuild.js to include worker configuration for countTokens.ts.
  • Modifies knip.json to ignore src/workers/**.

^{This description was created by for 8224d4c. You can customize this summary. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 8224d4c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR