The Incremental API aka "The Greedy File Watcher"

[johnnyreilly]

Heya!

Maintainer of the fork-ts-checker-webpack-plugin here. Under the covers, the plugin uses the TypeScript incremental API. Various issues have been raised when using this with the level of CPU usage on idle; i.e. it's high!

• CPU usage on idle? TypeStrong/fork-ts-checker-webpack-plugin#236
• React-scripts/fork-ts-checker-webpack-plugin constant high CPU usage when using material/icons facebook/create-react-app#6792

The marvellous @NeKJ did some digging and discovered that this could be remedied by setting an environment variable: TSC_WATCHFILE=UseFsEventsWithFallbackDynamicPolling.

This changes the behaviour of TypeScripts approach to file watching in a highly desirable fashion. You can see details of @NeKJ's investigation here:

TypeStrong/fork-ts-checker-webpack-plugin#256

As you'll see, this is a PR for fork-ts-checker-webpack-plugin that changes the TSC_WATCHFILE environment variable. If we merge it the users of fork-ts-checker-webpack-plugin should gain a better experience on the resources front.

However, it feels like this is potentially the wrong place to make this change. So before we merge and ship this I wanted to throw it back to you. If this affects users of fork-ts-checker-webpack-plugin then I'd hazard a guess other people who use the API are being burned too. Changing the default in fork-ts-checker-webpack-plugin won't help them.

Should the default file watching approach used by TypeScript be switched to TSC_WATCHFILE=UseFsEventsWithFallbackDynamicPolling or similar? So all users of this API have a better experience?

[DanielRosenwasser]

@sheetalkamat, thoughts? We did chat with VS Code a bit about this. Did they see improvements?

[johnnyreilly]

Hey @DanielRosenwasser / @sheetalkamat,

I was just wondering if there's been any discussion around this? I can certainly testify to lots of people who use create-react-app / fork-ts-checker-webpack-plugin logging issues related to this.

It would be great to know if this is something you're looking to address?

In the meantime (time permitting) I may blog about this to spread the word on the workaround that people can already use.

I've blogged about it in the meantime: https://blog.johnnyreilly.com/2019/05/typescript-and-high-cpu-usage-watch.html

Thanks for all your excellent work! ♥️🤗

[DanielRosenwasser]

I remember chatting with Sheetal about this in person, but it's been a couple of weeks. My understanding is that this really is something that can only be determined by the end-user because different setups can affect this (e.g. operating system, file system, remote file system, etc.)

Is that accurate @sheetalkamat? @RyanCavanaugh?

[johnnyreilly]

My understanding is that this really is something that can only be determined by the end-user because different setups can affect this (e.g. operating system, file system, remote file system, etc.)

Would it be possible for you to share the reasoning behind why the current default is the one used?

I ask as, to quote @NeKJ:

TS uses internally the fs.watch and fs.watchFile API functions of nodejs for their watch mode. The latter function is even not recommended by nodejs documentation for performance reasons, and urges to use fs.watch instead.

NodeJS doc:

Using fs.watch() is more efficient than fs.watchFile and fs.unwatchFile. fs.watch should be used instead of fs.watchFile and fs.unwatchFile when possible.

He suggests that the present default is a choice that node themselves advocate against using.

Also, there seems to a good number of people who are finding the present default problematic. Each day that passes seems to bring another user either voting up the existing issue or commenting 😄

Also, those people who have tried setting the environment variable to UseFsEventsWithFallbackDynamicPolling have come back with happy comments and upvotes 😁 - see facebook/create-react-app#6792 (comment)

It would great to understand the rationale for the current choice over one of the alternatives. Oh and just to add a bit more context, here's some investigation which @NeKJ performed:

I measured all of them, using $ top -b -n2 -d 30 -p <pid of node that runs service.js> which is a sample of 30 seconds, run 10+ seconds after the build has been completed and everything has settled.

Value	CPU usage on idle
TS default (TSC_WATCHFILE not set)	7.4%
UseFsEventsWithFallbackDynamicPolling	0.2%
UseFsEventsOnParentDirectory	0.2%
PriorityPollingInterval	6.2%
DynamicPriorityPolling	0.5%
UseFsEvents	0.2%

As we can see, the worst option of them all is the default behaviour of TS and next to it is the PriorityPollingInterval where it is a tad bit lower. DynamicPriority is at 0.5% and all rest at 0.2%. Therefore the UseFsEventsWithFallbackDynamicPolling is the best choice as it will in the best case choose UseFsEvents with 0.2% or in the worst use DynamicPriorityPolling which goes up to 0.5%.

These all are measurements on my own computer which is a PC running linux 64bit. To do this right,we will need measurements from other systems too (OSX, Windows etc).