CPU usage growing over time with DirectFileStore

[ethax-ross]

We've got an issue on our application where the CPU usage will slowly grow over time (eventually getting to 100%, unless we reboot the app to bring it back down):

We're using the DirectFileStore and find that if we clear the bin files manually (as is done on app boot) the CPU usage immediately drops off a cliff:

No issues with memory usage. Any thoughts on what could be causing this?

[dmagliola]

Hello, thank you for reporting this.
This sounds like it's very likely related to issue #143
Over time, too many files accumulate and that becomes an issue.

We don't have a solution just now, but i'm planning to make a change to the File Store so it keeps a single file per process, instead of a file per-metric-per-process, which I expect would help, but I won't know until we try it.

Since you are experiencing the problem, however, your particular setup is really interesting to us...

Could I ask you a number of questions?

• When this starts happening, how many files do you have in the Prom Store directory?
• How many processes are you running in your web server?
• How many metrics are you exporting?
• When the /metrics endpoint gets hit, and you are experiencing high CPU, what's the response time of that endpoint?

This will help me get an idea of the size of the issue, but also to generate some "mock files" that I can use to performance benchmark different alternatives.

Thank you for the report, and sorry I don't have an answer yet!