es circuit breaker

[bogdan-rosianu]

Reasoning

• if the underlying elasticsearch service is under high load, we should limit the number of requests we send

Proposed Changes

• added a proxy/wrapper over the elastic search handler that will apply the circuit breaker pattern over it.
• in case requests take longer than normally expected, we should block further incoming requests towards es until the response time decreases

How to test

[github-actions]

k6 load testing comparison.
Base Commit Hash: a383889
Target Commit Hash: ac458b0

Metric	Base				Target				Diff
	Avg	Max	90	95	Avg	Max	90	95	Avg	Max	90	95
Mex	46.30	199.63	50.38	52.26	45.68	607.61	49.80	51.35	-1.34% ✅	+204.37% 🔴	-1.16% ✅	-1.73% ✅
Transactions	61.65	2499.45	51.27	53.94	61.07	3883.13	50.53	52.94	-0.94% ✅	+55.36% 🔴	-1.45% ✅	-1.86% ✅
Tokens	46.99	594.27	50.42	52.26	45.86	503.80	49.78	51.29	-2.40% ✅	-15.22% ✅	-1.28% ✅	-1.87% ✅
Accounts	47.39	889.74	50.30	52.25	46.87	1343.32	49.68	51.15	-1.11% ✅	+50.98% 🔴	-1.23% ✅	-2.12% ✅
Pool	46.87	465.77	50.40	52.17	45.84	515.18	49.83	51.38	-2.20% ✅	+10.61% 🔴	-1.13% ✅	-1.52% ✅
Blocks	54.03	1414.59	50.89	53.53	52.33	1385.68	50.25	52.34	-3.15% ✅	-2.04% ✅	-1.26% ✅	-2.22% ✅
Nodes	46.67	289.49	50.40	52.25	47.05	1584.91	49.93	51.75	+0.81% 🔴	+447.48% 🔴	-0.95% ✅	-0.96% ✅
Test Run Duration	60001.87				60003.36

Legend: Avg - Average Response Time, Max - Maximum Response Time, 90 - 90th Percentile, 95 - 95th Percentile
All times are in milliseconds.

[cfaur09]

failureCountThreshold value is applied multiple times