Websockets by stefangutica · Pull Request #1528 · multiversx/mx-api-service

stefangutica · 2025-08-20T13:48:50Z

Reasoning

Allow clients to subscribe to endpoints instead of performing multiple request to fetch updates

Proposed Changes

Integrate websockets subscription for transactions, blocks and stats

How to test

Use a client simulation script and subscribe to multiple endpoints with different parameters and multiple topics on each client
Example:
client1: subscribeTransactions with payload {from=0, size=10}, subscribeBlocks with payload {from=0, size=2}
client2: subscribeTransactions with payload {from=0, size=2}, subscribeBlocks with payload {from=0, size=5}, subscribeStats

on each client you should only receive data the same as you would request that specific endpoint with those specific parameters and the server must perform only 1 computation per each room at a specific interval regardless of the number of clients in that room -> a room is unique by topic and payload (filter parameters)

github-actions · 2025-08-20T13:51:56Z

k6 load testing comparison.
Base Commit Hash: 91f20cc
Target Commit Hash: a4dfb21

Metric	Base				Target				Diff
Metric	Avg	Max	90	95	Avg	Max	90	95	Avg	Max	90	95
Transactions	76.58	1445.34	65.13	73.12	76.14	1654.38	64.16	71.35	-0.58% ✅	+14.46% 🔴	-1.49% ✅	-2.43% ✅
Blocks	67.67	532.14	65.33	72.42	67.71	708.27	63.87	74.65	+0.05% 🔴	+33.10% 🔴	-2.23% ✅	+3.08% 🔴
Mex	60.33	824.98	64.16	66.82	58.00	578.84	62.71	65.60	-3.86% ✅	-29.84% ✅	-2.26% ✅	-1.83% ✅
Nodes	60.25	635.35	64.22	66.87	58.91	910.16	62.85	65.76	-2.22% ✅	+43.25% 🔴	-2.14% ✅	-1.65% ✅
Tokens	59.86	463.28	64.15	66.76	58.11	326.50	62.72	65.65	-2.93% ✅	-29.52% ✅	-2.24% ✅	-1.66% ✅
Accounts	60.81	1078.37	64.16	66.65	59.25	943.70	62.86	65.89	-2.58% ✅	-12.49% ✅	-2.03% ✅	-1.14% ✅
Pool	59.45	292.74	64.15	66.84	58.65	800.41	62.78	65.74	-1.36% ✅	+173.42% 🔴	-2.13% ✅	-1.65% ✅
Test Run Duration	60003.30				60004.83

Legend: Avg - Average Response Time, Max - Maximum Response Time, 90 - 90th Percentile, 95 - 95th Percentile
All times are in milliseconds.

Signed-off-by: GuticaStefan <stefan.gutica@gmail.com>

bogdan-rosianu · 2025-09-03T14:13:15Z

  collectionPropertiesFromGateway: false
 features:
+  websocketSubscription:
+    enabled: true


make them disabled by default maybe? can be left as a TODO before PR's merging

# Conflicts: # config/config.mainnet.yaml

bogdan-rosianu · 2025-10-16T06:50:25Z

+    @ConnectedSocket() client: Socket,
+    @MessageBody(new WsValidationPipe()) payload: BlockSubscribePayload
+  ) {
+    const filterHash = JSON.stringify(payload);


this is not really a hash, so maybe we should update the naming to something like filterIdentifier. Please apply this suggestion in all the gateways

bogdan-rosianu · 2025-10-16T06:51:27Z

+    handleDisconnect(_client: Socket) { }
+
+    handleConnection(client: Socket, ..._args: any[]) {
+        client.setMaxListeners(12);


is it enough? should we extract this number to the config file?

if the number is exceeded we will only receive a warning(not error) from EventEmitter, but it doesn't affect the functionality. For the moment, until we add another class like blocks,pools,... for ws subscription, that warning should not appear

bogdan-rosianu · 2025-10-16T06:55:14Z

@@ -0,0 +1,29 @@
+// block-subscribe.dto.ts


remove this comment

bogdan-rosianu · 2025-10-16T06:55:25Z

@@ -0,0 +1,20 @@
+// block-subscribe.dto.ts


remove this comment

bogdan-rosianu · 2025-10-16T06:56:01Z

@@ -0,0 +1,16 @@
+// ws-validation.pipe.ts


remove this comment

* Enhance error handling in GatewayProxyController for block retrieval by hash (#1545) * use EsdtType instead of hardcoded values (#1544) * token service improvements (#1543) * processes 50 tokens concurrently using ConcurrencyUtils * run all three batch operations in parallel with increased concurrency * add more logs * apply mex prices * add logs ( to be removed ) * increse cron time expression * set cron to 2 minutes * Websocket subscriptions (#1528) * add websockets for blocks and txs * add support for subscribe to stats * use websockets rooms * remove logs * add lock on crons * check stats room exists Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> * fix indent spaces * add validation pipes + filters * add try catch + class validator fixes * fix linter * add pool subscription + reduce filters combinations for subscriptions * lint * add support for events subscription * lint * separate subscription websocket into separate app * add config * add path * fix * add path for events + config default settings * temp logs * temp logs 2 * added missing configs + remove temp logs * enable andromeda in config * add metrics on subscription * remove async + reschedule * refresh metrics every second * set max listeners to 12 * add EOL * add count on update + parallel broadcast to rooms * lower ttl for blocks count cache * remove comments * remove comments * renaming --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: bogdan-rosianu <bogdan.rosianu@yahoo.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com> * log instead of error for invalid legacy delegation contract (#1542) --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: Catalin Faur <52102171+cfaur09@users.noreply.github.com> Co-authored-by: Gutica Stefan <123564494+stefangutica@users.noreply.github.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com>

* Enhance error handling in GatewayProxyController for block retrieval by hash (#1545) * use EsdtType instead of hardcoded values (#1544) * token service improvements (#1543) * processes 50 tokens concurrently using ConcurrencyUtils * run all three batch operations in parallel with increased concurrency * add more logs * apply mex prices * add logs ( to be removed ) * increse cron time expression * set cron to 2 minutes * Websocket subscriptions (#1528) * add websockets for blocks and txs * add support for subscribe to stats * use websockets rooms * remove logs * add lock on crons * check stats room exists Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> * fix indent spaces * add validation pipes + filters * add try catch + class validator fixes * fix linter * add pool subscription + reduce filters combinations for subscriptions * lint * add support for events subscription * lint * separate subscription websocket into separate app * add config * add path * fix * add path for events + config default settings * temp logs * temp logs 2 * added missing configs + remove temp logs * enable andromeda in config * add metrics on subscription * remove async + reschedule * refresh metrics every second * set max listeners to 12 * add EOL * add count on update + parallel broadcast to rooms * lower ttl for blocks count cache * remove comments * remove comments * renaming --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: bogdan-rosianu <bogdan.rosianu@yahoo.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com> * log instead of error for invalid legacy delegation contract (#1542) * implement event logAddress filter (#1548) * implement event logAddress filter * implement topics filter * Token market cap updates (#1550) * token mcap update * add logs * extend logs * add support for timestampMs for account * fix unit tests * improve token fetch price (#1549) * improve token fetch price * remove logger * update spec * fix lint error * add logs for error * add price * Enhance NFT type support in cache warmer and collection services (#1552) * Refactor function filter application in ElasticIndexerHelper to enhance query conditions with AND operator and existence checks (#1554) * apply supply info for all tokens (#1558) * Refactor token market cap calculation to ensure price and circulating supply are checked before computation * add specs * fix lint * added reserved field to blocks (#1560) * add support for custom url custom headers (#1557) * Add custom URL headers support and enhance token data fetching * Enhance custom URL headers handling by adding JSON parsing and validation checks * add logs * fixes * remove app.hatom.com value * add EOL * fix indentation --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: Catalin Faur <52102171+cfaur09@users.noreply.github.com> Co-authored-by: Gutica Stefan <123564494+stefangutica@users.noreply.github.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com> Co-authored-by: GuticaStefan <stefan.gutica@gmail.com>

* extend features flags config values * fixes after review + test fix * set more default configs to false * fixes after merge * Enhance error handling in GatewayProxyController for block retrieval by hash (#1545) * use EsdtType instead of hardcoded values (#1544) * token service improvements (#1543) * processes 50 tokens concurrently using ConcurrencyUtils * run all three batch operations in parallel with increased concurrency * add more logs * apply mex prices * add logs ( to be removed ) * increse cron time expression * set cron to 2 minutes * Websocket subscriptions (#1528) * add websockets for blocks and txs * add support for subscribe to stats * use websockets rooms * remove logs * add lock on crons * check stats room exists Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> * fix indent spaces * add validation pipes + filters * add try catch + class validator fixes * fix linter * add pool subscription + reduce filters combinations for subscriptions * lint * add support for events subscription * lint * separate subscription websocket into separate app * add config * add path * fix * add path for events + config default settings * temp logs * temp logs 2 * added missing configs + remove temp logs * enable andromeda in config * add metrics on subscription * remove async + reschedule * refresh metrics every second * set max listeners to 12 * add EOL * add count on update + parallel broadcast to rooms * lower ttl for blocks count cache * remove comments * remove comments * renaming --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: bogdan-rosianu <bogdan.rosianu@yahoo.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com> * log instead of error for invalid legacy delegation contract (#1542) * implement event logAddress filter (#1548) * implement event logAddress filter * implement topics filter * Token market cap updates (#1550) * token mcap update * add logs * extend logs * add support for timestampMs for account * fix unit tests * improve token fetch price (#1549) * improve token fetch price * remove logger * update spec * fix lint error * add logs for error * add price * Enhance NFT type support in cache warmer and collection services (#1552) * Refactor function filter application in ElasticIndexerHelper to enhance query conditions with AND operator and existence checks (#1554) * apply supply info for all tokens (#1558) * Refactor token market cap calculation to ensure price and circulating supply are checked before computation * add specs * fix lint * added reserved field to blocks (#1560) * add support for custom url custom headers (#1557) * Add custom URL headers support and enhance token data fetching * Enhance custom URL headers handling by adding JSON parsing and validation checks * add logs * fixes * remove app.hatom.com value * add EOL * fix indentation * API-741: staking v5 support * bugfix * fix lint * refactor * fixes * try-catch for queue computing * hotfix disable snapshotless * revert * add staking v5 activation epoch feature * added devnet config * small fix * added mainnet config * add stakingv5 e2e configs * Deprecate relayed v1/v2 (#1564) * Add transaction reordering by nonce for account sent transactions * add relayed deprecation to transfers as well * add custom subscriptions * fix lint * fix lint & unit tests * hardcode class fields * lint * refactor * add tests for room key generator * iterate over round only if there are active subscriptions * improvements * add support for unsubscribe * lint * add support for events * add support for timestampMs in ES * lint * hardcode timestamp supernova in the future * extra checks + clean code * fixes * fixes * typo * use latest chainsim * fix * add unit tests for transaction reordering by nonce for account sent transactions (#1572) * add chain simulator e2e tests * lint * fix * missing comma * fix * Refactor account, NFT, and transaction services to improve concurrency and caching * improve NFT bulk fetch + applyOwner * add no empty payload validator for custom subscriptions * add global subscription limiter * remove log * add e2e tests for events * lint * added metric per topics * updates * implement ConcurrencyUtils in collection service * add tests for general subscriptions * add newline * use timestampMs * revert e2e mainnet config * custom verbose * use timestampMs on chain sim network * add subscription limits to config * fix * update configs * use barnard config * typo * added readme * typo * add lock on subscriptions * use same key for lock * fix * add async mutex * lint * return success status on stats * return unsubscribed status on stats * config update * collection service improvments * fix spec * fixes * fixes * add support for transfers subscription * add support for token transfers subscriptions * add ws transfers e2e tests * fix unit tests * fix transction filters cache * remove duplicate code * add new line * encapsulate into variable * renaming * refactoring + skip unneeded iterations for duplicate keys * explanation comments * add unit test for disallowed field comb constraint * fixes after review * add configurable broadcast interval * add check keys present for transfers + events filtered * remove async await * fix * fix nft attributes source * fix lint * allow admins to trigger nft updates * print * revert log --------- Signed-off-by: GuticaStefan <stefan.gutica@gmail.com> Co-authored-by: Catalin Faur <52102171+cfaur09@users.noreply.github.com> Co-authored-by: Gutica Stefan <123564494+stefangutica@users.noreply.github.com> Co-authored-by: cfaur09 <catalinfaurpaul@gmail.com> Co-authored-by: GuticaStefan <stefan.gutica@gmail.com>

add websockets for blocks and txs

5239843

stefangutica marked this pull request as draft August 20, 2025 13:49

stefangutica added 6 commits August 21, 2025 17:46

add support for subscribe to stats

6be90b6

use websockets rooms

1442509

remove logs

7d322ee

add lock on crons

62974c7

check stats room exists

8006d5e

Signed-off-by: GuticaStefan <stefan.gutica@gmail.com>

fix indent spaces

d09f688

cfaur09 reviewed Aug 22, 2025

View reviewed changes

Comment thread src/endpoints/blocks/blocks.gateway.ts Outdated

cfaur09 reviewed Aug 22, 2025

View reviewed changes

Comment thread src/endpoints/blocks/blocks.gateway.ts Outdated

cfaur09 reviewed Aug 22, 2025

View reviewed changes

Comment thread src/endpoints/network/network.gateway.ts Outdated

stefangutica added 10 commits August 22, 2025 14:40

add validation pipes + filters

62fa109

add try catch + class validator fixes

a7e71ea

fix linter

58be392

add pool subscription + reduce filters combinations for subscriptions

cb92d2b

lint

344330e

add support for events subscription

404b18f

lint

d12208d

separate subscription websocket into separate app

c0833a7

add config

4f9741d

add path

889a75b

stefangutica marked this pull request as ready for review September 3, 2025 13:54

fix

cdfc711

bogdan-rosianu reviewed Sep 3, 2025

View reviewed changes

stefangutica and others added 5 commits September 3, 2025 17:20

add path for events + config default settings

788284b

temp logs

207ef83

temp logs 2

524fd9e

added missing configs + remove temp logs

e644794

enable andromeda in config

c5dc598

stefangutica and others added 9 commits September 8, 2025 15:10

add metrics on subscription

cad49bc

remove async + reschedule

8e92d63

refresh metrics every second

578303a

set max listeners to 12

6679b1f

Merge branch 'development' into websockets-poc

703a19e

add EOL

f6f747b

add count on update + parallel broadcast to rooms

d86907f

Merge branch 'development' into websockets-poc

5ceec0e

# Conflicts: # config/config.mainnet.yaml

lower ttl for blocks count cache

bdb82de

bogdan-rosianu reviewed Oct 16, 2025

View reviewed changes

stefangutica added 3 commits October 16, 2025 09:58

remove comments

341cbe3

remove comments

e0c2295

renaming

a4dfb21

bogdan-rosianu approved these changes Oct 16, 2025

View reviewed changes

cfaur09 approved these changes Oct 16, 2025

View reviewed changes

bogdan-rosianu merged commit 2a9f59d into development Oct 16, 2025
6 checks passed

bogdan-rosianu deleted the websockets-poc branch October 16, 2025 07:41

Conversation

stefangutica commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reasoning

Proposed Changes

How to test

Uh oh!

github-actions Bot commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stefangutica commented Aug 20, 2025 •

edited

Loading

github-actions Bot commented Aug 20, 2025 •

edited

Loading