Skip to content

fix: cap _list_head request limit at _MAX_HEAD_ITEMS#869

Merged
vdusek merged 1 commit intomasterfrom
fix/list-head-request-limit
Apr 20, 2026
Merged

fix: cap _list_head request limit at _MAX_HEAD_ITEMS#869
vdusek merged 1 commit intomasterfrom
fix/list-head-request-limit

Conversation

@vdusek
Copy link
Copy Markdown
Contributor

@vdusek vdusek commented Apr 20, 2026

Summary

ApifyRequestQueueSingleClient._list_head used max instead of min when computing the limit passed to the Apify API:

requested_head_items = max(self._MAX_HEAD_ITEMS, desired_new_head_items + len(self._requests_in_progress))

Two problems with this:

  1. Over-fetch in the common case. When desired + in_progress < 1000, max forces the limit to 1000, pulling up to 5× more items than the logic actually wants (e.g. 1000 instead of 200 on a fresh queue).
  2. API cap breach when the queue is busy. When len(self._requests_in_progress) > 800, the sum exceeds _MAX_HEAD_ITEMS = 1000 and max picks that larger value as the limit, violating the documented API cap.

The comment on the preceding line describes the intended "pad, then cap" behavior — the code should pad by in-progress count and then cap at the API maximum. Fix is one character: maxmin.

Added three parametrized regression cases covering no-in-progress, mid-queue padding, and the cap boundary.

`_list_head` used `max` instead of `min` when sizing its `list_head` request, which both over-fetched in the common case and exceeded the documented API cap of 1000 once in-progress requests passed 800. Switching to `min` restores the intended pad-then-cap behavior.
@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Apr 20, 2026
@vdusek vdusek self-assigned this Apr 20, 2026
@github-actions github-actions Bot added this to the 139th sprint - Tooling team milestone Apr 20, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label Apr 20, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.67%. Comparing base (16bf07d) to head (55d478a).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #869      +/-   ##
==========================================
+ Coverage   86.64%   86.67%   +0.03%     
==========================================
  Files          48       48              
  Lines        2920     2920              
==========================================
+ Hits         2530     2531       +1     
+ Misses        390      389       -1     
Flag Coverage Δ
e2e 37.87% <0.00%> (ø)
integration 59.31% <100.00%> (+0.03%) ⬆️
unit 75.37% <100.00%> (+0.68%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek requested a review from Pijukatel April 20, 2026 12:36
@vdusek vdusek merged commit 1716a2a into master Apr 20, 2026
31 checks passed
@vdusek vdusek deleted the fix/list-head-request-limit branch April 20, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants