feat: add math cairo tests under vm/src/tests/cairo_test_suite

[naor-starkware]

TITLE

Description

Description of the pull request changes and motivation.

Checklist

• Linked to Github Issue
• Unit tests added
• Integration tests added.
• This change requires new documentation.
  • Documentation has been added/updated.
  • CHANGELOG has been updated.

---

This change is 

[naor-starkware]

• feat: add math cairo tests under vm/src/tests/cairo_test_suite #2379 👈 (View in Graphite)
• feat(makefile,ci): add cairo_test_suite_programs target and CI integration #2380
• feat: add test_helpers module (error_utils, test_utils) behind test_utlis flag #2381
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Benchmark Results for unmodified programs 🚀

Command	Mean [s]	Min [s]	Max [s]	Relative
base big_factorial	2.287 ± 0.041	2.249	2.385	1.01 ± 0.02
head big_factorial	2.271 ± 0.009	2.256	2.282	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
base big_fibonacci	2.186 ± 0.025	2.163	2.250	1.00
head big_fibonacci	2.210 ± 0.078	2.172	2.431	1.01 ± 0.04

Command	Mean [s]	Min [s]	Max [s]	Relative
base blake2s_integration_benchmark	7.622 ± 0.118	7.472	7.786	1.00 ± 0.02
head blake2s_integration_benchmark	7.588 ± 0.106	7.484	7.793	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
base compare_arrays_200000	2.285 ± 0.031	2.256	2.344	1.00
head compare_arrays_200000	2.293 ± 0.021	2.266	2.334	1.00 ± 0.02

Command	Mean [s]	Min [s]	Max [s]	Relative
base dict_integration_benchmark	1.483 ± 0.004	1.477	1.490	1.00
head dict_integration_benchmark	1.497 ± 0.023	1.481	1.545	1.01 ± 0.02

Command	Mean [s]	Min [s]	Max [s]	Relative
base field_arithmetic_get_square_benchmark	1.267 ± 0.014	1.251	1.300	1.00 ± 0.02
head field_arithmetic_get_square_benchmark	1.263 ± 0.013	1.253	1.296	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
base integration_builtins	7.713 ± 0.054	7.648	7.784	1.00
head integration_builtins	7.766 ± 0.187	7.631	8.276	1.01 ± 0.03

Command	Mean [s]	Min [s]	Max [s]	Relative
base keccak_integration_benchmark	7.836 ± 0.062	7.782	7.982	1.00 ± 0.02
head keccak_integration_benchmark	7.835 ± 0.109	7.709	8.116	1.00

Command	Mean [s]	Min [s]	Max [s]	Relative
base linear_search	2.281 ± 0.013	2.266	2.310	1.00
head linear_search	2.295 ± 0.026	2.274	2.349	1.01 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
base math_cmp_and_pow_integration_benchmark	1.580 ± 0.005	1.572	1.588	1.00
head math_cmp_and_pow_integration_benchmark	1.587 ± 0.008	1.575	1.605	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
base math_integration_benchmark	1.555 ± 0.010	1.543	1.572	1.00
head math_integration_benchmark	1.562 ± 0.016	1.549	1.593	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
base memory_integration_benchmark	1.303 ± 0.010	1.295	1.329	1.00
head memory_integration_benchmark	1.306 ± 0.015	1.294	1.348	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
base operations_with_data_structures_benchmarks	1.639 ± 0.006	1.628	1.647	1.00 ± 0.01
head operations_with_data_structures_benchmarks	1.634 ± 0.007	1.626	1.646	1.00

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base pedersen	566.8 ± 3.8	562.7	574.7	1.00
head pedersen	568.3 ± 3.0	563.0	571.5	1.00 ± 0.01

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base poseidon_integration_benchmark	672.6 ± 1.7	670.0	676.0	1.00
head poseidon_integration_benchmark	674.3 ± 5.7	668.7	688.4	1.00 ± 0.01

Command	Mean [s]	Min [s]	Max [s]	Relative
base secp_integration_benchmark	1.939 ± 0.008	1.928	1.955	1.00
head secp_integration_benchmark	1.958 ± 0.013	1.945	1.991	1.01 ± 0.01

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
base set_integration_benchmark	680.3 ± 1.3	678.2	681.7	1.00
head set_integration_benchmark	682.2 ± 2.1	678.8	685.4	1.00 ± 0.00

Command	Mean [s]	Min [s]	Max [s]	Relative
base uint256_integration_benchmark	4.355 ± 0.022	4.330	4.390	1.00 ± 0.01
head uint256_integration_benchmark	4.354 ± 0.020	4.330	4.389	1.00

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.20%. Comparing base (78eae20) to head (6e6f1d8).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2379   +/-   ##
=======================================
  Coverage   96.20%   96.20%           
=======================================
  Files         107      107           
  Lines       37963    37963           
=======================================
  Hits        36522    36522           
  Misses       1441     1441           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

**Hyper Thereading Benchmark results**




hyperfine -r 2 -n "hyper_threading_main threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_main' -n "hyper_threading_pr threads: 1" 'RAYON_NUM_THREADS=1 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 1
  Time (mean ± σ):     24.822 s ±  0.051 s    [User: 24.388 s, System: 0.431 s]
  Range (min … max):   24.787 s … 24.858 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 1
  Time (mean ± σ):     24.564 s ±  0.009 s    [User: 24.106 s, System: 0.455 s]
  Range (min … max):   24.558 s … 24.571 s    2 runs
 
Summary
  hyper_threading_pr threads: 1 ran
    1.01 ± 0.00 times faster than hyper_threading_main threads: 1




hyperfine -r 2 -n "hyper_threading_main threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_main' -n "hyper_threading_pr threads: 2" 'RAYON_NUM_THREADS=2 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 2
  Time (mean ± σ):     13.313 s ±  0.001 s    [User: 24.502 s, System: 0.433 s]
  Range (min … max):   13.313 s … 13.313 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 2
  Time (mean ± σ):     13.221 s ±  0.024 s    [User: 24.304 s, System: 0.453 s]
  Range (min … max):   13.204 s … 13.238 s    2 runs
 
Summary
  hyper_threading_pr threads: 2 ran
    1.01 ± 0.00 times faster than hyper_threading_main threads: 2




hyperfine -r 2 -n "hyper_threading_main threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_main' -n "hyper_threading_pr threads: 4" 'RAYON_NUM_THREADS=4 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 4
  Time (mean ± σ):     10.162 s ±  0.085 s    [User: 35.703 s, System: 0.557 s]
  Range (min … max):   10.102 s … 10.222 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 4
  Time (mean ± σ):      9.929 s ±  0.169 s    [User: 35.500 s, System: 0.594 s]
  Range (min … max):    9.809 s … 10.048 s    2 runs
 
Summary
  hyper_threading_pr threads: 4 ran
    1.02 ± 0.02 times faster than hyper_threading_main threads: 4




hyperfine -r 2 -n "hyper_threading_main threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_main' -n "hyper_threading_pr threads: 6" 'RAYON_NUM_THREADS=6 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 6
  Time (mean ± σ):      9.714 s ±  0.193 s    [User: 35.776 s, System: 0.620 s]
  Range (min … max):    9.578 s …  9.851 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 6
  Time (mean ± σ):      9.633 s ±  0.038 s    [User: 36.030 s, System: 0.603 s]
  Range (min … max):    9.607 s …  9.660 s    2 runs
 
Summary
  hyper_threading_pr threads: 6 ran
    1.01 ± 0.02 times faster than hyper_threading_main threads: 6




hyperfine -r 2 -n "hyper_threading_main threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_main' -n "hyper_threading_pr threads: 8" 'RAYON_NUM_THREADS=8 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 8
  Time (mean ± σ):      9.793 s ±  0.084 s    [User: 36.806 s, System: 0.648 s]
  Range (min … max):    9.733 s …  9.852 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 8
  Time (mean ± σ):      9.488 s ±  0.023 s    [User: 36.403 s, System: 0.656 s]
  Range (min … max):    9.472 s …  9.505 s    2 runs
 
Summary
  hyper_threading_pr threads: 8 ran
    1.03 ± 0.01 times faster than hyper_threading_main threads: 8




hyperfine -r 2 -n "hyper_threading_main threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_main' -n "hyper_threading_pr threads: 16" 'RAYON_NUM_THREADS=16 ./hyper_threading_pr'
Benchmark 1: hyper_threading_main threads: 16
  Time (mean ± σ):      9.733 s ±  0.096 s    [User: 37.218 s, System: 0.683 s]
  Range (min … max):    9.666 s …  9.801 s    2 runs
 
Benchmark 2: hyper_threading_pr threads: 16
  Time (mean ± σ):      9.796 s ±  0.007 s    [User: 36.835 s, System: 0.718 s]
  Range (min … max):    9.791 s …  9.801 s    2 runs
 
Summary
  hyper_threading_main threads: 16 ran
    1.01 ± 0.01 times faster than hyper_threading_pr threads: 16

[OmriEshhar1]

@OmriEshhar1 reviewed 17 files and all commit messages, and made 1 comment.
Reviewable status: 17 of 18 files reviewed, 1 unresolved discussion (waiting on naor-starkware and YairVaknin-starkware).

---

a discussion (no related file):
I'm not sure about the directories structure.
put each cairo test file is a separate folder?
or keep the python structure (have a big 'common' folder)?
@YairVaknin-starkware wdyt?

[OmriEshhar1]

@OmriEshhar1 partially reviewed 3 files and resolved 1 discussion.
Reviewable status: 16 of 18 files reviewed, all discussions resolved.

[OmriEshhar1]

@OmriEshhar1 partially reviewed 2 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on naor-starkware).