Created
February 16, 2026 21:08
-
-
Save Flamefire/474b2ed3e327869308373fe2533b7beb to your computer and use it in GitHub Desktop.
(partial) EasyBuild log for failed build of /tmp/easybuild-tmp/eb-r39em19f/files_pr25240/p/PyTorch/PyTorch-2.9.1-foss-2024a.eb (PR(s) #25240)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cpuwr4ejhdiclsorzh2d7xuuz2mawhtb4sfayhjsy3zbgatl7caa.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-9b65970495403d7b.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 398 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu free(): invalid next size (normal) | |
| Fatal Python error: Aborted | |
| Thread 0x00007f8c70fed640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f9056b96740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpmsucet2e/yz/cyzzlppatzyscvecv5ksjrafoc5hyvk546sv3yth4wszovhpb3ey.py", line 1301 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 485 in run_test_with_call | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 1720 in test_windowed_partial_block_vs_sdpa | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cpuwr4ejhdiclsorzh2d7xuuz2mawhtb4sfayhjsy3zbgatl7caa.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-e83ae522e4d03748.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 398 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu Fatal Python error: Fatal Python error: Segmentation faultSegmentation fault | |
| Thread 0x00007f37e0d86640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Thread 0x00007f3be449d740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmposrscqf1/yz/cyzzlppatzyscvecv5ksjrafoc5hyvk546sv3yth4wszovhpb3ey.py", line 1301 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 485 in run_test_with_call | |
| File " | |
| Extension modules: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.pynumpy.core._multiarray_umath", line 1720 in test_windowed_partial_block_vs_sdpa | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py, "numpy.core._multiarray_tests, line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", , line numpy.linalg._umath_linalg589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634, in numpy.fft._pocketfft_internalrun | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", , line numpy.random._common3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", , line numpy.random.bit_generator3406 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py, "numpy.random._bounded_integers, line 517 in run | |
| File ", /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.pynumpy.random._mt19937", line 690 in __call__ | |
| File , "numpy.random.mtrand/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest, | |
| numpy.random._philox File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173, in numpy.random._pcg64pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", , line numpy.random._sfc64103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py, "numpy.random._generator, line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line , 241torch._C in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", , line torch._C._dynamo.autograd_compiler341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", , line torch._C._dynamo.eval_frame240 in call_and_report | |
| File ", /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.pytorch._C._dynamo.guards", line 135 in runtestprotocol | |
| File , "torch._C._dynamo.utils/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File ", /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.pytorch._C._fft", line 103 in _multicall | |
| File ", /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.pytorch._C._linalg", line 120 in _hookexec | |
| , File torch._C._nested"/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| , File torch._C._nn"/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in , pytest_runtestlooptorch._C._sparse | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line , 103torch._C._special in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, , gmpy2.gmpy2numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, psutil._psutil_posix, gmpy2.gmpy2, markupsafe._speedups, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cpuwr4ejhdiclsorzh2d7xuuz2mawhtb4sfayhjsy3zbgatl7caa.kernel, (total: cpuwr4ejhdiclsorzh2d7xuuz2mawhtb4sfayhjsy3zbgatl7caa.kernel29 (total: )29 | |
| ) | |
| double free or corruption (out) | |
| Fatal Python error: Aborted | |
| Thread 0x00007f37e0d86640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f3be449d740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmposrscqf1/yz/cyzzlppatzyscvecv5ksjrafoc5hyvk546sv3yth4wszovhpb3ey.py", line 1301 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 66 in inner | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File Got exit code -11 (SIGSEGV) | |
| Test failed consistently, continuing with the rest of the tests due to continue-through-error being set | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-2214395302aada9d.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 399 already run items. | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu munmap_chunk(): invalid pointer | |
| Fatal Python error: Aborted | |
| Thread 0x00007f54d4fed640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f58bb19c740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmp49hyf3w2/gf/cgfxk7v6acwbbk3q73fw5lpbqymlsj3zi5qvxhtiuzxdpip4w6al.py", line 1571 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/nn/attention/flex_attention.py", line 1449 in flex_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 616 in run_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 742 in run_test_with_call_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 1752 in test_windowed_partial_block_vs_sdpa_paged_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cftnvohphvva4p2butcw5nhe7bykzfdcip4elphyvfqk4i7l7w6q.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-6715fa705e6b52b2.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 399 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu munmap_chunk(): invalid pointer | |
| Fatal Python error: Aborted | |
| Thread 0x00007fb5d6d83640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007fb9e2934740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmp3gepyddh/gf/cgfxk7v6acwbbk3q73fw5lpbqymlsj3zi5qvxhtiuzxdpip4w6al.py", line 1571 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/nn/attention/flex_attention.py", line 1449 in flex_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 616 in run_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 742 in run_test_with_call_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 1752 in test_windowed_partial_block_vs_sdpa_paged_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cftnvohphvva4p2butcw5nhe7bykzfdcip4elphyvfqk4i7l7w6q.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Retrying single test... | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-08933c14e7701de3.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 399 deselected / 1 selected | |
| stepcurrent: skipping 399 already run items. Running only test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu | |
| Running 1 items in this shard | |
| inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu free(): invalid size | |
| Fatal Python error: Aborted | |
| Thread 0x00007f9188d86640 (most recent call first): | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1010 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1073 in _bootstrap_inner | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/threading.py", line 1030 in _bootstrap | |
| Current thread 0x00007f958d096740 (most recent call first): | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmp9zdhmgzy/gf/cgfxk7v6acwbbk3q73fw5lpbqymlsj3zi5qvxhtiuzxdpip4w6al.py", line 1571 in call | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/output_code.py", line 613 in __call__ | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 526 in wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 724 in inner_fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/utils.py", line 129 in call_func_at_runtime_with_args | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 353 in runtime_wrapper | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1130 in forward | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1044 in _fn | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/nn/attention/flex_attention.py", line 1449 in flex_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 832 in compile_wrapper | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 616 in run_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 742 in run_test_with_call_paged_attention | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 1752 in test_windowed_partial_block_vs_sdpa_paged_attention | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 426 in instantiated_test | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3224 in wrapper | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 589 in _callTestMethod | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 634 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3376 in _run_custom | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 3406 in run | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_device_type.py", line 517 in run | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/lib/python3.12/unittest/case.py", line 690 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/unittest.py", line 344 in runtest | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda> | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/pytest-rerunfailures/15.0-GCCcore-13.3.0/lib/python3.12/site-packages/pytest_rerunfailures.py", line 566 in pytest_runtest_protocol | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 364 in pytest_runtestloop | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 339 in _main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python-bundle-PyPI/2024.06-GCCcore-13.3.0/lib/python3.12/site-packages/_pytest/config/__init__.py", line 178 in main | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_dynamo/test_case.py", line 62 in run_tests | |
| File "/tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy/lib/python3.12/site-packages/torch/_inductor/test_case.py", line 15 in run_tests | |
| File "/dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/inductor/test_flex_decoding.py", line 2101 in <module> | |
| Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, cftnvohphvva4p2butcw5nhe7bykzfdcip4elphyvfqk4i7l7w6q.kernel (total: 29) | |
| Got exit code -6 (SIGIOT) | |
| Test failed consistently, continuing with the rest of the tests due to continue-through-error being set | |
| Test results will be stored in test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-26f641b4b37ddde2.xml | |
| ============================= test session starts ============================== | |
| platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0 -- /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/Python/3.12.3-GCCcore-13.3.0/bin/python | |
| cachedir: .pytest_cache | |
| hypothesis profile 'dev' -> database=None, max_examples=10, suppress_health_check=[HealthCheck.too_slow] | |
| rootdir: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1 | |
| configfile: pytest.ini | |
| plugins: xdist-3.6.1, subtests-0.13.1, rerunfailures-15.0, flakefinder-1.1.0, hypothesis-6.103.1, typeguard-4.3.0 | |
| collecting ... collected 400 items / 400 deselected / 0 selected | |
| stepcurrent: skipping 400 already run items. | |
| Running 0 items in this shard | |
| - generated xml file: /dev/shm/s3248973-EasyBuild/PyTorch/2.9.1/foss-2024a/pytorch-v2.9.1/test/test-reports/python-pytest/inductor.test_flex_decoding/inductor.test_flex_decoding-26f641b4b37ddde2.xml - | |
| =========================== 400 deselected in 0.32s ============================ | |
| The following tests failed consistently: ['test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod0_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod0_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod0_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod0_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod1_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod1_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod1_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod1_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod2_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod2_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod2_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod2_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod3_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod3_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod3_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod3_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod4_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod4_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod4_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod4_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod5_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod5_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod5_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod5_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod6_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod6_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod6_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod6_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod7_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod7_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod7_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod7_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod8_BLOCK_SIZE2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod8_BLOCK_SIZE3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod8_BLOCK_SIZE_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_different_block_size_float32_score_mod8_BLOCK_SIZE_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod4_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod4_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod5_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod5_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod6_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod6_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod7_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod7_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod8_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_builtin_score_mods_float32_score_mod8_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_captured_buffers_all_dims_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_captured_buffers_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_captured_scale_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_do_not_trigger_dynamic_shapes_on_empty_block_mask_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_fully_masked_out_rows_0_check_gqa_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_function_composition_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod4_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod4_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod5_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod5_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod6_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod6_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod7_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod7_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod8_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_head_dependent_mask_mod_float32_score_mod8_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims1_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims2_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims0_batch_dims3_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims2_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims1_batch_dims3_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims1_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims2_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod3_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod4_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod5_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod6_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod7_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_kv_batch_broadcast_float32_head_dims2_batch_dims3_score_mod8_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_head_seq_batch_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_seq_batch_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_load_from_bias_seq_only_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_max_autotune_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_max_autotune_with_captured_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_njt_causal_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_multi_token_offset_mask_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_offset_mask_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_divisible_offset_mask_with_captured_buffer_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod0_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod0_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod1_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod1_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod2_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod2_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod3_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod3_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod4_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod4_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod5_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod5_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod6_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod6_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod7_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod7_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod8_float32_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_non_equal_head_dims_score_mod8_float32_head_dims1_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_padded_dense_causal_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod0_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod1_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod2_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod3_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod4_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod5_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod6_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod7_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims0_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_128_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_256_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_paged_attention_page_size_float32_score_mod8_head_dims2_page_size_64_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_recompile_changed_score_mod_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_seq_masking_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_silu_on_score_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_skip_odd_keys_float32_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s0_v_s3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s1_v_s3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s2_v_s3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s0_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s0_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s1_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s1_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s2_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s2_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s3_head_dims0_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_strided_inputs_float32_k_s3_v_s3_head_dims2_cpu_float32', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_full_mask_vs_sdpa_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_full_mask_vs_sdpa_paged_attention_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_no_mask_vs_sdpa_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_no_mask_vs_sdpa_paged_attention_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_cpu', 'test/inductor/test_flex_decoding.py::TestFlexDecodingCPU::test_windowed_partial_block_vs_sdpa_paged_attention_cpu'] | |
| FINISHED PRINTING LOG FILE of inductor/test_flex_decoding 1/1 (test/test-reports/inductor.test_flex_decoding_1.1_2bb2af4a9e244b2c_.log) | |
| Finished inductor/test_flex_decoding 1/1 in 2898 minutes | |
| inductor/test_flex_decoding 1/1 failed! | |
| Running test batch 'tests to run' cost 226083.35 seconds | |
| inductor/test_flex_attention 1/1 failed! | |
| inductor/test_flex_decoding 1/1 failed! | |
| functorch/test_ops 1/1 failed! | |
| inductor/test_flex_decoding 1/1 failed! | |
| == 2026-02-16 21:28:51,717 filetools.py:423 INFO Path /tmp/easybuild-tmp/eb-r39em19f/tmpd1f2pemy successfully removed. | |
| == 2026-02-16 21:28:57,923 pytorch.py:712 WARNING Found 2 individual tests with failed assertions: TestOperatorsCPU.test_vmapvjp_linalg_tensorsolve_cpu_float32, TestOperatorsCPU.test_vmap_autograd_grad_nn_functional_conv2d_cpu_float32 | |
| == 2026-02-16 21:28:58,140 build_log.py:233 ERROR EasyBuild encountered an error: Failing because not all failed tests could be determined.Tests failed to start, crashed or the test accounting in the PyTorch EasyBlock needs updating! | |
| Missing (2): inductor/test_flex_attention, inductor/test_flex_decoding | |
| You can check the test failures (in the log) manually and if they are harmless, use --ignore-test-failure to make the test step pass. | |
| 2 test failures, 0 test errors (out of 261591): | |
| Failed tests (suites/files): | |
| functorch/test_ops (2 failed, 7489 passed, 2731 skipped, 0 errors) | |
| Could not count failed tests for the following test suites/files: | |
| inductor/test_flex_attention (Undetected or did not run properly) | |
| inductor/test_flex_decoding (Undetected or did not run properly) (at easybuild/easyblocks/p/pytorch.py:800 in test_step) | |
| Callstack: | |
| easybuild/easyblocks/p/pytorch.py:800 in test_step | |
| easybuild/framework/easyblock.py:3075 in _test_step | |
| easybuild/framework/easyblock.py:4788 in run_step | |
| easybuild/framework/easyblock.py:4947 in run_all_steps | |
| easybuild/framework/easyblock.py:5135 in build_and_install_one | |
| easybuild/main.py:180 in build_and_install_software | |
| easybuild/main.py:614 in process_eb_args | |
| easybuild/main.py:798 in main | |
| easybuild/main.py:847 in main_with_hooks | |
| easybuild/main.py:862 in <module> | |
| == 2026-02-16 21:28:58,142 build_log.py:233 ERROR EasyBuild encountered an error: An error was raised during test step: Failing because not all failed tests could be determined.Tests failed to start, crashed or the test accounting in the PyTorch EasyBlock needs updating! | |
| Missing (2): inductor/test_flex_attention, inductor/test_flex_decoding | |
| You can check the test failures (in the log) manually and if they are harmless, use --ignore-test-failure to make the test step pass. | |
| 2 test failures, 0 test errors (out of 261591): | |
| Failed tests (suites/files): | |
| functorch/test_ops (2 failed, 7489 passed, 2731 skipped, 0 errors) | |
| Could not count failed tests for the following test suites/files: | |
| inductor/test_flex_attention (Undetected or did not run properly) | |
| inductor/test_flex_decoding (Undetected or did not run properly) (at easybuild/framework/easyblock.py:2578 in report_test_failure) | |
| Callstack: | |
| easybuild/framework/easyblock.py:2578 in report_test_failure | |
| easybuild/framework/easyblock.py:3077 in _test_step | |
| easybuild/framework/easyblock.py:4788 in run_step | |
| easybuild/framework/easyblock.py:4947 in run_all_steps | |
| easybuild/framework/easyblock.py:5135 in build_and_install_one | |
| easybuild/main.py:180 in build_and_install_software | |
| easybuild/main.py:614 in process_eb_args | |
| easybuild/main.py:798 in main | |
| easybuild/main.py:847 in main_with_hooks | |
| easybuild/main.py:862 in <module> | |
| == 2026-02-16 21:28:58,142 build_log.py:329 INFO ... (took 62 hours 49 mins 33 secs) | |
| == 2026-02-16 21:28:58,142 filetools.py:2172 INFO Removing lock /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock... | |
| == 2026-02-16 21:28:58,148 filetools.py:423 INFO Path /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock successfully removed. | |
| == 2026-02-16 21:28:58,148 filetools.py:2176 INFO Lock removed: /data/horse/ws/s3248973-EasyBuild/easybuild-rome/software/.locks/_data_horse_ws_s3248973-EasyBuild_easybuild-rome_software_PyTorch_2.9.1-foss-2024a.lock | |
| == 2026-02-16 21:28:58,148 easyblock.py:398 INFO Closing log for application name PyTorch version 2.9.1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment