sarathi-lean/sarathi/cache_ops.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c1021throwNullDataPtrErrorEv

[zhu1365377615]

sarathi-lean/sarathi/cache_ops.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c1021throwNullDataPtrErrorEv

[ramyaprabhu-alt]

can you please run python setup.py install/develop once under sarathi-lean folder and then check if this issue persists?

[dingzhiqiang]

i also meet this error, and already run python setup.py install, python setup.py develop@ramyaprabhu-alt

[zhu1365377615]

@ramyaprabhu-alt after run python setup.py install，meet a new problem：
Traceback (most recent call last):
File "/export/zsk/vattention/sarathi-lean/sarathi/benchmark/main.py", line 1, in
import logging
File "/export/anaconda3/envs/va/lib/python3.10/logging/init.py", line 26, in
import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
File "/export/anaconda3/envs/va/lib/python3.10/re.py", line 124, in
import enum
File "/export/anaconda3/envs/va/lib/python3.10/enum.py", line 2, in
from types import MappingProxyType, DynamicClassAttribute
File "/export/zsk/vattention/sarathi-lean/sarathi/benchmark/types/init.py", line 1, in
from typing import Dict, List, Tuple
File "/export/anaconda3/envs/va/lib/python3.10/typing.py", line 25, in
import contextlib
File "/export/anaconda3/envs/va/lib/python3.10/contextlib.py", line 6, in
from functools import wraps
File "/export/anaconda3/envs/va/lib/python3.10/functools.py", line 22, in
from types import GenericAlias
ImportError: cannot import name 'GenericAlias' from partially initialized module 'types' (most likely due to a circular import) (/export/zsk/vattention/sarathi-lean/sarathi/benchmark/types/init.py)

after i rename the types dir，then meet a new problem：
Traceback (most recent call last):
File "/export/zsk/vattention/sarathi-lean/sarathi/benchmark/main.py", line 3, in
from sarathi.benchmark.benchmark_runner import BenchmarkRunnerLauncher
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/init.py", line 5, in
from sarathi.engine.arg_utils import EngineArgs
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/engine/arg_utils.py", line 5, in
from sarathi.model_executor.attention import AttentionBackend
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/model_executor/init.py", line 1, in
from sarathi.model_executor.model_loader import get_model
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/model_executor/model_loader.py", line 11, in
from sarathi.model_executor.models import * # pylint: disable=wildcard-import
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/model_executor/models/init.py", line 1, in
from sarathi.model_executor.models.falcon import FalconForCausalLM
File "/export/anaconda3/envs/va/lib/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/sarathi/model_executor/models/falcon.py", line 41, in
from sarathi.model_executor.parallel_utils.pipeline_parallel.mappings import recv, send
ModuleNotFoundError: No module named 'sarathi.model_executor.parallel_utils.pipeline_parallel'

[JasonHe-WQ]

python setup.py install/develop

The same problem occurs. And after using python setup.py develop, it seems to be ok.
Also, changed flash infer to 0.1.0

root@hfan-NUC9:~/vattention/sarathi-lean# pip list | grep flash
flash-attn                2.5.9.post1
flashinfer                0.1.0+cu121torch2.3

[zhu1365377615]

after using python setup.py develop, it seems to be ok. but when i run "python scripts/benchmark_e2e_static_trace.py --test" to test, this problem occurs : ModuleNotFoundError: No module named 'sarathi'

[JasonHe-WQ]

after using python setup.py develop, it seems to be ok. but when i run "python scripts/benchmark_e2e_static_trace.py --test" to test, this problem occurs : ModuleNotFoundError: No module named 'sarathi'

Neither can I

root@hfan-NUC9:~/vattention# python scripts/benchmark_e2e_static_trace.py --test
Running command: python /root/vattention/sarathi-lean/sarathi/benchmark/main.py --model_name 01-ai/Yi-6B-200k --model_tensor_parallel_degree 1 --request_generator_provider synthetic --synthetic_request_generator_length_provider uniform --synthetic_request_generator_interval_provider static --uniform_request_length_generator_max_tokens 32768 --uniform_request_length_generator_min_tokens 32768 --uniform_request_length_generator_prefill_to_decode_ratio 500 --replica_scheduler_provider vllm --trace_request_length_generator_prefill_scale_factor 1 --trace_request_length_generator_decode_scale_factor 1 --replica_scheduler_max_batch_size 16 --vllm_scheduler_max_tokens_in_batch 32768 --model_max_model_len 32768 --metrics_store_enable_op_level_metrics false --metrics_store_keep_individual_batch_metrics true --output_dir /root/vattention/experiments/e2e_static_eval/model_yi-6b_tp_1_attn_fa_vattn_2mb_cl_32768_pd_500_reqs_5/ --synthetic_request_generator_num_requests 5 --trace_request_generator_max_tokens 32768 --model_block_size 2097152 --model_attention_backend fa_vattn --gpu_memory_utilization 0.9
Traceback (most recent call last):
  File "/root/vattention/sarathi-lean/sarathi/benchmark/main.py", line 1, in <module>
    import logging
  File "/usr/lib/python3.10/logging/__init__.py", line 26, in <module>
    import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
  File "/usr/lib/python3.10/re.py", line 124, in <module>
    import enum
  File "/usr/lib/python3.10/enum.py", line 2, in <module>
    from types import MappingProxyType, DynamicClassAttribute
  File "/root/vattention/sarathi-lean/sarathi/benchmark/types/__init__.py", line 1, in <module>
    from typing import Dict, List, Tuple
  File "/usr/lib/python3.10/typing.py", line 25, in <module>
    import contextlib
  File "/usr/lib/python3.10/contextlib.py", line 6, in <module>
    from functools import wraps
  File "/usr/lib/python3.10/functools.py", line 22, in <module>
    from types import GenericAlias
ImportError: cannot import name 'GenericAlias' from partially initialized module 'types' (most likely due to a circular import) (/root/vattention/sarathi-lean/sarathi/benchmark/types/__init__.py)
Error in sys.excepthook:

[ramyaprabhu-alt]

"pip install" doesn't install Sarathi properly. The "python setup.py install" does. But if you ran pip install first, then that installed package takes priority during module loading. Deleting "[USER]/python3.10/site-packages/sarathi-0.1.7-py3.10-linux-x86_64.egg/" allowed the python setup.py version to be loaded

[J1shen]

face the same problem, is Okay to use
'python setup.py install && python setup.py develop'?

[AKKamath]

Yes. Please avoid pip install and use python setup.py install instead.