Error with ModalReranker when running sqa.py

[eranhirs]

I get the following error when running sqa.py:

NotFoundError(\nmodal.exception.NotFoundError: Lookup failed for Function            
                             \'inference_api\' from the \'ai2-scholar-qa\' app: App \'ai2-scholar-qa\'            
                             not found in environment \'main\'.\n'))

https://github.com/allenai/agent-baselines/blob/main/agent_baselines/solvers/sqa/sqa.py#L142

My goal is to reproduce the outputs and evaluations of the scholarqa baseline. I successfully run scholarqa from its own repo, but then I'm missing the evaluation part.

[amanpreet692]

Hello, thanks for using our evaluation and pointing this out.
You were correct to try and run scholarqa from its own repo. A similar implementation is required here as well.
ScholarQA requires to rerun reranker to select the top 50 from the retrieved result. The implementation and params provided in sqa.py assumes the reranker hosted on modal and provides sample params.

To run and evaluate scholarqa, you can replace the reranker object on Line 141 with your own hosted reranker based on instructions from scholarqa lib readme as follows:

Replace the following

reranker = ModalReranker(
        app_name="ai2-scholar-qa",
        api_name="inference_api",
        batch_size=256,
        gen_options=dict(),
    )

with

#In case of reranker model hosted on modal.com GPUs
reranker = ModalReranker(
        app_name=<app name>,
        api_name=<function name>,
        batch_size=<batch size>,
        gen_options=dict(),
    )

# or in case of locally hosted GPUs, pick one of the base classes from 
#https://github.com/allenai/ai2-scholarqa-lib/blob/main/api/scholarqa/rag/reranker/reranker_base.py, eg.
reranker = CrossEncoderScores(model_name_or_path="mixedbread-ai/mxbai-rerank-large-v1")

We will add this change to agent-baselines along with instructions soon as well, to enable initializing the reranker object dynamically.