ablation experiment

[zixian2021]

I was curious about the predictive power of tweet sentiment, so I performed an ablation experiment. When I removed the tweet and sentiment from the prompt, the model still achieved 87% prediction accuracy. my prompt looks like "previous label:2 ROC: bullish RSI: bullish ".

[hamidm21]

Hello, and apologies for the delayed response to your issue.

You're absolutely right to highlight this concern, and we can confirm your findings. This observation is precisely why we performed SHAP analysis on multiple samples—to investigate whether the tweet and sentiment features were truly influential in the model's final decision.

Interestingly, as noted in the first paragraph of the second column on page 8 of the paper, even features like ROC, RSI, and the previous label can have varying levels of impact. That said, it's likely that even after removing those elements from the prompt, the model may still retain a relatively high predictive accuracy.

The key issue here lies in the forward-looking nature of the Triple Barrier Labeling (TBL) method, which can introduce a form of lookahead bias—particularly when using the "previous label" as a prompt component. Because TBL defines labels based on future price movements, referencing the "previous label" (i.e., yesterday's TBL-assigned class) might unintentionally leak future information, artificially boosting the model's performance.

A more appropriate alternative would be to use the "last resolved label"—that is, the most recent finalized trend window prior to the current date—to avoid this bias.

Despite these concerns, it's worth noting that the Context-Unaware model still outperforms traditional sentiment-based approaches. Nonetheless, we're currently working to revise the methodology and update the reported results to ensure a fairer, bias-free evaluation.

Thank you again for your thoughtful experiment and for helping improve the robustness of this work.