bug: pods start erroring out after a couple of days of running

## Overview
A problem since a long time now, but if we run the pods for longer timeframes, then after a point the pods for chains go into `CrashLoopBackOff` stopping the chains and making the deployment completely useless. 

## Potential fix
Having persistent volume claims for the pods, allow them to gracefully restart if and when required.