Playback delay issues with RIFE AI

I installed SVP today and I am trialing it. I have noticed that there is a huge improvement in artifact management with RIFE AI compared to the standard interpolation. But the trade off is ridiculous. While the standard interpolation plays smoothly, RIFE AI takes ~10 seconds to regain normal playback once I seek to a new timestamp. Is this normal? It seems to be the same whether I use vulkan or TensorRT.

I am currently using mpc-hc with an rtx 3070 with the default RIFE model. Is my gpu not powerful enough or are there better/lighter external models I should be looking at?