I FINALLY figured it out!! It was due to SVP's "GPU-acceleration OpenCL" feature being 'Enabled'!
I simply disabled it and now I can leave multiple tweaked videos (by SVP) being paused without worrying about GPU's continuous high load!
One last thing to note that this doesn't work if "NVIDIA CUVID" is used as Hardware Decoder in LAV Video options! (also, it makes GPU load high even before the video gets tweaked!).
Good luck fixing those bugs in the upcoming versions of SVP (if not just me of course).
Cheers,
Salaam