NVIDIA Instant Replay causing frame drops and stutters

Typically I see around 300fps in this game, however when I enable Instant Replay I see frames drop to between 50 and 200 with lots of stuttering.

Specs:
4070ti
32gb ddr5 6600
Ryzen 7700X