What I have used was a wide blade 1" chisel and a hammer but not striking too hard. When that EV gasket sticks it will tear.
That butter knife may help too. I have assembled speakers with rubber sealing tap. After that removal I will never do that again. I wonder of 'Saran wrap' would work on the baffle board, place a layer down over sized, cut out the hole in the center, mount speaker and then remove excess wrap. I have not tried this but may prevent the hard to remove speaker when you want to change or replace it (keeping that paper gasket intact). In a 412 cab, rear mounting is the only method due to the grill clearance you have between baffle and grill (depends on what speaker you are mounting too).