I think the Swasbuckler would get a lot more mileage out of the Riposte feature if it was more like Crane Flutter (Monk Feat): if a foe misses an attack against you, you can Strike it with a -2 penalty. If it crit misses, you can Strike it at no penalty.
This also gives players a (minor) tactical choice: do you take your Reaction on a foe's first miss? Or do you wait to see if they attack again with MAP and more chance of a crit miss?