Pay him. You agreed upon shipping price, and terms of the deal, so pay it.
If they had came damaged I can see where you’d try to re-coop price, but in this case you are wrong.
What he actually pays for shipping vs what you agreed on doesn’t mean you get a refund. I charge 10-15 for shipping everytime on eBay even though I know USPS Priority is $4.10.