You need to pay because you agreed to it period. Even if the seller was “ripping” you off you agreed to it.
Lets say your selling a car and you price it $1000 above average selling value. Someone offers you what your asking and then gives you a $500 deposit. You agree to take your car to his house and he will give you the rest of the money. So you take off a day from work and spend $10 in gas to get there and back. When you get to the guys house he then tries to offer you less for the car then what you previously agreed to.
Your doing the same thing, stop it.