Do I have to give a deposit back?

I agree. The only time you really have to give the deposit back is if you sold the car to someone else.

The only sketchy part is there was really no time limit set on it. You did say a couple weeks (ie. less than a month IMO) so you are fine. Keep it. His loss.