If you are going to do it, pay off the car first. I did my auto->manual swap when I still owed on the car. Probably could have had the thing paid off way ahead of time and then done the swap feeling better about my financial situation.
If you’re not going to do the work yourself don’t bother. Like everyone said sell the car and just buy a manual version.
That being said… I love my manual swap and it was worth it. Although it has kinks still that I’m just too lazy to fix.