When is switching to all business casual bad?

replace the dress shirt with a nice thin sweater or a cashmere long sleeved polo …and remove the jacket when around in the office.

/problem