When is switching to all business casual bad?

I always wear dress pants and dress shirts to work. I cant stand Khakis n’ shit. I would wear what you have.