When is switching to all business casual bad?

khakis or dress pants with a polo, nice sweater, or button down shirt at gorman enterprises… not too bad. suits on important days and trips of course