I was young and my husband of ten years didn't have a casual bone in his body, except he did play golf so he wore polo or "golf" shirts, but once I divorced him I wanted a casual down to earth guy who didn't wear suits every day.  In time I found out that wearing suits was not the problem. 
     In choosing what you do want instead of what you do not want, what you have to do is first analyze yourself.  I happen to be a very introspective analytical kind of a person. I now realize, that what I wanted & needed was a man with a very out going personality. Why? To balance off my extreme shyness. So what you want and what you really want could be two different things.