Well, I wouldn’t let this get you down – you don’t really want these clients anyway.
As part of your early conversations with the potential clients, you should tell a story, to feel them out, as to how the project should go. IE: do you want something amazing or something run of the mill? Do you want to be happy or merely satisfied or disappointed? It’s important for the builder to understand and respect my design, if you want to be amazed by the result, this is only possible if they are able to ask the questions before building, be a partner in the process and understand your goals for the project.
I do a lot of commodity level residential work – where the results aren’t so important. But when I’m doing higher end, it’s important to have this conversation early and establish yourself as the project manager – because if they don’t let you do the PM work, they won’t get the result they want. You can actually have this conversation with them and establish quality expectations.