Certainly a straight guy saying he finds women more attractive than men isn't homophobic--no matter how crudely said--because that's how all straight guys feel. Unless all straight guys are homophobic. But they aren't.
And certainly considering homosexual conduct immoral isn't homophobic. It doesn't show a person fears or feels contempt for gays--even if he finds their behavior immoral.
I must be missing something. Of course I also think you can say racial group A is, on average, smarter than racial group B without being a racist.