I think the mistake here is to assume that ethical principles must be justified by one essential principle. In the real world, ethics seems to result from a variety of human intuitions, along with a whole mess of social phenomena. It seems clear to me that we have taboos against killing people for many reasons- many of which do not apply to animals. One might argue that these reasons are flawed, but it seems to me that ethical vegans just insist that one criteria for what makes killing wrong is all that's relevant.
I happen to be more of a utilitarian, and I think carnivory can be justified on utilitarian grounds, even if non-human animals receive the same ethical weight as humans. But I see this as an open question- it really depends on many contingent facts about the world, many of which I don't know for certain. I also realize that most people are not strict utilitarians, and this is relevant for how I choose to live as an moral agent.
Even if you are completely convinced that veganism is a moral imperative, I think you have to contend with the fact that most people disagree with you, and try to really think about the practicalities of enforcing your belief. But maybe this is where I just deeply disagree with ethical vegans, and with all strong deontologists: I think ethics without consideration of many facts in the real world is just pointless.