More of my questions have been marked as favorite but are still down voted to the point of a question ban. Can a handful of people out of thousands down vote a user into a question ban?
"Favorite" is a bad name. It's just a bookmark. Marking a question "favorite" really only means that person wants to be able to get back to it easily in the future. It does not necessarily mean they like it or that it is one of their favorite questions, as the name unfortunately implies.
Votes are the mechanism for passing judgement on a question. Marking a question for easy future reference has nothing to do with that.