I forget which it was, where Harry insulted a student's mother, that for me, is what broke the camel's back for me.
Malfoy started it by insulting Mrs. Weasley. Harry just responded with his own insult.
Honestly, what's up with the double standarts? Harry got bashed for using Sectumsempra on Draco in HBP, but the fact that Draco nearly use Cruciatus on him is forgotten.
Harry should have been punished for that, but he got away with it.
And Dursleys should be punished for years of child abuse, but they got away with it.
As for the scene in GoF, let's not forget what happened. Harry got entered into a deadly tournament. Nobody seem to believe him. It's like CoS and 'Heir of Slytherin' nonsense once again. Harry assumes that at least his best mate has enough common sense to understand that he wants nothing to do with the tournament. But oblivously three years spent in the same House mean nothing for Ronniekins. Maybe he spent too much time in a company of his pet rat, and got the same issues with loyalty. :)