How would we like to be accused as Moses was, then witness God Himself make a dramatic entrance and hear His voice boom out in poetry in our defense, saying that we are without peer amongst all the people? God says to Moses, "There is no one like you." He was without peer among the holy. That is pretty impressive! It has not happened very often in mankind's history.
But, on the other hand, there has only been one Moses. There were a number of ordinary prophets, who had to be content with visions and dreams, but God spoke to Moses personally. Moses was in a class by himself. Nobody on earth was more intimate with God than Moses, and, as a result, Moses was entrusted with God's estate. And Hebrews 3:2 comments, "Moses also was faithful in all His house."
"All His house" is a figure of speech, indicating that "house" is put for itself (that is, the building) and everything in it. What is normally in a house is a family. Moses, then, was faithful—he was without peer—in all of God's Household, God's Family.
Nobody was faithful like Moses was faithful, therefore he could interpret God's will to Israel with full authority. God backed His prophet up, saying that Miriam and Aaron were completely out of line. This is why He says, "Why were you not afraid to speak against [or, accuse] My servant Moses?"
It is clear what set Moses apart from others: He was faithful. This can be seen when he is contrasted to the rest of Israel, the very people that he was leading, who comprised God's Family at that time. They were anything but faithful! In fact, the reason that the Israelites failed was because of their lack of faith. And without faith, of course, one cannot be faithful.