Note the plural pronoun "them." To this time, there was only one covenant, so if he were referring to the covenant, he would have had to say, "for finding fault with it." But God did not find fault with the Old Covenant. Everything that God does is of the highest order, and the covenant He gave to Israel was more than adequate for His intention at the time. It was not the covenant that failed. It was them—the people—who failed. They did not live up to what the covenant stipulated.
Everything God does is pure, right, and true. People who say the law is done away and the Old Covenant was a failure imply that there was something wrong with what God gave the people to do. God does not do things like that! We cannot afford to allow that kind of thinking to get into our minds because it puts us on the trail to error and will not help us in our relationship with God. It will greatly affect the way we approach the Bible—the Word of God.
The Old Covenant is part of the Word of God, and Proverbs 30:5 says, "Every word of God is pure." Therefore, the problem was not with the covenant but was in the people. Specifically, the problem was in their hearts; they were uncircumcised, to use the Bible's term. Their hearts were filled with self-will and therefore rejected what God had to say.