Prior to God's intervention in their lives, when they did not have a relationship with Him, the Galatians (in particular) and the world (in general) were in bondage to and slaves of the Babylonish system, even a worship of demons—"so-called gods" (I Corinthians 8:5).
In the New Testament, there are two Greek words that are translated as "to know"—ginooskein (Strong's number G1097) and eidenai (Strong's number G1492). According to Vincent's Word Studies in the New Testament, ginooskein is knowledge grounded in personal experience or apprehension of external impressions. It is used to describe relationships, even up to the most intimate of relationships—marriage ("And Adam knew Eve his wife; and she conceived"). Eidenai, the word used in Galatians 4:8, is a mental perception in contrast with conjecture or knowledge derived from others.
The Jews at least knew of God and knew about God, but they did not really know God in terms of having a relationship with Him. He revealed Himself to Israel when He brought them out of Egypt and gave them the law, and the knowledge that such a God existed never really passed from all of the generations. After a remnant of the two southern tribes, Judah and Benjamin, returned to Jerusalem from captivity, they restored the proper worship of God and began adhering to the law that He had given to them. Later, various sects (Pharisees, Sadducees, Essenes, etc.) arose and began putting their own spin on the original God-given law. They wanted to make absolutely certain that they would not transgress His law in even the smallest degree, so they would not have to go back into captivity.
What developed was Halakhah, which was loosely based on the Old Covenant but contained ordinances and judgments that are far from God's original ideal. This, in combination with Hellenism, developed into what is now called Judaism. So at the base of all this, the Jews at least know that there is one true God, but their emphasis on Halakhah made them reject Christ when He came as a man. There was at least a "mental perception" (eidenai), even though there was not a real relationship (ginooskein).
The Gentiles, on the other hand, did not even have a concept (eidenai) of the true God. They worshipped and served a wide variety of pagan deities, and in actuality, this worship was inspired by and centered on demons. In the letter to the Galatians, Paul was addressing not only the dangerous slide into Judaism, but also the return to pagan rites inspired by Gnosticism.