Transgress in verse 4 means rebellion, not just sin. God considered Israel's syncretistic approach to religion to be an outright rejection of His way of life.
Amos is speaking sarcastically when he suggests that the people sacrifice and tithe more often. "If you bring your tithes every three days (NKJV, AMP) instead of every three years," he says, "maybe your god, Baal, will respond." This sounds somewhat like Elijah's sarcastic comments in I Kings 18:27.
Amos mentions "leaven" in verse 5. Leaven was not allowed to be in any sacrifice: "No grain offering which you bring to the LORD shall be made with leaven, for you shall burn no leaven nor any honey in any offering to the LORD made by fire" (Leviticus 2:11). Only one offering, the wave loaves on Pentecost, was made with leaven (Leviticus 23:17). A sin offering preceded the offering of the wave loaves, the leavening in them representing the sins still in the congregation of Israel.
Here, Amos' sarcasm continues. The Israelites might as well have been making all their sacrifices with leaven because all their traditions, doctrines, customs, and religious duties were nothing but vanity. Even though they were sincere in doing them, they were nevertheless a leaven brought in from the world. In like manner, Jesus tells us to beware of the leaven of the Pharisees and Sadducees (Matthew 16:6-12), that is, of their doctrine and their traditions.
Even a quick glance at modern religious practices reveals how thoughtlessly people accept the doctrines and traditions they have learned—without proving them. Millions of sincere people attend church every week, celebrate the holidays, and send their children to church schools without ever proving their beliefs. They sing in the choir and donate generously when the plate is passed, but they do not really know—have an intimate relationship with—the god they worship. They just blindly accept the leaven they were taught while growing up.