Fighter jets were scrambled to intercept the transport planes on Monday after detecting "suspicious" activity over the South China Sea.
Malaysia's foreign ministry described the manoeuvre as a "serious threat to national sovereignty".
China, however, said its aircraft had abided by international law.
Malaysia, the Philippines, Brunei, Taiwan and Vietnam all contest China's claim to almost all of the South China Sea. The row has rumbled on for decades but tensions have increased in recent years.
Beijing continues to claim an area known as the "nine-dash line" and has backed its claim with island-building and patrols, expanding its military presence while maintaining that its intentions are peaceful.
According to the Malaysian air force, the Chinese aircraft were "flying in tactical formation" at up to 27,000 ft (8.2km) and came within 60 nautical miles (110km) of Sarawak, a Malaysian state on the island of Borneo. Attempts to contact them went unheeded, a statement said.
Malaysian fighter jets were then scrambled from the Labuan Air Base to carry out "visual identification".
Foreign Minister Hishammuddin Hussein said the planes had entered the country's "maritime zone" and that a complaint would be lodged with Beijing.
He said the Chinese ambassador was being summoned to explain the "breach of the Malaysian airspace and sovereignty".
"Malaysia's stand is clear - having friendly diplomatic relations with any countries does not mean that we will compromise on our national security," he said in a statement.
Read More (...)