Josep Borrell told the BBC's Lyse Doucet that the war had reached a "dangerous moment".
His remarks come as Russia begins a partial mobilisation and moves to annex four regions of Ukraine.
Mr Putin has faced setbacks on the battlefield, with his forces pushed back by a Ukrainian counter-offensive.
"Certainly it's a dangerous moment because the Russian army has been pushed into a corner, and Putin's reaction - threatening using nuclear arms - it's very bad," Mr Borrell said.
Seven months since Russia's invasion of Ukraine began, analysts agree that President Putin's forces are on the back foot, but he said a "diplomatic solution" must be reached, one that "preserves the sovereignty and territorial integrity of Ukraine".
"Otherwise, we can finish the war, but we will not have peace, and we will have another war," he said.
In a rare address to the nation earlier this week, Mr Putin said his country had "various weapons of destruction" and would "use all the means available to us", adding: "I'm not bluffing."
"When people say it is not a bluff, you have to take them seriously," Mr Borrell said.
In the same speech President Putin announced the call-up of 300,000 Russians who have done compulsory military service, sparking protests and reports of people fleeing the country to avoid being sent to the front line.
It comes after a rapid counter-offensive in which Ukraine says it took more than 8,000 sq km (3,088 sq miles) back from Russian forces.
Now self-styled referendums on joining Russia are being held in four occupied regions. Ukraine has denounced these as annexation attempts, and reported that armed Russian soldiers are going door-to-door collecting votes.
Ukraine has dominated this year's UN General Assembly as this costly war drags on with no clear sense of a way out. Europe's foreign policy chief was surprisingly blunt and visibly pained.
He shared the anxious lament he was hearing everywhere he went. From friends on holiday, to leaders from around the world attending the UN General Assembly this week, they were all asking him when this war would end. "Stop this war, I can't pay my electricity bill," was, he regretted, a common refrain.
Mr Borrell was willing to say in public what many express in private - that Europe and its allies were struggling to control the narrative in this war as Russia spins the view that European sanctions against Russia were to blame for this suffering.
But Moscow's new and worrying threats, including a thinly-veiled nuclear one, are also concentrating minds. Most Western leaders, including Mr Borrell, are still categorical about the need to stay the course in a conflict with many far-reaching consequences, most of all for Ukraine, but many others too.
Mr Borrell dismissed concerns that the EU's arms supplies were running low, and said it must continue providing military support to Ukraine, as well as applying economic sanctions against President Putin and his allies and conducting diplomatic activity.
He admitted that the rising cost of energy prices caused by the conflict was a matter of concern.
"People in my country tell me the price of the gas means we cannot continue working, we cannot continue making my business run," the Spanish politician said, adding he had heard similar concerns from leaders from Africa, South America and Southeast Asia.
Mr Borrell called on President Putin to play his part in reaching a negotiated solution, saying "in order to dance the tango, you need two".
"Everybody who has gone to Moscow, to the Kremlin to talk to Putin, they came back with the same answer, 'I [Putin] have military objectives, and if I don't get these military objectives I will continue the fight.' This is certainly a worrisome direction, but we have to continue to support Ukraine," he said.