Yevgeny Prigozhin has vowed to transfer control of the city to the Russian army by 1 June, but Kyiv says it still controls pockets of the city.
He said his forces were ready to return if the Russian regular army proved unable to manage the situation.
The battle for the city has been the longest and bloodiest of the war.
Wagner mercenaries have led the fighting there for the Russian side, and Mr Prigozhin this week said that 20,000 of its fighters had died in Bakhmut.
"We are withdrawing units from Bakhmut today," Mr Prigozhin said in a video released on Telegram from the destroyed city.
BBC Verify has geolocated the video to an area near a pharmacy in the east of Bakhmut.
Mr Prigozhin - who announced the capture of the city on Saturday - is seen telling his men to leave ammunition for the Russian army. He adds that some Wagner fighters will stay behind to assist Russian troops.
"The moment when the military are in a tough situation, they will stand up," he says, before warning two fighters to not "bully the military".
The Wagner boss has repeatedly targeted top Russian military officials, criticising them publicly for not supporting his troops. Last month, he even threatened to pull his troops out of the city if they were not provided with much-needed ammunition.
Despite Wagner's claims to be handing over Bakhmut, Ukraine has not conceded that the city has fallen.
Ukraine's Deputy Defence Minister, Hanna Maliar, said on Thursday that its forces still control part of the Litak district in the southwest of the city.
"The enemy has replaced Wagner units in the suburbs with regular army troops. Inside the town proper, Wagner forces are still present," she posted on Telegram.
Analysts say Bakhmut is of little strategic value to Moscow, but its capture would be a symbolic victory for Russia after the longest battle of the war in Ukraine so far.
Wagner mercenaries have concentrated their efforts on the city for months and their relentless, costly tactic of sending in waves of men seems to have gradually eroded Kyiv's resistance.
Mr Prigozhin has emerged as a key player in Russia's full-scale invasion of Ukraine launched in February 2022, in charge of the private army of mercenaries.
He recruited thousands of convicted criminals from jail for his group - no matter how grave their crimes - as long as they agreed to fight for Wagner in Ukraine.
Around half of the 20,000 Wagner fighters to have died in Bakhmut were convicts, Mr Prigozhin said this week.
Earlier this month, the US said it believed more than 20,000 Russian soldiers had been killed in the battle for Bakhmut and another 80,000 wounded. The BBC is unable to independently verify the figures.
The capture of Bakhmut would bring Russia slightly closer to its goal of controlling the whole of Donetsk region, one of four regions in eastern and southern Ukraine annexed by Russia last September following referendums widely condemned outside Russia as a sham.
However, when Russia fought fiercely to claim the cities of Severodonetsk and Lysychansk last summer, Ukraine soon reclaimed swathes of territory elsewhere.
There were about 70,000 people living in Bakhmut before the invasion, but only a few thousand remain in the devastated city, once best known for its salt and gypsum mines and huge winery.
<p data-chorus-optimize-field="dek">President Zelenskyy visited the G7 and Arab League summits to make Ukraine’s case.</p> <br />