When the realisation suddenly hit, there was no doubt of the need to act quickly. Every day lost would be another day towards inevitable election defeat.
It would be hard to explain to voters why the man they'd been asked to trust was suddenly the very last man who should be allowed to lead the country.
But the alternative -- humiliating failure -- would be even worse. There was no alternative -- the leader must go.
And that's why Iain Duncan Smith should be on Ed Miliband's speed-dial.
There's no question about the Conservatives' capacity for ruthlessness, deposing Margaret Thatcher after three election wins, and throwing IDS on the scrapheap before he could lose one.
But Labour's attempts at regicide have always been more amateurish, half-hearted, and usually doomed to fizzle out after a few hours.
So, while a significant proportion of Labour MPs may feel Mr Miliband is incapable of delivering victory next May, they're almost certainly stuck with him.
The not-so-grand plan
It's no great surprise there is internal despair at Labour's position. Six months from a General Election, their opinion poll lead is evaporating, and there seems little optimism that decline can be reversed.
But why make it public? And why now?
It's often seemed Mr Miliband would be quite happy to see Labour get 35% at the election, and use the inbuilt advantage in the electoral map to carve out a narrow majority.
The only problem with that strategy is it allows so little room for movement. And it assumes the situation in 2015 would be broadly similar to five years earlier.
Mr Miliband has in fact made a series of assumptions. Left-leaning Liberal Democrats, turning away from Nick Clegg in disgust at the Coalition, would return to Labour. The party would stack up its usual bumper haul of seats in Scotland and northern England, and use a handful of gains in the south to crawl over the victory line.
Now suddenly the SNP pose a huge threat in Scotland, UKIP is slicing off a chunk of traditional Labour voters in the north-west and north-east of England, and further south there are signs the Green Party's picking up some of those disgruntled Lib Dems.
Suddenly, Labour's hovering around 30% in the polls -- and its prospects next May look very, very fragile.
It wasn't meant to be like this
Former Bank of England governor Mervyn King predicted whoever won the 2010 election would be forced to impose such pain they would be "out of power for a generation".
Some within Labour assumed the same -- submerging any doubts about Mr Miliband, believing spending cuts would make the Conservatives unelectable.
Now the Tories think there's every chance they can get more votes than Labour next May, even though our electoral system means that wouldn't necessarily equate to more seats.
The sudden reversal of fortune has revived all those old fears about whether they chose the wrong Miliband.
The fact is Ed Miliband does sometimes struggle to connect with ordinary voters, he can look uncomfortable in the kind of situations he'll face every single day while campaigning.
Maybe this shouldn't matter -- but it does.
Voters need to feel confident in the people who seek to lead their country -- how can they have confidence in a man whose own allies are apparently conspiring against him?
But it's simply too late to throw Mr Miliband off the bus, and hope a replacement can sparkle enough magic dust around to secure a win next year.
We'll find out next May whether Labour made a mistake in choosing Ed Miliband. If he finds his way into Downing Street, all this will be forgotten.
For a while at least.