Labour should "seriously consider" ditching Gordon Brown following the Henley by-election drubbing, the party's former chief fundraiser Lord Levy said.

The Prime Minister was given an electoral mauling in Henley on Thursday, with Labour finishing fifth - behind the Greens and BNP - and losing its deposit.

The seat, vacated by London mayor Boris Johnson, was held by the Conservatives and David Cameron said the emphatic win showed his party was a credible alternative government.

In an interview for BBC2's Newsnight, Lord Levy said it was for Labour Party members to decide whether Mr Brown should be sacked.

But, he added: "I certainly, seeing the polls, would have to say that this is something that needs to be very seriously considered."

Mr Brown, on a visit to Manchester to launch his new public services reform programme, said: "By-elections come and by-elections go."

He added: "Of course we have to listen to what people say.

"But my main job is to improve our public services, to get the economy moving forward, to make sure that in the health service and education people have the best services that they want and I am going to continue to do that.

"And I think people know that we are going through difficult times in the economy. It's my job to steer us through these difficult times.

"And people facing higher petrol bills, higher gas and electricity bills, people facing high food prices, it's my job to make sure I can do more to help people's standard of living improve."

Mr Cameron said the by-election was "disastrous" for Labour and he said he was heartened that, for the first by-election in a long time, the Tory candidate also picked up votes from the Liberal Democrats.

Speaking outside his London home, Mr Cameron said: "To have a further swing to Conservatives in such a strong Conservative seat, I really think is a very good result.

"It is obviously a disastrous result for the Labour Party, but I think I am right in saying it is the first time in a long time when there has been a contest between Conservatives and Liberal Democrats when there has been a swing to the Conservatives.

"I think what we are seeing is that people who voted for all sorts of different parties - including the Liberal Democrats - are now looking at the Conservatives and saying 'Yes, this is an alternative to the Government that I can believe in'."

As expected, Conservative candidate John Howell took the Oxfordshire seat by a comfortable margin of 10,116 in yesterday's by-election.

While Labour had no hope whatsoever in winning in what is natural Conservative territory, party strategists will be alarmed at the dramatic collapse in its vote in a seat where it finished third in the last general election.

Mr Howell took the seat with 19,796 votes to the Lib Dem candidate Stephen Kearney's 9,680.

Labour's Richard McKenzie could only poll 1,066 votes, behind the Green Party's Mark Stevenson on 1,321 and the BNP's Timothy Rait on 1,243.

There was further bad news for Mr Brown with a YouGov poll for The Daily Telegraph showing almost two-out-of-three voters now believe he is a "liability".

The poll to mark Mr Brown's first year as Prime Minister showed a sharp fall off in his personal rating over the past 12 months.

When he entered No 10, almost half - 48% - believed he was an asset to the Labour Party. That figure has now slumped to 21%.

Now 61% believe that he is a liability to the party as against 21% a year ago.

Last year, 62% thought Labour would win the next general election, but that has dropped to 16% while 67% now think that the Conservatives are on course for victory.

And a BBC survey of 135 Labour constituency party chairs showed more than half believed Labour was too close to big business and wanted Mr Brown to forge stronger links with the trades unions.

Some 70 chairmen and women agreed that "since 1997 the Labour Government has moved too close to big business and too far from the trades unions and Gordon Brown should reverse this trend".

And 89 disagreed with encouraging more private involvement in health and education.

Olympics Minister Tessa Jowell said a widely rumoured Cabinet reshuffle was a matter for the Prime Minister, but added that prominent Blairites currently on the backbenches - including former health secretary Alan Milburn - were "undoubtedly people of great talent".

She told BBC Radio 4's The World at One: "I think that it's been a very tough first year.

"The point at which the opinion polls went against us coincides almost exactly with families feeling the impact of the credit crunch - the price of fuel going up, the price of the weekly shop going up and the anxiety that accompanies that."

YouGov interviewed 2,163 electors across Britain online between June 23 and 25.