Romantic novel Captain Corelli's Mandolin may have put Louis de Bernieres in the bestseller lists, but it's been difficult for the author to move on. "I'm just waiting for the refreshing time when people don't mention Captain Corelli's Mandolin when they are interviewing me," he says. As you see, he's not quite there yet.

He took 10 years to write another novel after Corelli, which was made into a film starring Nicolas Cage. He said: "After Captain Corelli, I got a fit of writer's stroppiness. I felt I had the world looking over my shoulder and it was irritating.

"I was all that time working on the next novel, Birds Without Wings, but never very flat out. I finally wrote most of it in the few months when I was expecting my first child, because I knew that after he was born I wouldn't get much time."

Eventually, he ignored the pressure to produce another bestseller and went his own way. "You tell me what to do and I'll do the opposite," he says. "It's just a character flaw."

His latest novel, A Partisan's Daughter, is an unusual love story about Chris, a middle-aged salesman trapped in a loveless, sexless marriage in the 1970s, who one night invites a prostitute into his car.

Roza is Yugoslav, recently moved to London, and the daughter of one of Tito's partisans. She proceeds to tell him tales from her life; of danger, romance and tragedy. You never really know if she's telling the truth, but Chris is mesmerised anyway.

Another character in the book is 'Bob Dylan Upstairs', who lives in Roza's building. "Bob Dylan Upstairs was me in 1979. I was living in exactly that house with pretty much that woman, who used to collar me and tell me her stories. She was very fascinating and quite scary. I wrote all her stories down. I never really found a way of turning it into a good novel, because it was just one thing after another. Finally I came up with a plot in the form of Chris."

The novel is set in the winter of discontent, when the country was in the grip of strikes, power cuts and misery.

"I remember that time very vividly. It was the most depressing period I've ever lived through. I hated being young then. All of the optimism that had come along with the hippie revolution had just completely disappeared with the hippies.

"Punk had arrived. I thought it was gross and vulgar. I loathed the music, although funnily enough I like it a bit more now."

He grew up in Surrey, boarded at prep and public schools and signed up with the Army as a teenager, but only lasted four months at Sandhurst. He couldn't stand people telling him what to do and didn't like telling people what to do either, he says.

Instead, he studied philosophy at Manchester University, then taught in Colombia, was a cowboy in Argentina. Back in England his jobs included being a car mechanic and landscape gardener.

South America inspired his first three novels. Then his fourth, Captain Corelli's Mandolin, put him on the bestsellers list.

Now 53, he lives in a Georgian rectory in Norfolk with his partner, the actress and director Cathy Gill, and their two children, Robin, aged three, and four-week-old Sophie. Captain Corelli did not make him an instant millionaire, he says.

"Because the book became successful very slowly, the money arrived slowly. Consequently, I didn't have a time when I suddenly went barmy.

"One of the things that money and success can do is make you very paranoid, because you suspect that people are after you, not because they like you, but because they want something out of you.

"I'm glad I didn't have to cope with that. Nobody recognises me, thankfully. Nobody recognises writers unless you're Salman Rushdie. That's not the kind of recognition I want."

He always preferred to stay out of the limelight, even when Captain Corelli was at its peak. "I'm happy to be well known but I don't want to be a celebrity. I absolutely determined not to mess up my life by doing that. And anyway, I'm not young, beautiful and slim."

Away from writing, he collects instruments and indeed plays quite a few of them, including the flute, the guitar and his beloved mandolin.

While other writers are incredibly disciplined, setting themselves targets and working a particular number of hours a day, de Bernieres simply writes when he feels like it.

"I don't get writer's block because I never try to write when I don't feel like it. I do have other things to keep me contented.

"I have got far too many clarinets and flutes and guitars and mandolins. I play them all and every now and then I do concerts with friends of mine - maybe one or two a month."

His two children happily also take up a lot of his time, he says. Coming into fatherhood late has been a joy. "The overwhelming sensations of love that go through you make you much happier," he reflects. "If I'd had children when I was younger, I probably wouldn't have been such a good father. I would have resented the intrusion much more. "Fatherhood has changed my life. I'm now permanently horrified as to whether I can afford to carry on living," he laughs.

"It's been very good for me emotionally, but it hasn't been very good for my writing. I wrote A Partisan's Daughter in the garden shed."

Despite distractions, he currently has three books on the go. Two are collections of short stories and he's started his next epic.He still has ambitions, but they are quirky ones, he admits. "In the past, my huge ambition has been to do as near as I can get to something as good as War And Peace. Captain Corelli was one attempt which fell a bit short. Birds Without Wings was much more like it.

"I have ambitions like wanting to become good at the classical guitar. I don't really want to be Prime Minister."

While he had mixed feelings about the film version of Captain Corelli, a bittersweet love story set during the wartime occupation of the Greek island Cephallonia, he hasn't ruled out his other novels being made into movies.

"I thought Nicolas Cage was miscast in the role, but I was mostly annoyed by naff changes to the story, which is entirely the director's and producer's fault. I was also puzzled as to why there was no mandolin music.

"If there was a film made of another of my books, it might be a triumph, it might not. But I'd quite like to do the soundtrack myself."

A Partisan's Daughter is published by Harvill Secker at £16.99. The author is at the Oxford Literary Festival on April 5.