Cat Power: 'I'm your worst nightmare – get your dancing shoes on'

For an artist renowned for her madness and sadness, it's surprising just how downright saucy Chan Marshall is. "You're using your teeth!" she crows when I suck a wedge of post-tequila shot lime. "I'm jus' kiddin' – you're suckin' – I'm just kiddin'."

It's late and we're in a fabulously tacky Miami Beach bar, the kind where the umbrella-bedecked drinks are wearing more clothes than the clientele. This night began when, hours after I had interviewed Marshall in her beachside condo, I answered my phone to a hammy cockney voice talking nonsense. When I asked who it was she'd answered: "Your worst nightmare – get your dancing shoes on."

As far as drinking companions go, Marshall is the opposite of a nightmare – she buys rounds, proffers cigarettes and charms and disarms a swelling crowd of strangers with her irrepressible shimmying around the dancefloor.

Tonight, she's the life and soul, but the more common image of her is as tortured singer-songwriter Cat Power, an emotionally intense, psychologically fragile artist who for the past 17 years has bewitched fans with a voice that prompted the New Yorker, in 2007, to deem her "a conjurer" worthy of comparison with Patti Smith and Nina Simone.

Back in 2005, at university, I walked into a stairwell flooded with the sound of 1998's Moon Pix, Marshall's fourth album and the one that ensnared a legion of ardent fans. Her voice stopped me in my tracks. The body of that album was written during one night spent alone, hallucinating in a South Carolina farmhouse, and it sounds like it – a keening, sad, dark, confused nightmare of a record, as gorgeous as it is unsettling. The friend playing it at full volume was, like seemingly every other boy in my year, hopelessly in love with her.

But it's a fallacy that the best music always comes out of anguish. Six years after her last collection of original material, Marshall is finally about to release a new album, and the shock is not just that Sun is her best yet, but that it is also the most jubilant.

When we first meet, we sit on the roof terrace of her Miami Beach condo. Or, rather, I sit, while, like a little girl arranging a play house, she fusses happily with a rug and table for our drinks, narrating all the while in a breathless logorrhoea. She seems childlike now, but on the album, which she produced and on which she plays every note of every instrument, she sounds like an artist in supreme control of her gifts.

Finally settled with tequila and cigarettes, I tell her this. She takes a drag and slowly, grimly shakes her head. "Thank God," she exhales. "It was very trying. It was totally a challenge."

But she admits that she's done much less "shaking hands with the dark side" on this one. Marshall has always used the word "triumphant" to describe her music, which, hearing the raw sorrow of her first three albums, seemed ridiculous. Now she's made an album that fits the description. Even the most plaintively titled track, Always On My Own, has at its heart an affirmative "I want to live my way of living" attitude. That sentiment is magnified on the album's penultimate song, a joyfully unspooling 11 minutes titled Nothing But Time, on which she exults: "It's up to you/ To be a superhero/ It's up to you/ To be like nobody." There are shouts of "they wanna live!" in the background before Iggy Pop joins her in the refrain: "The world is just beginning."

When she talks about her triumphs, her words come in staccato, elliptical bursts, and it can be a struggle to keep up. "[I have been] empowering myself to realise that no other human, no matter who it is, can invalidate my way of thinking. I'm triumphing over all the different ways in my life I've been invalidated and I can still smile and laugh and know that in my heart I'm making the right choices for myself. Maybe somebody else doesn't agree but that's not my problem."

And then she stops and a child-like smile creeps on to her face. Why is she laughing?

"Because I don't know what I'm saying! We're born and we die alone, I guess. So you have to take care of yourself."

Is she better at doing that now?

"I'm better at understanding that I have to. I always thought I was going to pass away or whatever. I used to wish I had a lobotomy. But I don't want to pass away any more. I want to keep [she drops her head and claps her hands twice above her in the air, smartly, like she's sealing a spell] living."

She once said that all creativity comes from a place where we don't have love and she's standing by that.

"Because," she says, upping her southern twang, "if we had love we'd be busy fucken'!"

I wonder whether not having love is a perverse sort of blessing, in terms of yielding songs such as these.

She frowns. "You crazy? Love is much better than creativity!"

So if she had to choose between never writing a song again and…

"Love!" she cries before I can finish. "You kiddin' me?"

Most of Sun was made while Marshall was with actor Giovanni Ribisi. He ended their relationship of three years in March this year with a phone call. Four months later, he married model Agyness Deyn. At one point, she says "my boyfriend" and corrects herself to "my ex". Later, she begins: "Since the break-up…", but is characteristically derailed by another thought. Marshall had moved to LA in 2009 to be with him and his teenage daughter, with whom she's particularly close, and in interviews conducted during this time she'd talked about how much she was enjoying being a mother.

Her relationship with her own mother has been volatile. Charlyn Marie Marshall was born in Atlanta, Georgia in 1972 to Myra Lee and Charlie Marshall, a blues musician. She and her younger sister, Miranda, had an unstable and itinerant childhood, suffering their parents' alcoholism, mental health problems and divorce. Marshall finally settled back in Atlanta, where she worked in a pizza parlour called Fellini's for three years and gained local fame as the hottie behind the counter.

In 1992, she moved to New York with her friend, the late musician Glen Thrasher, through whom she experienced the more avant garde edges of the city's rock scene. Two years later, she recorded 20 songs in a single day, and they formed her first two albums, 1995's Dear Sir and 1996's Myra Lee. Later in 1996 she signed to Matador and released the acclaimed What Would the Community Think. From there on her stardom was assured, even if her feelings about it were anything but.

"First interview," she explains. "Bam. I didn't want to do it. Didn't know where I was, what was going on. Losing my mind. Thoughts of Satan and all that shit. Control and oppression. Secrets, lies, blah blah blah." She remembers hiding in a hotel room bed while a 17-year-old fanzine editor waited anxiously to talk to her.

"Eventually, I just say [to the fan], 'I just wanna fucking kill myself!' And then she turns to me, and she's full of anger and sobbing, and she says, 'If you give up, then I give up'. When she told me that, it was a transference of self… I just had to help her."

She hasn't always dealt with her fans' emotional hunger so simply. Her notoriously irregular live shows reached their nadir in a well-documented performance at Manhattan's Bowery Ballroom in 1999. Prostrate and foetal on the floor of the auditorium, she sang the dirge-like Cross Bones Style with her nose pressed into the ground, while fans gathered round her awkwardly, stroking her and murmuring consolations.

She admits that it's taken her years even to believe in her fans' existence, let alone their love for her. "It's out of fear of rejection. Because I went to so many different schools and I was always the outsider. You know, the new kid who doesn't have friends."

The song I Don't Blame You, from 2003's You Are Free, has been taken as a rejection of fans' demands, specifically the lines: "They never owned it/ And you never owed it to them anyway." Online message boards have been filled with speculation over who it's about – Marshall herself or some other fame-troubled singer?

"I've never told anybody this," she says, "but that is about Kurt Cobain. It's about him blowing his head off."

That song is not her only moment of Cobain kinship. Hate takes the title of a Nirvana track for its refrain – "I hate myself and I want to die" – and it's the bleakest moment on 2006's The Greatest, her soul-soaked and languorous seventh album. Recorded with the feted Memphis Rhythm Band, it brims with a resigned, hard-won peace. And yet weeks before the record's release, and with her label preparing for her stardom to go stratospheric, she suffered a psychotic breakdown and was hospitalised. She's been on medication for bipolar disorder since then.

All of which somehow makes improbable her confession that she would like to be a comedian, specifically a performer on Saturday Night Live. "I've been trying for years," she says. "Kristen Wiig's gone now so… I'll just have to lose a bunch of weight. She got so skinny."

The next night, I get a text message: "Shelborne Hotel NOW!!! KARAOKE!!!!" Once again, Marshall orders rounds and rounds of tequila and sodas, pushing them up and down the bar towards people while chain-smoking from a packet stashed in a cowboyish black leather holster.

Hours later, the party moves to beside her pool and some of that southern hospitality kicks in as she bustles around lending swimsuits and assembling a tray of drinks and snacks. At some point – bottles of tequila drunk, joints smoked – her iPod speakers are casting Nina Simone's Wild is the Wind, a song that Marshall has covered, out into the sultry Miami night.

"For we're creatures of the wind," Simone sings, "and wild is the wind," and then, by some uncanny magic, the palm trees begin to stir and the night erupts into a rainstorm. Rods of water begin hammering the surface of the pool, but Marshall can't be persuaded indoors. She remains sitting between those palm trees in the turquoise pool light, cross-legged, rocking back and forth, smoking in the rain.