Mariah Carey’s late sister Alison Carey did not have an easy life – but according to her friend and advocate, she was still able to see her children before she died on August 24th.

David Baker, who knew Alison, 63, for nine years, tells PEOPLE she “had a rough life” before she died at her home in Coxsackie, New York.

“We saw it coming, but it’s still a shock,” he says. “She got sick pretty quickly and a month later she died.”

Baker had previously announced on X on August 3 that Alison had been placed in hospice care. He was the one who helped her find her permanent home in Coxsackie, New York, where she had lived for the past three years. He says that before the move, there was talk of putting her in a nursing home, but he offered to help her if it meant she could continue to live on her own.

“Her wish was granted,” he says. “She died in her own home.”

Alison was a mother of four children – three sons and a daughter – and Baker says her children had visited her two or three weeks ago, including her daughter, who lives nearby and has been visiting daily.

As for her famous sister, Baker says Alison isn’t entirely sure when they last saw each other, but it was either 1994 or 2002. The two had a complicated relationship, and Mariah wrote in her memoir that, at least at that time, it was “emotionally and physically safer for me to have no contact” with her or her brother Morgan.

Alison also did not know that her mother Patricia died on the same day, as Alison died shortly after midnight on August 24th.

“Alison was a highly intelligent, very sensitive person,” he says.

Baker and Alison first crossed paths in 2015, when Alison suffered a severe brain injury during a burglary. She was living on Long Island, New York at the time and was hospitalized there, but was eventually transferred to the upstate. During her rehabilitation, she was hospitalized again after suffering a brain hemorrhage.

At that point, Baker says, her brother Morgan asked for help from a Facebook group he was part of, looking for someone in the area who might be able to visit her.

“When I saw the request, I knew I was the only person near Albany, New York, so I said, ‘Okay, I’ll do that,'” he recalls. “They ended up sending me to the hospital and I met Alison.”

Over the past nine years, Alison and Baker have grown closer and he says “hardly a day went by” when they didn’t see each other in person or speak on the phone.

“I saw someone who needed help and I knew I could do it,” he says. “She told me she was so glad I was there. Recently, before she got sick, whenever I went somewhere for an hour, she would call and ask, ‘When are you coming back?’ And I’d say, ‘Okay, I’ll be there in 20 minutes.’ And then she’d call a second time and ask, ‘When are you coming back?’ Because short-term memory was affected by the brain injury.”

PEOPLE confirmed on Monday, August 26, that Carey’s mother Patricia and sister Alison died on the same day over the weekend.

“I am heartbroken to have lost my mother this past weekend. Unfortunately, in a tragic turn of events, my sister lost her life the same day,” the 55-year-old Grammy winner told PEOPLE in an exclusive statement. “I feel blessed to have been able to spend the last week with my mother before she passed away. I am grateful for everyone’s love and support and respect for my privacy during this impossible time.”