I think at the time Angel had very good reasons for leaving but once his outlook changed he should've come back to Buffy a lot sooner than End of Days/Chosen.
Buffy was his link to humanity. Buffy was his life's mission. Buffy was his salvation. Buffy was who the Powers That Be guided into his life. Buffy was his happiness.
Buffy was his first love. Buffy was his first real girlfriend. Buffy was the only person he had yet to truly love. Buffy was the hope he held on to for 100 years as he suffered and fought in Hell.
That is a lot to put onto one person and one relationship.
He showed signs throughout BtVS s1-3 that he didn't think he was worthy of her. That he believed it was selfish to put his need and desire for her above everything she deserves and could have without him.
"I did a lot of thinking today. I really can't be around you. Because when I am.... when I am all I can ever think about is how badly I want to kiss you. I'm older than you, and this can't ever....I better go. I should....." --- s1, Angel
"Look, this can't....I just gotta....I gotta walk away from this." -- s1, Angel
"I knew this was gonna happen. You don't know what you're doing. You don't know what you want. Listen, if we date you and I both know one thing's gonna lead to another. I'm just tryin to protect you. This could get outta control. This isn't some fairytale. When I kiss you, you don't wake up from a deep sleep and live happily ever after." -- s2, Reptile Boy
"I just want you to be able to have some kind of normal life. We can never have that, don't you see? It's over. It has to be! It doesn't matter, it doesn't matter what I feel." -- s2, I Only Have Eyes For You
"I've been thinking....about our future. And the more I do, the more I feel like us, you and me being together, is unfair to you. You deserve more. You deserve something outside of demons and darkness. You should be with someone who can take you into the light. Someone who can make love to you...And children...You have no idea how fast it all goes, Buffy. Before you know it, you'll want it all, a normal life. Right, you'll always be a Slayer. But that's all the more reason why you should have a real relationship instead of this freak show. I'm sorry. Buffy, you know how much I love you. It kills me to say this....I'm trying to do what's right here, okay? I'm trying to think with my head instead of my heart." -- s3, Prom
He decided to become a person, reenter the world, because he knew what her life as a Slayer, engulfed within and without by so much darkness, would do to her, and he wanted to keep her heart warm and her light shining, to preserve her humanity as she restored his faith in it.
He never intended to be more than her helper in the fight, guardian angel living in the periphery of her life, ready to be there when she requires. He didn't want to steal her innocence, break her heart, traumatize her, or make her life harder. Angel felt what Spike said to Buffy in Touched, "When I tell you that I love you, it's not because I want you or because I can't have you. It has nothing to do with me. I love what you are, what you do, how you try."
Angel: I'll never hurt her.
First Evil: You were born to hurt her. Have you learned nothing? As long as you are alive....
Angel: Then I'll die.
He thought she was every bit a fairytale princess but that he would always be a beast regardless of how much he wanted to be her Prince charming. She made him feel empathy and love and brought out his personality and awoke parts of his soul he didn't know existed. But it wasn't until the Shanshu prophecy he believed he could transform completely. "I can walk like a man but I'm not one." "Buffy will always be a part of me and that's never going to change. But she's human and I'm....not, and that's also never going to change."
In Beauty & the Beast, the reckless child turned miserable monster just had to find love and be loved in return to break the spell, to give her up to get her. Angel had done that and it wasn't enough. It took becoming a champion and learning how to do what she does to make him worthy.
He became defender of a city, like her.
He became leader of a group, like her.
He patrolled alone at night, like her.
He protected the innocent, like her.
He formed friendships, like her.
He formed a found family, like her.
He became a single parent, like her.
But somewhere along discovering his moral center and realizing his purpose and gaining confidence, he forgot it didn't work with Buffy because he had yet to be this person. A convertible driving, hotel owning, businessman raising a bratty kid alongside friends and comrades.
In AtS he tried to give Cordy everything he denied Buffy. She was the same age, mortal, had the same profession, visions, part demon, learning martial arts, friends with a British watcher, living in LA.
He substituted his literal soul mate for a friend who he saw as a replica of her. That was so disrespectful and messed up. At least when Buffy started to do the same with another souled vampire it was the result of several years of stalking, psychological manipulation, harassment, and sexual assault. And even then she fought it tooth and nail and didn't realize Angel was an option (he left, the ball was in his court) until he showed up ready to be with her again.
You see it throughout BtVS s1, he is more in the world than he has been in decades but he's still in the shadows. Choosing to be part of Buffy's life meant a huge glow up (homeless to hottie) but he still kept their interactions sporadic and formal, albeit flirty. "Angel? I can just see him in a relationship. Hi, honey. You're in grave danger. I'll see you next month."
Buffy accepted him. "I want my life to be with you." "You're the one freaky thing in my freaky world that makes sense." He had yet to accept himself. He delayed revealing he was a vampire. He had to be reassured of her love before confessing his evil past. He flinched when she tried to touch his face while he was bumpy. He was jealous of Xander for getting to hang out with her in daylight.
But he was getting there. He staked Darla. He told her about the Romani tribe and killing his own family. He kissed her in vamp face. He showed his romantic side and how vulnerable he felt. He initiated their ice skating and movie dates. He patrolled with her, slept in bed, kissed, held her hand, trained together, placed a ring on her finger.
There are so many comparisons to Doyle & Harry but there is a huge difference. Doyle thought Harry would live fully as a human in the human world, that he was the entirety of her involvement with demons. But Angel knew Buffy lived half in light and half in darkness. She would always be connected to demons, be part demon herself. He just didn't want her to forgo the best half for him. "When you kiss me I want to die." "When I look into the future, all I see is you. All I want is you " "This is how we're going to spend our nights when I'm 50 and you're the same age you are now."
I think it was Angel causing Buffy pain (Angelus, Acathla, kissing Faith, feeding on her) that tipped the scales. He didn't know how to prevent it and couldn't see a future with her unless he was sure he'd never hurt her again.
It's why he gave up humanity in IWRY but worked hard for the Shanshu. It had to be earned and proven right by a higher source than his yearning to be with her again. He could trust divine forgiveness.