Three is a tricky number in relationships, whether it's with family, friends, or co-workers. One person inevitably feels left out as the other two become tighter. It's terrific that you're in touch with your feelings regarding this situation and aren't keeping them bottled up inside of you.
The next step (and the only way to make it better) is to communicate them to your mother and brother. They may have no idea that you're feeling this way. Suffering in silence will only build up tension and resentment inside of you.
Since three is tricky, make an effort to spend time alone with your mom and time alone with your brother. That way you won't always feel like a third wheel and you'll grow closer to each one of them. You won't feel like you're in competition for Mom's attention as siblings so often do.
If you feel this problem is more than you can handle on your own, talk to your mom about the three of you attending family counseling together. It focuses on improving the dynamic within the family unit—exposing dysfunctional patterns of relating and working to correct them. If you've never talked to a therapist about your father's abandonment, this would provide a wonderful opportunity to do that in a safe space. Most certainly, his departure and lack of involvement in your lives is still affecting each one of you in profound ways.
Your mother and brother love you and want the best for you. Please talk with them so the three of you can work to make this better. Take care!