I'm so sorry this is happening to your granddaughter and your family. Her mother is behaving selfishly and hurting the little girl in the process. The first four years of life are hugely consequential, and she has bonded with you and your son in a profound way.
It makes no difference to her that you're not biologically related; you are her family. Not having all of you in her life feels like a major rejection and can cause enormous heartache, especially if she doesn't understand why it's happening. We can only imagine what her mother is saying to explain your sudden absence from her daily life.
I encourage you to see a lawyer and find out what your rights are in your particular state. I'd fight hard for this little girl and let her know that you're doing so. Sadly, though, our legal system doesn't always put the interest of children first.
That's why it's important to stay in the mother's good graces, even if it's beyond frustrating. Convince this self-absorbed woman that you want to lighten her load by watching your granddaughter. Since she's in a new relationship, she may appreciate having some alone grownup time with her partner. If she's financially in need, she may find it appealing if you take the girl shopping for clothes, toys, and school supplies.
Even if you detest this woman, swallow your pride so you can stay connected to your granddaughter. Hopefully, by appealing to this woman's self-interest, you and your son can have access to the little girl. She'll grow up much happier and more confident if you remain a consistent, loving presence in her life. Best to all of you!