I don’t understand how any father could say such a thing. Was he drunk or on drugs? That makes no sense to me.
You ask me if you should do something. Yes, I think you should. I think you need to discuss this with your mother and your grandparents and anyone else you can think of who has an interest in your father’s mental health because that doesn’t sound normal at all.
I can’t tell if you are frightened or just disappointed. If you are frightened, tell people and ask for help.
Please talk to adults who know you and your dad. If you think you and your brother are unsafe around him, insist you be allowed to stay with other family or friends.
You need to take care of yourself.