Yes, it’s okay to nix having a relationship with him. A person’s past behavior is the single best indicator of their future behavior. Therefore, you’re using sound judgment and keen instincts in being wary of him.
His neglectful parenting and criminal activity expose him as someone of low character. He didn’t establish a bond when you were young so, quite naturally, you don’t have a strong, loving attachment to him now. It’s sad but normal and to be expected. That’s why the famous quote from Maya Angelou is crucial for fatherless daughters: “If someone shows you who they are, believe them the first time.”
Please don’t let people guilt you into having a relationship with him by saying trite comments such as: “He’s your father! You can’t break up with your own dad!” Just because you share DNA in common doesn’t mean you’re obligated to have a connection with him.
I hear from so many women who’ve hung on for decades to their irresponsible no-good dads. They would be far better off psychologically today if they had let go of him years ago. Their time and effort would have been better spent on those who enriched their lives and inspired them to be better people.
Some fatherless daughters refer to their biological dads as “sperm donors” to remind themselves that there’s no paternal bond there. It helps them keep the relationship in perspective and not pretend that it’s anything more. I wish you well with your decision.