Telling your dad you don't want to see him anymore will no doubt hurt his feelings and might damage your relationship long term so I suggest you think about other ways to deal with the problems you are experiencing. It would be a shame if you later regret your actions, so let's talk about your relationship with your dad and consider a few options.
It takes time for things to settle down after parents separate, and for kids to get used to the new 'normal'. When parents separate and divorce, you're in the process of making the move from one family unit to identifying with two families. It feels sad and strange and it is normal for kids to apportion blame to one parent or the other. But life goes on and everyone is left trying to pick up the pieces.
Your dad is trying to 'buy your affection', but at least he's trying. He hasn't pushed you away so you have to give him some credit for that when you think about him.
I suggest you tell your dad you think it would be a good idea if he keeps spending time with your brother alone because he really enjoys it ... and maybe the two of you could go catch a movie or go bowling or something once a month. You'd rather spend a small amount of time just the two of you instead of with your little brother all the time.
That would give you a chance to have more mature conversations with your dad as you work to address any issues. It would also mean you don't have to be uncomfortable being on the sidelines with your younger brother.
Don't forget to discuss this with your mother first, though. If she's not comfortable with your little brother being alone with your dad, she might want you to tag along. However I'm thinking it sounds like your dad is trustworthy enough, and your family is just passing through that difficult stage of no longer being a single family unit.