It sounds very painful for you to even think of letting go of your kids. The question is, as they grow, at whatever pace, how much will they actually need you. You may need to give yourself small challenges to transition into giving them space, for example, a structured camp for kids or recreational/cultural trips and activities offered by local agencies.
I hope you're working with a counselor, especially regarding your son. Despite his developmental delays, he still needs the chance to be independent in his own time with support and appropriate resources. That means you'll have to grieve the need to protect him and slowly let go. It will take time, be gentle with yourself. Focus on the love and teachings you've already instilled in them and trust that it will be enough.