I think when you move away from your religious history, you might have a propensity to move away from people, as well.
I believe that is unfortunate.
In fact, if there is a reason at all to have religion, it is to bring people together (at-one-ment, as it were). I am starting to see a purpose of life having a lot to do with bringing people together.
A church allows you, perhaps in an imperfect, artificial, or superficial way, to have contact with people who you feel are somehow “like you”. That is better than not having any contact at all.
Finding other ways to come together with others is still important.