Nude and alone
No, you're not allowed to interact only with female nudists. Doing so creates obvious problems with the perception and/or reality of your marriage - and it's not going to happen, anyway. You might find a co-ed venue that will accept you without a female companion - certain resorts; beaches of course. But there is a very, very, long tradition of men being naked together - sports, camping, gym, the military. Granted, that's less common these days for a number of reasons, but it's still vastly more likely that you'll find guys to hang naked with than that you'll find women. No, it's not a gay thing. No, it's not a sexual thing.
I live in a large metro area and have been a solo nudist for decades. Joined TN 2 years ago to meet other like-minded people. In that time I have only meet 5 people, 1 of whom I consider a friend. Want to encourage you to keep trying on your efforts. Good luck.