You don't have to join twitter to follow tweets. If you have a RSS reader you just subscribe to the page of the twitterer.
Oh, don't like the idea of that - that's a bit like stalking someone! There's a reason I keep mine to followers only 
OK, but I find that a little bizarre. It's like inviting people into your house for a party and then saying that they should be prosecuted for breaking and entering.
What I mean is that I use twitter as a social chat medium - the idea of someone watching an RSS feed of the chitchat with various friends and colleagues is a little odd. And I sincerely doubt I say anything that interesting! Just the way I use it I guess

If the idea is to have a conversation, you are not part of it if you're using RSS rather than tweeting.