This week apple pointed out that with 225 million ipods sold to date they own 73.8% of the market. With this in mind if you divide 225 million by 73.8 you get 3.048 (which would be one percent). If you then multiply that by 100 you 304.8 million portable devices.
While you don't need an ipod (or any other portable device) to listen to a podcast. I know I've given up listening to “Binky and the Whiz” (or whatever your morning show is call) in the morning in favor of pdocasts. Hopefully some of those 305 million (lets round up shall we?) do to.
One last thought if you only have a link to itunes on your site as a subscription option, that means you are potentially closing the door on roughly 80 million listeners.
Now with the ipod touch and nano getting more features and a lower price tag, those numbers are only going to grow.