It's been my observation that the real limiting factor in making money in photography is that it takes energy and time, both of which eat into our photography time. I nipped into a fast food restaurant for supper (I'd run out of bread) and am working late. I noted the tacky artwork on the wall and thought, I should be displaying my work here - or perhaps at my favourite movie theatre lobby, then I remembered, I already have work at the Glenbow museum gift shop and Trains N' Such and just when the hell am I going to find the time to make prints, matte, frame and deliver pictures, all of that before selling a single one. I quit the farmers market, not because it wasn't successful - it was coming along very nicely, but because it was a hassle and took a lot of time and things weren't getting done around the house and my wife was paying a significant price, both in me not being available, or being tired and grumpy.
Truth is,the biggest limiting factor in being successful as a part time photographer is the limitations in your energy and time and your commitments to others. If you are single and don't have an overly demanding job and a goodly amount of energy and ambition, then the sky is the limit - more power to you.
For those of us with more mortal limits on our capacities, perhaps we shouldn't worry so much about selling. Perhaps we should be looking in other directions for affirmation of our work - through publication and through our website and maybe the occasional show just for the ego of it.