Interpreted at the most literal (or cynical) level, public defenders can seen as a subsidy for criminals because society pays for their legal fees. Of course, not everyone who is arrested is guilty of a crime, but if police are doing their job correctly, most people who are arrested will be guilty of something.
To quote a former police officer: "I don't want to put anyone that's innocent in jail. But, I try not to bring anyone into the interview room that's innocent."
Free legal counsel reduces the cost of being arrested, increasing the net "payoff" to a life of crime. In an anarcho-capitalist system (which some of my classmates at GMU think would be just swell) there'd be no public defenders, the accused would pay the cost of legal representation, and in theory this would provide an additional deterrent to criminal activity.
That's all very well and good, as far as it goes. Ideally, the criminal justice system should take criminals off the streets, and also deter potential criminals with the threat of punishment.
But, there's also a very serious negative externality to the justice system when it puts innocent people in jail. This takes several forms: harm to the person imprisoned and their family and friends, but also society as a whole. People in jail become burdens on the taxpayer and produce nothing (except maybe license plates). The justice system loses credibility when it convicts the innocent. Finally, for each person wrongfully imprisoned there is a criminal walking the streets with impunity, out committing more crimes.
Economists are often skeptical of externality justifications for government subsidies. But, I'd submit that the harm of imprisoning innocent people is so great, that money spent on public defenders is well worth it. Even if career criminals benefit slightly as well, that's a tradeoff worth making (80% of people in jail confess anyway, according to the police officer quoted above, so the harm of that cross-subsidy is pretty minimal in my view).
A last tidbit for thought: being convicted in a criminal trial requires evidence "beyond a reasonable doubt" which amounts to 98% or 99% certainty from a jury. There are a little over 2 million people incarcerated in the U.S. currently. If 99% certainty means being wrong 1% of the time, that's 20,000 people sitting in jail for crimes they didn't commit, at the minimum. Combined with the over-confidence effect (people are lousy at estimating confidence intervals around what they are certain of) it's likely to be much more. A tragic situation, and one which might be made even worse without access to public defenders.