A lot of philosophy discusses whether people should or should not believe specific propositions based on evidentiary support. Kyle discusses this issue: "Although it acceptable, in certain circumstances, to believe things without adequate evidential support, it is not acceptable to act on those beliefs." Jeffery Archer discusses this issue directly.
Technically, a skeptic never actually believes anything other than the evidence of her senses. If it is possible to believe or disbelieve some proposition, then she holds an epistemic belief about the probability or plausibility of some proposition; she never has a bare epistemic belief about the proposition itself. She does not believe that p; she believes P(p|E) = n. (If she's a statistician then she has a belief about the probability of the probability, but a simple probability is good enough for practical purposes.)
So it is misleading to ask whether we should believe some proposition on the basis of evidence. The better question should be: How should we act given that we know the probability that some proposition is true? In some cases, we can just distribute our actions according to the probabilities (e.g. I can diversify my investment portfolio according to the probabilities that some companies will fail or succeed). But in many cases, we cannot distribute our actions: We must act as if some proposition were definitely true or definitely false. We don't have to "believe" some proposition to act as if it were true; we can be conscious and explicit about separating the choice of action from the evaluation of the probability.
I know, for example, that there's a non-zero, finite probability that I will die in a fiery car crash on my way to work. But I have only two choices: act as if I will definitely die (and stay home) or act as if I will definitely not die (and drive to work). I can't 99.99% drive to work and 0.01% stay home. But I'm not deluding myself that I'm certain I won't die in a car crash, even though I'm acting as if I definitely would not.
Quite a lot of "weird" epistemic cases can be resolved simply by understanding that we don't actually need to believe definite, true/false propositions, but rather that we need to have an accurate understanding of the probability or plausibility of some proposition. The confusion comes from taking a metaphor for literal truth: If the belief that the probability is sufficiently high that some proposition is true, one's actions are indistinguishable in practice from being absolutely certain the proposition is true, and we talk about "believing" the proposition. But there's simply no specific, constant threshold where the belief that the probability of p is indistinguishable from the belief that p. The threshold varies according to the consequences of acting as if p were true or false.