Actual physical time spent is indubitably an important component of labor, but it's not the whole story. If we're going to track labor, we have to track socially necessary abstract labor.
In Starship Troopers*, Heinlein (among others; I'm sure he's simply repeating another's argument) oh-so-profoundly notes that an hour spent making a mud pie is not equivalent to an hour spent making (IIRC) an apple tart. I suppose Heinlein cannot be faulted for this error; the noted philosopher George Santayana admits to the same error himself. But an error it is, an error that Marx avoids in the earliest chapters of Capital, by introducing the notion of socially necessary abstract labor time. Marx does not, I think, adequately define this notion; he relies on our intuition to carry the ontological burden. But he does make clear that he is not simply comparing raw hours to raw hours.
*Ayn Rand fans take heed: If you're going to write an eliminationist manifesto disguised as a work of fiction, complete with wooden characters and an implausible plot, you cannot do much better than Starship Troopers.
Marx explicitly defines labor as time spent producing commodities; if you're not spending your time producing commodities, you're not laboring. But how you spend your time is as important as what you spend your time doing. Our intuition and ordinary observation can supply several more fundamental components of labor time.
First, we have intensity and intrinsic desirability of labor time. One person who slacks off will produce less in the same amount of time as another who hustles. And eight hours in an air-conditioned office sitting in a Herman Miller chair seems intuitively more intrinsically desirable than the same amount of time up to one's crotch in shit cleaning out a sewage system. It's not so easy to actually measure — especially to get a scalar (numerical) measurement rather than just a comparative measurement — these fundamentally subjective considerations.
Second, we have to amortize the time spent acquiring a skill. An hour with a physician does not just consume just an hour of her own time: we must factor in her own years of medical school, internship and residency, as well as the time of her professors, instructors, and trainers to provide her education. And they themselves are skilled professionals, so we amortize their skill acquisition time. It's possible to measure all this time, but it's a decidedly non-trivial task.
Third, we have the organizational context (which underlies the "socially necessary" component of Marx's term): ceteris paribus, one person spending 8 hours using modern equipment in an efficient cooperative environment will produce more than someone using obsolete equipment in an inefficient environment.
Even if we try to use time as some component of what we think money represents, we must still have money represent some output or result of a complicated model. This is not to say that we shouldn't try to measure socially necessary abstract labor time — a quantity that money in a capitalist framework does not effectively measure at all — just that it is not as straightforward a task as directly measuring more-or-less fundamental properties of physical objects such as length, mass, etc.