People are bad at planning and estimation.
Studies show that people underestimate task difficulty. Estimating relative sizing between tasks rather than time may help. This is the core purpose of story points. As such, time-based estimation may be more the problem.