Story points can be based on relative size compared to the smallest task.

In this style, the smallest item in the set is assigned a value of 1, and all other estimates are rated with that as the basis—a task roughly 5x as difficult would be set as 5 points, for example. This can obviate the fact that people are bad at estimating time but good at comparison 1A2B.