We tested the hypothesis that large areas of small hard drusen (diameter <63 µm) and intermediate drusen (diameter 63–124 µm) are associated with the incidence of age-related macular degeneration (AMD). Eyes of 3344 older adults with at least two consecutive visits spaced five years apart over a 20-year period were included. A 6-level severity scale, including no drusen, four levels of increasing area (from minimal (<2596 µm²) to large (>9086 µm²)) of only small hard drusen, and intermediate drusen, was used. The five-year incidence of AMD was 3% in eyes at the start of the interval with no, minimal, small, and moderate areas of only small drusen and 5% and 25% for eyes with large area of only small drusen and intermediate drusen, respectively. Compared to eyes with a moderate area of small drusen, the odds ratio (OR) of developing AMD in eyes with a large area of only small drusen was 1.8 (p < 0.001). Compared to eyes with large area of only small drusen, eyes with intermediate drusen had an OR of 5.5 (p < 0.001) of developing AMD. Our results are consistent with our hypothesis that large areas of only small drusen are associated with the incidence of AMD.