There’s no shame in talking about breasts or how they tend to change as we go through our hormonal and biological changes. Unlike popular beliefs, sagging breasts can happen to both young and older women, with large or small breasts. As we age, fat replaces some of the breast tissue, reducing their elasticity and firmness with time.