"It's all downhill from here"

What does the phrase mean? Some folks use it to mean that everything hereafter will be easier, as in the hard part of climbing to the peak is past; now we can coast; everything is now better. Others use it to mean that nothing hereafter will be as great as it was at its peak; now everything will be worse.