The earliest recorded use of the phrase dates back to the 1650s, appearing in a book called 'Adam in Eden'. The relevant sentence reads: "But the common or garden nightshade is not dangerous." Thus, the phrase was initially one about plants/gardening, whereas we now use it in all sorts of circumstances to indicate that the thing being spoken of is in no way out of the ordinary.