The original part of the house, where the central chimney is located ( Entry Parlor), was built in the early 1700's. In 1820 it was moved to its present location from its original site on Setauket (Conscience) Bay by Dr. John Elderkin.

After the death of Dr. Elderkin in 1836, his son, John, moved from New York City and took up residence in the old home. He added the sections to the right and left of the original house, now referred to as the "Board Room" and the "Victorian Parlors", sometime around 1835. The house then became "Ye Old Elderkin Inn."

Besides running the inn, John continued his drug business, which began in the city, had a general store, post office, bank, and a Franklin Library in the building. During the 1860's, the inn served as a home for a stage coach line which ran between Setauket and the Lakeland Railroad Depot near Ronkonkoma before the north shore Long Island Railroad line was built.

John Elderkin's wife, Renelcha was a woman of great refinement, energy and ambition, and the inn was noted for its hospitality and fine food. Many famous people made this their stopping place. Robert Bonner of the New York Ledger was a frequent visitor and many residents remembered his famous trotting horses, Dexter and Goldsmith Maid.