The Inn dates back to the 1800s, when it is believed the building was constructed in the middle of green fields as a manor house for a wealthy ship owner. It is believed to have been called Zion Manor and the cottages behind the pub still bear this name.
The front of the manor had a sloping garden now called The Lawn through which ran a pathway directly down to a quay by what is now the famous Bosuns Locker Chandlery. This is where the ship owner would dock his ships to load and unload. The cargo being carried up the hill to the storage sheds behind his manor house. Over time the manor was divided into two dwellings, Dolphin House and the current pub.
In the Victorian building boom a road was created along the front of the building connecting the Old School House further up the hill and the town outskirts via Wodehouse and Clare Terraces.