In 1873 James and Phoebe Sowle opened Sowle's Boarding House beside Westport Harbor next to the Point of Rocks. Affluent Fall River textile industrialists were attracted to the Harbor's cool breezes, sandy beaches, and strip bass fishing. Sowle's Boarding House and Wharf was purchased in 1911 by E. P. Charlton. One wing of the boarding house was added to Charles Macomber's General Store and Stable to form the north end of the Harbor Inn. The wharf became known as Charlton's Wharf.