Built in 1760, the Wentworth-Gardner house is one of the finest examples of Georgian architecture in the United States. The house was built as a wedding gift by the Wentworth family for their son Thomas. In 1915, the house was purchased by Wallace Nutting, a photographer and antiquarian. The home was one of five homes that made up Nutting's "Chain of Colonial Picture Houses." Nutting restored the house in the Colonial Revival era and used it as a backdrop in which to photograph models. The Metropolitan Museum of Art purchased the house from Nutting in 1918. The Met prized the house for its detailed carving and Georgian architecture. They had hoped to install the whole structure, or at least some of its carvings, in the museum's new American Wing.
The stock market crash of 1929, and the ensuing Depression put an end to ideas of moving the house to New York City. The Society for the Preservation of New England Antiquities (now Historic New England) furnished the house and briefly operated it as a house museum. In 1940, the house was purchased by a group of preservation-minded local citizens who established the Wentworth-Gardner and Tobias Lear Houses Association.