Frank was born November 19, 1882 in Savannah, Georgia, the son of Captain William H. Spencer and Mary Elizabeth Wilson.
He was educated in Savannah and studied navigation and seamanship at the New York Nautical College in New York City. He served seventeen years at sea before being appointed Master Pilot for the Port of Savannah, a position he held for thirty years. His early association with the sugar trade set his determination in 1915 to survey the upper Savannah River for possible industrial use. With the backing of the Pilots Association, Spencer set out with a tug boat to sound out all portions of the upper Savannah River and he marked the channel which made it navigable for deep draft vessels.. His original survey attracted the interest of the government which made port improvements and stimulated the establishment of new industries along the upper harbor.
In 1935 he was made a member of the first Board of Directors of Armstrong College and served as a member of the local Board of Public Education for eighteen years. He championed the cause of equal education and initiated the old "separate-but-equal" policies that preceded national schools integration. The school board named him honorary president when he retired from it in the 1950s.
Frank was married to Otelia Buckshaw. Their children were: Margaret Rhodes, Adelaide Elizabeth, Frank Wilson Jr, Hugo Johnson, and Jeanne Spencer. He was married second on August 18, 1938 to Lillian Ruth Windau.