The Mayor's House was for many years at the South Litchfield Beach causeway and stoplight, across from Latte Litchfield and a bit north of The McKenzie Beach Motel. I would, in fact, give the location as "Litchfield" rather than "Pawleys Island", but the later designation is creeping steadily northward.
I always had the impression that The Mayor's House was a) "Too Fancy" and b) "Too Seafood" for me (what with the big pink shrimp and the Fried Lobster come-on), so I never got around to eating there despite liking the pier-like aspect of being built over a pond.
Looking for the address of the place did turn up an online review site which was decidedly... mixed. It did confirm my impression that it was probably too rich for my blood. I do wonder what happened to the shrimp though.