The story begins by describing the ancestors of Sir Arthur Jermyn, a British nobleman. His great-great-great-grandfather, Sir Wade Jermyn, had been an early explorer of the Congo region, whose books on a mysterious white civilization there had been ridiculed. He had been confined to an asylum in 1765. Lovecraft describes how the Jermyn family has a peculiar physical appearance that began to appear in the children of Wade Jermyn and his mysterious and reclusive wife, whom Wade claimed was Portuguese.
Wade's son, Philip Jermyn, was a sailor that joined the navy after fathering his son, and disappeared from his ship one night as it lay off the Congo coast. Philip's son, Robert Jermyn, was a scientist who made two expeditions into the interior of Africa. He married a daughter of the (fictional) 7th Viscount Brightholme and fathered three sons, one of which, Nevil Jermyn, had a son, Alfred, who was Arthur Jermyn's father. In 1852, Robert Jermyn met with an explorer, Samuel Seaton, who described "a grey city of white apes ruled by a white god". Robert killed the explorer after hearing this, as well as all three of his sons. Nevil Jermyn managed to save his son Alfred before his death.
Alfred Jermyn grew up to inherit his grandfather's title, but abandoned his wife and child to join a circus, where he became fascinated with a gorilla "of lighter colour than the average". He became its trainer, but was killed in Chicago after an incident in which he attacked the gorilla. Arthur Jermyn inherited the family possessions, and moved into Jermyn House with his mother.
Arthur Jermyn is described as having a very unusual appearance, and supposedly the strangest in the line descended from Sir Wade Jermyn. Arthur became a scholar, eventually visiting the Belgian Congo on a research expedition, where he heard tales of a stone city of white apes and the stuffed body of white ape goddess, which had since gone missing. Returning to a trading post, Arthur talks to a Belgian agent who offers to obtain and ship the goddess' body to him. Arthur accepts his offer and returns to England. After a period of several months, the body arrives at Jermyn House. Arthur begins his examination of the mummy, only to run screaming from the room and later commit suicide by dousing himself in oil and setting himself alight.