Danbury was settled by colonists in 1685, when eight families moved from what are now Norwalk and Stamford, Connecticut. In recognition of the wetlands, the settlers chose the name Swampfield for their town, but in October 1687, the general court decreed the name Danbury instead.

The first settlers bought land from the Algonquian-speaking Pahquioque Native Americans, believed to have been a band of the Paugusset people, who occupied lands along the Still River.

Danbury is nicknamed "Hat City" because it was the center of the American hat industry for a period in the nineteenth and early twentieth centuries. By 1800, Danbury was producing 20,000 hats annually, more than any other city in the U.S. By 1859 hat production in Danbury had risen to 1.5 million annually, and by 1887, thirty Danbury factories were producing 5 million hats per year. During this time, the city's motto was "Danbury Crowns Them All."

In 1802, President Thomas Jefferson wrote a letter to the Danbury Baptist Association, a group expressing fear of persecution by the Congregationalists of that town, in which he used the expression "Separation of Church and State". It is the first known instance of the expression in American legal or political writing. The letter is on display at the Unitarian-Universalist Congregation of Danbury.

Danburite is a calcium boron silicate mineral with a chemical formula of CaB₂(SiO₄)₂. It is named for Danbury, where it was first discovered in 1839 by Charles Upham Shephard.

The use of mercuric nitrate in the felting process poisoned many workers in the hat factories, creating a condition called erethism or "mad hatter disease." The condition, known locally as the "Danbury shakes", was characterized by slurred speech, tremors, stumbling, and, in extreme cases, hallucinations. Factory workers began lobbying for controls on mercury in the early 20th century, but a government study on the health effects of mercury was not conducted until 1937. The State of Connecticut announced a ban on mercury in hatmaking in 1941.

Danbury's sewage plant was renamed the "John Oliver Memorial Sewer Plant" in honor of comedian John Oliver after a lighthearted social media exchange between Oliver and mayor Mark Boughton following Oliver's satirical criticism of Danbury on Last Week Tonight with John Oliver in August 2020. Oliver donated $55,000 to local charities in exchange for the renaming.

During World War II, Danbury's federal prison was one of many sites used for the incarceration of conscientious objectors. One in six inmates in the United States' federal prisons was a conscientious objector, and prisons like Danbury found themselves suddenly filled with large numbers of highly educated men skilled in social activism. Due to the activism of inmates within the prison, and local laborers protesting in solidarity with the conscientious objectors, Danbury became one of the nation's first prisons to desegregate its inmates.

The flawed primary mirror of the Hubble Space Telescope was ground and polished in Danbury by Perkin-Elmer's Danbury Optical System unit from 1979 to 1981. It was mistakenly ground to the wrong shape due to the use of a miscalibrated testing device. The mistake was not discovered until after the telescope was in orbit and began to be used. The effects of the flaw were corrected during the telescope's first servicing mission in 1993.

The once-proud Hearthstone Castle now sits abandoned in Danbury's scenic Tarrywile Park, a forgotten edifice, boarded up and home to tangled weeds and various opportunistic critters. It is said to be haunted by spirits that sometimes mess with hikers by throwing sticks at them. Shadowy figures, glowing orbs, and even a ghost dog have also been reported.