The club was founded as the Kansas City Scouts in Kansas City, Missouri, in 1974. The Scouts moved to Denver, Colorado in 1976 and became the Colorado Rockies. In 1982, they moved to New Jersey and took their current name.

Scott Stevens served as captain of the Devils from 1992 to 2004. Personifying the team's defense-first mentality, he led the Devils to four Stanley Cup Finals appearances in nine years, winning three championships. At the time of his induction into the Hockey Hall of Fame in 2007, he had the most penalty minutes of any player enshrined there. The Devils raised his #4 jersey to the rafters on February 3, 2006.

On April 22, 1988, Patrik Sundstrom set a Stanley Cup playoffs record by recording eight points (three goals and five assists) in New Jersey's 10-4 victory over the Washington Capitals in Game 3 of the Patrick Division Final. This broke the previous record of seven points, recorded on three occasions by Wayne Gretzky while with the Edmonton Oilers.

Widely regarded as one of the greatest goaltenders of all time, Martin Brodeur played 21 seasons for the New Jersey Devils, with whom he won three Stanley Cup championships and five Eastern Conference championships in 17 postseason campaigns. He is the only goalie in NHL history with eight 40-win seasons.

Stevens was often credited with changing the momentum of a game not with a goal, but with a hard check, earning him the nickname "Captain Crunch".

The Devils sent Adam Larsson, who became a second-line defenseman for the Oilers, for Taylor Hall, who won the Hart Memorial Trophy as league MVP in 2018.

On May 6, 1988, after a 6-1 loss to the Boston Bruins, Devils coach Jim Schoenfeld confronted Koharski in the runway that leads to the dressing rooms in the Meadowlands arena. During the argument Koharski fell down and accused Schoenfeld of pushing him. The coach's famous utterance (minus the expletive) was: "You fell, you fat pig! Have another doughnut!" The NHL decided to suspend Schoenfeld for one game, only to have the Devils win a court injunction allowing him to coach. The infuriated officials responded with a one-game strike.

Pat Verbeek, a.k.a. "The Little Ball of Hate", was selected 43rd overall by the New Jersey Devils in the 1982 NHL Entry Draft. He is the only player in NHL history to total over 500 career goals and 2500 career penalty minutes.

On May 15, 1985, one of Pat Verbeek's thumbs was cut off by an auger in a farming accident. Thanks to his father and brother his thumb was saved, and after extensive rehabilitation, Verbeek returned to hockey. He did not miss any regular-season NHL games because of the injury.

In 1995, the Devils made their first appearance in the Stanley Cup Finals, sweeping the Detroit Red Wings 4-0 to win their first championship.