Kilroy left the Cowboys for the New England Patriots as their director of player personnel in 1971, where he built up the team to fringe contenders by the mid-to-late 70s. Kilroy then moved higher up the executive ladder to the general manager position in 1979, again acquiring talent. Three years later, he moved up the vice president's role, and with his help, the Patriots made their first Super Bowl, though they thumped by the Chicago Bears.

In 1994, Kilroy moved to a different role, back to scouting as a consultant, and he was instrumental in advising the Patriots in the acquirement of key players that would help them win their first three Super Bowls. Kilroy worked until 2007, passing away at age 86.