The Colts agreed to send Allen and a sixth-round pick to New England for a Patriots fourth-round pick, according to league sources.
Allen did likewise to the Colts and their fans and said he looked forward to playing for New England.
"I want to give my sincerest thanks to the Irsay family for taking a chance on a kid from Fayetteville, NC," Allen said in a statement posted to Twitter. "It's been an honor wearing the horseshoe and representing the Colts on and off the field. My teammates will always be my brothers, and the friends that I've made are for life.
"I'm both humbled and excited for the challenge ahead of earning the right to be called a New England Patriot. May God continue to bless the city of Indianapolis and the Indianapolis Colts."
Before a 2014 game between the Patriots and Colts, New England coach Bill Belichick had high praise for Allen, saying, "I've been really impressed with Allen's blocking. I think he's one of the best blocking tight ends we'll see."
Allen re-signed with the Colts last offseason on a four-year, $29 million contract. He finished with 35 receptions for 406 yards and six touchdowns while playing in 14 games in 2016.
Allen was part of the Colts' 2012 draft class that also featured Andrew Luck, T.Y. Hilton and Coby Fleener. Former general manager Ryan Grigson chose to re-sign Allen over Fleener in 2016 because he felt Allen was the better all-around tight end. Allen, however, never reached his potential in Indianapolis because of consistent injury problems. He has missed 23 games over the past four seasons.
The Colts re-signed tight end Jack Doyle on Tuesday. A source told ESPN's Adam Caplan that the deal is for three years and $19 million, with $9.5 million guaranteed. The deal can max out at $21 million with incentives.
Doyle, 26, made a smooth transition from being known as a blue-collar tight end and third on the Colts' depth chart to an all-around player who was arguably their best tight end last season.
He had career highs in receptions (59), yards (584) and touchdowns (five) in 2016. His 584 receiving yards and 75 targets were second on the team behind Hilton's 1,448 receiving yards and 156 targets.
Bennett was acquired by the Patriots in a trade from the Chicago Bears last March. The sides had discussed extending his contract through 2017 at the time of the trade but couldn't come to an agreement.
That set Bennett up to become an unrestricted free agent this offseason. He is widely viewed as the top tight end available.
The 6-foot-6, 265-pound Bennett totaled 55 receptions for 701 yards and seven touchdowns in the 2016 regular season, playing in all 16 games for the Pats. He earned the respect of many teammates by playing through an ankle injury initially suffered on Oct. 9.
A supersized personality, Bennett was acquired with the hope of forming the NFL's most lethal 1-2 tight end combination alongside Rob Gronkowski. But that never truly materialized because Gronkowski was limited to eight games due to hamstring and back injuries.
That thrust Bennett into a more consistent role as the team's No. 1 option at the position.
ESPN's Mike Reiss and Mike Wells contributed to this report.