Andre Drummond added 19 points and 16 rebounds for the Pistons, who had six scorers in double figures.
Johnathan Motley led the Mavericks with career highs of 26 points and 12 rebounds.
Detroit led 101-98 with a minute left in regulation, but Dallas coach Rick Carlisle kept starters Harrison Barnes, Dwight Powell and Dennis Smith Jr. on the bench.
Luke Kennard missed a clinching 3-pointer, and after Dallas turned the ball over, Jackson missed a shot that would have made it a five-point game with 30 seconds left.
Motley tied the score with a 3 with 10.1 seconds left, and Jackson's buzzer-beater bounced off the front rim.
Detroit moved ahead 110-105 with 1:29 in overtime, and after a Dallas offensive foul, Jackson's floater sealed the victory.
With both teams eliminated from playoff contention and playing short-handed, the quality of play suffered. On one first-quarter possession, the Pistons were forced into a 3-point try by Drummond as the shot clock expired. Drummond missed and fell to 0-for-10 on 3s this season and 5 for 29 in his six-year career.
Maxi Kleber's buzzer-beating jumper gave the Mavericks a 54-53 lead at halftime. Motley and Barnes had 13 points each for Dallas, while Anthony Tolliver and Henry Ellenson combined for 20 points at Blake Griffin's power-forward spot.
Neither team was able to take control in the third quarter, but a late 3 by Langston Galloway gave the Pistons a 78-75 lead going into the final 12 minutes of regulation.
The Mavericks began the fourth with a 10-2 run, but couldn't hold the advantage.
Mavericks: Three of Dallas' top five scorers — Jose Juan Berea (rib strain), Dirk Nowitzki (ankle) and Wesley Matthews (leg) — missed the game. ... Nowitzki averaged 19.7 points and 8.0 rebounds in 19 games in Detroit, all at the Palace of Auburn Hills. This would have been his first, and possibly only, game at Little Caesars Arena.
Pistons: Reggie Bullock sat out with swelling in his left knee, while Griffin missed his fifth straight game with an ankle injury. . Drummond needs 48 rebounds in the final three games to become the first NBA player to average 16 per game since Dennis Rodman in 1996-97. Rodman, who did it five times with the Pistons, San Antonio Spurs and Chicago Bulls, is the only player to average 16 for a season since Moses Malone in 1978-79.
Mavericks: At the Philadelphia 76ers on Sunday in their last road game of the season.
Pistons: At the Memphis Grizzlies on Sunday.