The Millers took the lead as early as the first minute through a Semi Ajayi header before Philip Billing equalised from the penalty spot in the 52nd minute and Joe Lolley grabbed a winner two minutes later.
And although the visitors have not won on their travels since April 2016, they came close to forcing extra-time with a series of good opportunities at the end.
Kieffer Moore was denied from a couple of yards out by a brilliant Joel Coleman save while Jonson Clarke-Harris missed a sitter and Will Vaulks' headed goal from a corner was ruled out for a foul on Michael Hefele.
"I thought apart from the 10-minute wobble after half-time we were the better team," Millers boss Paul Warne said after the game.
"I appreciate I look through Rotherham eyes but I thought we were really good.
“Jono's missed a pretty easy chance - he's apologised to the lads - Kieffer should have scored his and the disallowed goal, there's absolutely no foul by my player on that so I'm pretty disappointed in that and I will ask the referee to explain why he's done it.
"In short I'm a little bit disappointed. We threw the kitchen sink at them and it was quite enjoyable to watch it."