Andros Townsend headed in his first goal for Town but Blues went down to a Craig Beattie double and a Joe Allen strike in a controversial finish at Portman Road.
Carlos Edwards appeared to be pulled back in the box by Mark Gower five minutes from time and the Blues players were still protesting to referee Andy D'Urso when Beattie cracked in a superb third goal for the Swans.
Townsend gave Blues the lead with a header five minutes into the second-half but Town pressed the self destruct button again at the back with Beattie beating everyone to a Gower free-kick five yards out then Allen forcing the ball over the line after Blues had failed to clear.
Town boss Roy Keane made three changes to the team that beat West Brom in midweek, Gianni Zuiverloon coming in for his home debut, with Edwards and Jack Colback also included in the starting line-up. David Norris was ruled out through illness.
Beattie had a header saved by former Swans' 'keeper Brian Murphy in the opening five minutes, Andrea Orlandi then sending a free-kick over the bar.
Swansea dominated possession in the first 25 minutes but Town created the far better chances and Tamas Priskin was denied by Dorus de Vries after he latched on to Angel Rangel's back pass.
De Vries held firm from Townsend cross, while at the other end his Town counterpart Murphy gathered Scott Sinclair's shot low down.
Murphy saved comfortably from Beattie's header, then Jason Scotland - up against his former team-mates for the first time since leaving the Liberty Stadium in the summer of 2009 - was denied by de Vries after Grant Leadbitter and Jack Colback had linked up.
Priskin carved out an opening for himself on the edge of the box but his effort went just wide, Edwards then testing de Vries again with a low strike from 20 yards.
Darren O'Dea picked up the only booking of the half for a foul, Swansea's free-kick leading to nothing though and it was Town who finished the half the stronger, Troy Brown bringing an excellent save from de Vries with a powerful header from Edwards' corner.
The second-half was only five minutes old when Town got the breakthrough, Edwards the provider with a cross and Townsend arrived on time to head past de Vries from six yards.
De Vries saved from Priskin's header from another Edwards cross as Town threatened a second.
Rangel joined O'Dea in referee Andy D'Urso's book after a foul on Townsend but Swansea were level just past the hour, Beattie beating everyone to get on the end of Mark Gower's free-kick.
Town made a first change on 65 minutes, Rory Fallon on for his home debut with Priskin making way.
Beattie blasted over from close range in a great opening for the visitors, while at the other end Fallon sidefooted wide with an equally good chance after Townsend had led the breakaway.
Swansea struck again on 70 minutes, Allen taking advantage of another error at the back for Blues and forcing the ball over the line after Tommy Smith had failed to clear the danger.
Shane O'Connor replaced O'Dea in Town's second change, Darren Pratley coming on for Orlandi as Brendan Rodgers shuffled the Swansea pack.
Fallon got on the end of Alan Tate's mis-directed header, de Vries punched clear and the loose ball eventually fell to Scotland but he blasted his shot wide.
Jaime Peters replaced Townsend for the last 10 minutes and the game exploded and was decided second later, Edwards went down in the box for what looked a clear penalty after he was caught by Gower. Referee D'Urso waved away Town's frantic appeals and seconds later Beattie smashed in a third from 25 yards to seal the points for the visitors and inflict a fifth successive Championship defeat on Blues.
Town: Murphy, Zuiverloon, Brown, Smith, O'Dea (O'Connor 71) , Edwards, Leadbitter, Colback, Townsend (Peters 80), Priskin (Fallon 65), Scotland. Subs not used: Lee-Barrett, Livermore, Eastman, Healy. Booked: O'Dea.
Swansea: De Vries, Rangel, Tate, Williams, Taylor, Allen , Dyer (Dobbie 88), Gower, Orlandi (Pratley 75), Sinclair, Beattie. Subs not used: Ma-Kalambay, Cotterill, Van der Gun , Serran, Agustien. Booked: Rangel.
Att: 16, 978