Finally got the schematics of the PN532 Breakout board. Turns out the FTDI breakout pins has RX and TX reversed (Probably for null modem ease of use). All is well.
Oddly enough, I did try reversing pins earlier but that did not work. Now it does. Who knows.
CMata_Intel and KurtE, thanks for lending a hand.