The three pennies he starts out with are likely a reference to the 30 pieces of silver given to Judas for his betrayal of Jesus in Bible scripture.
The fact that Judas starts out with low HP and a Devil item suggests that Judas made a deal with the Devil, which costs hearts and gives you demon items.
The Unholy Edition art booklet calls Judas the "mage" with reduced HP. It also refers to his starting item as "Necro"; Judas was originally supposed to start out with the Necronomicon instead of the Book of Belial.
Judas' fez hat is a reference to Phil Fish, (creator of FEZ) who voted against Edmund Mcmillen and his comrades during the Independent Games Festival, therefore becoming "a Judas". (a traitor)
Edmund has stated that the fez was because he needed an iconic hat for Judas, and he went with the idea of Phil Fish as a "Judas" because it seemed funny.
When the player pauses the game right as they enter the Basement XL, Judas will appear as if he has The Pony. This goes away as soon as the pause screen goes away.