Looking at the schematic, it seems that the RAMPS board powers the Arduino with 12 V through its VIN pin, and lets the Arduino step it down to 5 V. So you must have fried the 5 V regulator on the Arduino. That also explains why it works with 5 V USB power -- the on-board regulator isn't used in that case.
It could be that D1 on the RAMPS is fried, or the diode on the Arduino board (immediately to the left of the header with IOREF, RESET, 3.3V etc).
The Arduino needs at least 7 V on its power connector, it definitely won't work with 5 V (although a LED or two may turn on). Powering the Arduino that way should also supply 5 V to RAMPS. If the regulator on the Arduino is broken, it will not work with external power, only with USB.
If it's the regulator, you can fix it by supplying 5 V yourself: use a LM7805 and a capacitor to get 5 V out of the 12 V supply. For a quick and dirty repair, you could simply solder the 7805 on top of the existing regulator (no capacitor needed in that case). I've circled the 5 V regulator on the Arduino in red.
If it's the D1 diode on the RAMPS, replace it, or simply short it
then never connect anything to Arduino's power jack (put some tape on it). If it's the diode on the Arduino board, replace it -- don't short that one, it can hurt the computer's USB if you do.