C?
lol
most people would use c++ for this type of thing because it can be run cross platform... (with correct headers... of course there is unmanaged C which I don't know much about and c# which is managed by microsoft and usually only works natively on windows)

I know most computer system OS are loaded up with a bootstrapper (a dll or function that loads drivers and functions and hooks into the OS and starts loading what needs to be loaded)
Most are done in assembly language or I guess they could be written in DOS....

So I imagine he found an exploit within the psp that allowed him to load a bootstrapper that in turn allowed him to run a program...
Or he just searched for ways to get the psp to hook into and run his custom programs...

from there he probably took the official firmware packages and edited them to suit his needs... (Either hex editing or disassembling them?)

That's just my guess >.<