Regardless of how you compress, the PSP will internally have to decompress it.  In fact, decompressed RCOs may work better.  RCO Editor internally compresses the images (I was too lazy to implement header compression, Z33's tool does that).
I wonder if not compressing the images would work better (the size of RCOs would be very large though).

But yeah, it's been known for a while that large icons can cause issues...