I there a way to move "layers" on top of each other...
for example:
the submenu icons are comming underneath the topmenu...
the button menu in the musicplayer is underneath the playicon and the playtext etc...

Is there a way to move these.. I already tried to rearange the musicplayer layers in the prx but don't know if I'm doing it right.

hope someone can help me out. This could take the vsh theming to a next level

im sure there is if u find the right prx and values. try playing around with them and eventually u'll find it

It's difficult to think of a starting point.

And it may not be just a simple hex edit either.

^ Yes layers exist everywhere, but where to start?

To achieve layers, the lowest layer is drawn first, followed by more higher up layers.  Some layers, like having a custom background may be skipped, as you can't have transparency/alpha with custom backgrounds.
Also, I don't know how it's internally stored.  It would be either:
1) Just loaded and drawn in a specific order
2) PSP stores each layer with a corresponding z-index, after all layers are loaded, it's drawn bottom-up

If the latter case is true, a simple hex edit may do the trick.  But if the former, it's unlikely a simple hex job would do the trick.