I never really thought I'd actually bother, but I have been doing some investigation into the VSMX file format, found in RCOs from video/music UMDs and two of the lftv RCOs in the PSP's flash0.
The VSMX file is basically a compiled script file which controls stuff like the UMD menus. I don't know whether VSMX scripts can be used in the XMB or not.
From a string constant I've seen in one of them, I'm guessing that the official Sony script compiler takes a Javascript file as input and outputs the compiled VSMX file, so I will consider a number of things here from a Javascript perspective.
So anyway, this post is just for anyone who happens to be interested in the file format, and doesn't really serve much other purpose. It's also rather "preliminary" in that it's _really_ incomplete (I've spent some time on this, but not that much).
Okay...
General File Layout
...basically looks like this:
VSMX header
Compiled instructions
String constants
Object property/method names?
Variable/function names
VSMX header
Compiled Instructions
The compiled instructions start at offset 0x34. An important concept here, I'll refer to as a "group". A group is a 64-bit sequence, consisting of 2 32-bit values. The compiled instructions section is entirely made of these groups.
The first value of the group is the identifier and the second specifies any additional information, or is 0 if not used (I will refer to this second value as the argument).
Here's a list of the identifiers that I know:
For structure, the variables/constants are just listed and operators apply to previous things.
For example, take the following hex sequence of 32-bit values:
(for reference purposes, wee'll assume the first variable name is "myVariable")
The above translates to the following Javascript:
Back to the hex, basically it means:
The assign operator gets applied to the previous two values, which happen to be "myVariable" and "12" and does its job on the two.
String constants
This is just a list of unicode (UTF-16) string constants, separated by a single null character (remember, null characters are 16 bits wide in unicode). There is no alignment.
Object property/method names
Pretty much the same as the string constants section.
Variable/function names
Like above section, except ASCII instead of unicode.