Endless Paradigm - [PSP] Rcomage v1.1.1 - new RCO manipulation tool

sandungas Wrote: [ -> ]My old windows xp x32 complained about 2 missing dll's the first time i tryed to use rcomage-dev-v1.1.2 (previous versions doesn't), i googled them in one of these dll pages and placed inside rcomage folder and it solved the problem, here are the ones that proved working for me https://www.sendspace.com/file/bkw2m6

Gah, I'll need to do a static compile. Forgot about that.

sandungas Wrote: [ -> ]Now i realize you are still interested in releasing new rcomage versions would you like to hear some suggestions for the new version ?

I don't have a PS3, so what I can do about it is a bit limited.
It depends.  This is a hobby project which I haven't touched for a number of years, so can't remember a lot of it.  I'm also a lot busier these days than I was back then.
Regardless, please feel free to put your ideas/suggestions forward.  If it interests me, I may look into it.

---

Note that GIM «> PNG conversion can be lossy.  If you're trying to guess settings, it's better to use GimConv to convert GIM to GIM with your settings (eg "gimconv in.gim -o out.gim [settings]").

sandungas Wrote: [ -> ]The problem with pixel channel is it allows for all combinations of RGBA channels in any order... this makes a total of 24 posible combinations (rgba, argb, gbar, barg, grab, etc...)
I suspect is needed to reorder this pixel channels... but well... is just one of the random things i was trying in my lasts attempts to identify them

There's generally no reason for weird combinations, so no need to try all of them.
RGB is usually together, with the A channel either before or after it, so ARGB or RGBA.
You might want to try reversing the order as well (ABGR and BGRA), in case of endian weirdness.

But anyway, I don't think Gimconv does RGB » DXT conversion, so a PNG will never convert properly back to a DXT GIM. (I'm guessing that Gimconv also mucks up the header, making it confusing)
Try using .dds files instead of .png

ZiNgA BuRgA Wrote: [ -> ]I don't have a PS3, so what I can do about it is a bit limited.
It depends. This is a hobby project which I haven't touched for a number of years, so can't remember a lot of it. I'm also a lot busier these days than I was back then.
Regardless, please feel free to put your ideas/suggestions forward. If it interests me, I may look into it.

Good enought, are just some small details like new language codes, a couple of problems with the custom config file of gimconv, and things like that, the most interesting that maybe motivates you a bit are related with vsmx format but i prefer to talk about it in another message, thanks :)

ZiNgA BuRgA Wrote: [ -> ]Note that GIM «> PNG conversion can be lossy. If you're trying to guess settings, it's better to use GimConv to convert GIM to GIM with your settings (eg "gimconv in.gim -o out.gim [settings]").

Nice trick, i didnt imagined it and i was using a GIM--»PNG--»GIM conversion that works pretty fine if the original GIM was made by using format --image_format rgba8888
From 245 images that are inside explore_plugin_full.rco from PS3 4.70 firmware, around 200 passed this test (exact MD5 hash after rebuilding GIM--»PNG--»GIM)
The others are the problematic ones and as yoou say it seems related with the DXT conversions that are lossy, it seems is not posible to create a DXT exactly like the original
But by making a GIM--»GIM conversion the DXT is not modifyed, this allows for a perfect match after rebuilding, nice method, better than what i was doing

The options needed for the image @SonyBlack and me was looking for are:
--format_style psp
--format_endian big
--image_format dxt5
Madwin

As simple as that... being very stricts, this list of options does the same:

--format_style psp
--format_endian big
--image_format dxt5
--pixel_channel default
--pixel_alpha default
The other values set to "default" completes the list of posible options related with pixel info (and also overrides any posible value preconfigured in the gimconv.cfg)

My problem is i was trying to reorder the pixel data by looking at the hexeditor after making a lossy GIM--»PNG--»GIM conversion, and in some way i was in the good road when i was messing with the --pixel_alpha 255... these zeroes at the left columns are related to alpha, by using 255 it seems lot of them appeared at left columns, but if you look at my previous message the column most at the left is telling exactly the value for the alpha (255 = 0xFF in my test)
But what really fills all the columns at left with zeroes is the value "default" that should be considered a different value than the others in the range of 0-255 (this effect of lot of zeroes was just a coincidence in my lossy png by the way, the only one that really stores the alpha info is the column most at left)
With the 24 posible combinations of the rgba channels for --pixel_channel rgba and his variations works in the same way... the total of options is 25 because the results of "default" is different than the others 24 (i tested all them, 25 different in total)

Hard to explain i guess, but what i mean is these "defaults" are critical for the results and are part of what defines this specific GIM format

Spoiler:

sandungas Wrote: [ -> ]Good enought, are just some small details like new language codes, a couple of problems with the custom config file of gimconv, and things like that, the most interesting that maybe motivates you a bit are related with vsmx format but i prefer to talk about it in another message, thanks :)

The language codes can be modified in miscmap.ini. If new languages have been added, just append them to the list there. Entries start at 0x0 (Japanese), 0x1 (English) ... 0x10 (Polish).

v1.1.2 should have pretty much all the VSMX opcodes in there. Anything you were particularly interested in?

In relationship with languages... it seems there is a small bug in one of the help screens when displaying the language ID's
The program crashes when doing a rcomage.exe help extract

Spoiler:

Not sure if is actually related with the function that reads the miscmap.ini but it happens with your miscmap.ini and other i modifyed

For the new language codes wee use to mantain a table in wiki that includes all latets sony consoles because they uses the same language codes in all them http://www.psdevwiki.com/ps3/Languages

This is what i have in my miscmap.ini

Code:

			[languages]
00_Japanese      ; jp
01_EnglishUS     ; en
02_French        ; fr
03_SpanishES     ; es
04_German        ; de
05_Italian       ; it
06_Dutch         ; nl
07_PortuguesePT  ; pt
08_Russian       ; ru
09_Korean        ; ko
10_ChineseT      ; zh_t
11_ChineseS      ; zh_s 
12_Finnish       ; fi
13_Swedish       ; sv
14_Danish        ; da
15_Norwegian     ; no
16_Polish        ; pl
17_PortugueseBR  ; pt_br
18_EnglishGB     ; en_gb
19_Turkish       ; tr
20_SpanishLA     ; es_la

The new ones in PS3 are:
17 Portuguese (Brazil)
18 English (United Kingdom)
19 Turkish

But you can also add
20 Spanish (Latin America)
They have only used lang 20 in PS4 and maybe it will never be included in PS3 but never minds, the point is wee know they follows this order and 20 is the next one in case they decides to add it to PS3 in the future

Some of the new languages are variations of previous ones so is needed to change the names you was using in previous rcomage versions
As an example... the new "EnglishGB" needs the old "English" to be renamed to "EnglishUS"... the same hapens with Portuguese (portugal VS brazil) and Spanish (spain VS latin america) if you decide to add lang 20

Another detail... the old name you was using for "Portugese" had a typo (is Portuguese)

Also... note the names im using in my miscmap.ini following the rule:
<lang_number_code>_<lang_name> ; <lang_letter_code>
This is more like a suggestion if you want to use this naming convention in your next release... do you think changing the names to this format will cause much problems to other tools that uses rcomage as a dependency ?
Because i think is a good idea to make the names start with the language number code, this way when the .xml files are extracted them will be ordered chronologically (from older to newer with 2 digits in decimal, starting from 00 for japan)... also, many people remember some of the number codes from memory so is easyer to locate an specific .xml file if you are looking at the folder with all the extracted .xml's

Edit:
About the language letter codes... in my miscmap.ini example i have not modifyed the previous ones (en, pt, and es) that now has a variant
The new ones should be something like en_gb, pt_br, and es_la (so not sure if is a good idea to rename the old ones to en_us, pt_pt, and es_es to difference them)
There is some place where wee can see the exact letter codes the firmware is using internally ?
I mean... maybe the letters appears embedded inside the .rco structure ?... or do you know other parts of the psp firmware where languages appears specifyed by his letter code ?
The only place i know in the PS3 where appears the languages by his letter code is a document.txt that comes with the official release of the "static ps3 theme compiler" (the same toolkit that contains gimconv.exe)
If i remember correctly, another place where appears is in the web browser... but this ones should not be considered the same letter codes... are specific for the browser

Thanks for pointing those out. I'll get around to fixing them.

sandungas Wrote: [ -> ]Also... note the names im using in my miscmap.ini following the rule:
<lang_number_code>_<lang_name> ; <lang_letter_code>
This is more like a suggestion if you want to use this naming convention in your next release... do you think changing the names to this format will cause much problems to other tools that uses rcomage as a dependency ?
Because i think is a good idea to make the names start with the language number code, this way when the .xml files are extracted them will be ordered chronologically (from older to newer with 2 digits in decimal, starting from 00 for japan)... also, many people remember some of the number codes from memory so is easyer to locate an specific .xml file if you are looking at the folder with all the extracted .xml's

I doubt there's any app that really depends on it, but I still prefer not to change it.
I don't see the point in exposing the language codes. People who deal with them often might remember them, but otherwise, they're really just unnecessary detail, and it's easier to find "English" by looking for the letter "E", for example.

sandungas Wrote: [ -> ]About the language letter codes... in my miscmap.ini example i have not modifyed the previous ones (en, pt, and es) that now has a variant
The new ones should be something like en_gb, pt_br, and es_la (so not sure if is a good idea to rename the old ones to en_us, pt_pt, and es_es to difference them)
There is some place where wee can see the exact letter codes the firmware is using internally ?

Anything after a ";" is a comment and ignored, so you can stick whatever there and it won't make any difference.
From memory, the short language codes were taken from the internal name used in the menus (dump whatever RCO contains the text for the language select menu, and you should find it in the text data).

ZiNgA BuRgA Wrote: [ -> ]
sandungas Wrote: [ -> ]About the language letter codes... in my miscmap.ini example i have not modifyed the previous ones (en, pt, and es) that now has a variant
The new ones should be something like en_gb, pt_br, and es_la (so not sure if is a good idea to rename the old ones to en_us, pt_pt, and es_es to difference them)
There is some place where wee can see the exact letter codes the firmware is using internally ?
Anything after a ";" is a comment and ignored, so you can stick whatever there and it won't make any difference.
From memory, the short language codes were taken from the internal name used in the menus (dump whatever RCO contains the text for the language select menu, and you should find it in the text data).

Thanks for the hint, i found 2 .rco's in ps3 where appears the list of languages
The one responsible to make appear the language text names in xmb settings column is inside sysconf_plugin.rco

Spoiler:

And there is another list inside strviewer_plugin.rco (in this one are ordered chronologically)

Spoiler:

I have no idea what is strviewer_plugin.sprx, how it works, or where is supposed to be displayed in xmb, but based in his name (string viewer plugin) i guess is something related with debug/tool ps3 models
His purpose never minds, the point is this second list loooks more strict and better ordered for some strange reason
The bigger difference is the first one display the language names in xmb localised in his native font/name and in the second one the language names are "translated" to english

Im posting this in case it helps you to decide what to use for rcomage miscmap.ini

Thanks for the info!

The small incompatibility problem about gimconv.exe i mentioned before is related with the changes that was made in gimconv.cfg for rcomage
I made a detailed comparison of the config file provided with rcomage and the config file that comes inside the "PS3 Custom Theme Toolbox" sony released --» http://uk.playstation.com/ps3/support/se...uidelines/

I made a list with all the differences here, please take a look --» http://www.psdevwiki.com/ps3/Talk:GimConv
Are separated in 3 groups, for things that was "changed", "added" or "removed"

CHANGED
The most important problem are the settings limit_image_width and limit_image_height that was reduced to 512 pixels for rcomage... but originally was 4096 pixels
Inside PS3 RCO's there are images bigger than 512 (like this one with 30x900 pixels http://imgur.com/2rjGdbe ), and when trying to extract them with rcomage (with the additionally .png conversion)... the resulting extracted images are reduced to 512... and consequently is not posible to use them to rebuild the .RCO
I understand inside PSP there are not images bigger than 512 pixels, but is this restriction really needed ?... because is breaking PS3 compatibility, if there is not any problem could you restore the original value of 4096 pixels ?

About the other changed settings:
pixel_order changed from "normal" to "faster"... PSP uses the mode "faster" ?.... i ask because PS3 uses "normal" by default
output_sequence changed from "on" to "off"... i been testing both settings "on" and "off", and i could not see this has any effect, is really important ?... because incase of doubt i would prefer to have the default setting, just for simplicity in the list of changes

REMOVED
The first group of "PS3 OSD default settings" are overriding default settings at top of the .cfg file... and seems are specific for PS3, but not sure if they breaks some compatibility for PSP, so not sure if suggesting in restoring them
The others (options) removed i know are a bit useless, probably this is why was removed (and probably was removed at the same time when the "help" screen was modifyed), but i think is good if you restore them because doesn't makes any bad

ADDED
The -ps3 option is taken by both... rcomage.exe and gimconv.exe as a setting, right ?
I mean... in gimconv.exe im sure is used when passed from rcomage as --gimconv-flags -ps3... but when using it this way rcomages does something special with the rco ? (like selecting the endianess of the rco header based on the presence of -ps3 option ?)

I ask this because i think it could be good to add 2 new options for PS3 (are the 2 more used .gim formats, strictly specifyed overriding default settings)

Code:

			option -ps3bpp32 {
	format_style = psp
	format_endian = big
	image_format = rgba8888
	pixel_order = normal
	pixel_channel = default
	pixel_alpha = default
}
option -ps3dxt5 {
	format_style = psp
	format_endian = big
	image_format = dxt5
	pixel_channel = default
	pixel_alpha = default
}

By using this new option names.... rcomage is not given the -ps3... this is not a problem for rebuilding the rco structure ?

------
In resume, what im asking is to fix this small problem with the 512 pixels image size limits, and to make a review of the modifications made to cimconv.cfg trying to minimize the number of modifications that was made to it
Thanks for your patient

I'm fine with reverting to the stock Gimconv.cfg file, with minor edits.
It was originally only designed for PSP GIMs, so the option changes reflect that.  Also, it was only meant for use with Rcomage, not really for end users, hence removal of some stuff.  The -ps3 option wasn't really tested (cause I can't do much of that really).

output_sequence: PSP doesn't support sequences (multiple frames) from what I know.  I guess the PS3 does...

-ps3 option was added specifically for Rcomage to use.  Rcomage will automatically append that if the RCO is for PS3.  That's it really.  Manually specifying "--gimconv-flags -ps3" makes no difference.

Rcomage currently can't differentiate between DXT/RGB GIMs, so it can't auto select the correct setting to use.
I'll think about what to do about that.

The GimConv.cfg file I may end up using would probably be this:
https://github.com/zingaburga/RCOMage/bl...imConv.cfg

I don't know well what rcomage is doing internally because im not so skilled to understand the code, and i think you understood me, but i think i can explain the problem with the gim formats a bit better
What i see is the "ps3" flag is stored in the rcoxml as <RcoFile type="ps3"> for rebuilding purposes, the other option is "psp" and in this sense is ok because is working as a generic identifyer of the "rco type"
Psvita rco's has been dumped and doesn't uses the same format, and im not so sure if ps4 uses rco's but probably are based in the same format used by vita (is a format that was used in ps3 too with .qrc .p3t and .raf files)
So this is like a "rco type" flag needed by rcomage.exe... and the list of posible values is "psp", or "ps3"... this completes all and is accurate enought

But for the "gim type" flag that is passed to gimconv.exe are needed different ones because there are several gim formats used for each console
PS3 uses 2 different gim formats as far i know, one is rgba8888 based and the other dxt5 based. This is why i proposed for PS3 the labels "ps3bpp32" and "ps3dxt5" in gimconv.cfg.. and for psp i don't know if is needed only one or more than one

The reason why i think is better to separate the flags for "rco type" and "gim type" by using different names is because what rcomage is doing actually is passing the "rco type" flag to gimconv.exe (but the rco type doesn't defines accuratelly the gim type)
The "gim type" flag because is a per-file setting is something that you could store in the rcoxml as an attribute for each <Image> under the <ImageTree>
I have no idea how to do the automatic identification of the .gim type though

Do you know if someone made a list of all the different .gim types/formats used in PSP ?
And why are added the "convenience options" in gimconv.cfg ?... it means the PSP has support for other formats than the ones used in ofw ? (lower or higher quality)

------------
By the way, i just realized the value of the attribute "minFirmwareVer" in the rcoxml looks a bit messed up (the unknownId0x130 in the example below), is this normal ?, and is refered to the system firmware version ?, in this case can be converted to decimal with the dot, as example "1.00" ?

Code:

			<?xml version="1.0" encoding="iso-8859-1"?>
<!-- This XML representation of an RCO structure was generated by Rcomage v1.1.2 -->
<RcoFile UMDFlag="0" rcomageXmlVer="1.1" type="ps3" minFirmwareVer="unknownId0x130">
	<MainTree name="explore_plugin">
		<ImageTree>
			<Image ............