Develop Bugs

Question on "hand modified XIP chain"

Could you explain please (very short description) how you modified the xip chain for rom kitchen?
All I can see is the following:
- no length (0)
- no RSA1 signature
- only file entries
What I want to know:
- how to find phys. (ROM) position (do you use unused holes in rom?)
- is 0 length for ROM = autolength
- how to choose the RAM position
- why can length of RAM be 0
Please help. (I need this info for a smartphone project)

I did not bother setting the length, only the 'pvAddr' field is used.
I only make fileentries, because I have yet to implement the generation of modules. ( if I ever do ).
yes, I use unused holes in the rom.
actually, if you don't care about xip updates of other sections, you
may use addresses anywhere in the rom, where your data fits.
It does not nescesarily have to be contigous.
I just copied the ram setting from the other xip entries.

Thank you for the information.
Why don't you take romimage.exe from platformbuilder to generate a XIP block. You only have to write a little .bib file for it. This tool can handle modules and compression as well.
John
P.S. Source code for romimage.exe is available in PB 4.2 private build tree.

I hadn't found that tool yet when I wrote makexip, and then we couldn't have made the romkitchen with it, since romimage.exe runs only under windows.

Don't waste your time with this crap tool (romimage.exe). Some needed files are missing (e.g. bin2xip.exe).
How can I be sure to choose a good phys. addr.? There might be some memory mapped devices...
I have one additional difficult question:
Modules are relocated when embedded into XIP's. Even there seems to be a modification to the import table of the module (e.g. references to coredll.dll will be checked/updated?)
If I extract a module (e.g. a *.dll) from a XIP of an other phone do I have to re-relocate it / modify it's import section even if I place it in a FILES section?
Thanks
John

converting bin to xip is not that difficult. see http://www.xs4all.nl/~itsme/projects/xda/wince-flashfile-formats.html
do you mean the 'physfirst' field in the romheader? that is just the startaddress in the rom.
since the relocation information is not stored in rom, the only way to really
recover it, is to disassemble the file, and find the offsets to stuff that
needs to be reallocated.
so that is a lot of work. and dumprom only extracts nonrelocatable .exe and .dll modules.
if your extracted dll is fixed to a memory location that overlaps with an already existing dll, you will have a problem I think.
I am not even sure, if an extracted dll works at all, I only use them for reverse engineering.

Yes, I mean phys first field. But how can I be sure to choose a valid address for the new XIP block?. My idea is to use address space between existing XIP blocks. Or can I simply choose a very high address (e.g. 8F000000) and hope not to use a region where memory mapped devices are located?
Since I used (your?) dumprom to extract the *.dll files do you think they are "nonrelocated"?
John
I ask so much because I crashed my lovely smartphone a week ago. :-(
My new XIP seems to be invalid... so it doesn't boot anymore. Unfortunately I've killed the bootloader too...
When I try next time (I've ordered a new one) this must not happen!

I am sure they are nonrelocated, fixed to run from a specific memory location.
( just wrote another post about this )
maybe even the module loader does not allow non-xip modules to be loaded in xip reserved memory.

THANK YOU VERY MUCH
I've got it. My Smartphone now have a new XIP block with some files in it...
Only thing left is to rewrite some *.dll files (only resource dlls with no function exports) to extend the language of the MIO 8380.
Are you familiar with languages on smartphone? There are multiple .mui files (resource dlls containing all the dialogs and strings). I've exported all resources and (re)created the dll's as resource only. Unfortunately they don't work ... yet ...
Are there some other files for language extension? What about "wince.nls" or "mxip_lang.vol" ?
Thanks again for your great tools. I will setup a site containing detailed information about this hack soon.
John Smith

cool, I am always interested to see how things work out that I haven't actually tried myself yet.
is this how you create resource only dll's:
http://www.xs4all.nl/~itsme/projects/programming/icondll.html

Currently I'am a little bit confused. PB 4.2 docu says MUIs are resource only .dlls and sample in smartphone sdk adds a dllmain...
I will have to investigate this things a little bit more...
John

O.K.
I've tried anything. The only thing left is that the new resource dlls are not XIPed as modules...
The sample mui app works fine (regardless of resource only / normal dll).
John
P.S. I've successfully changed all other settings some things already appear in the new language. Only poutlook, homescreen and control panel will not change!

Now after some more testing (included a dllmain into the mui file which logs the loading/unloading to file) it seems that my mui.dll is never loaded by system (if I load it manually with LoadLibrary the log is written).
Here is my question:
I've looked a little bit deeper into the dumped mui.dll and found a pointer in security section (pe header) which points to nowhere (just after the [virtual] end [rva] of all of the e32/o32 sections).
Could it be, that I've missed something? Does dumprom fill in this values correctly?
One other interesting idea could be to exchange only the data section of the module (since I want to patch resource only .dlls). But since english is a very short term language all other files will be bigger...
John

>>> I've got it <<<
the new (mui-language) modules have to be REAL xip modules...
So I've build a custom.bib file and used RomImage from CE3.0 Platformbuilder. Even compression is possible now.
Note: romimage.exe does the same thing as makexip.pl
To share my results here is the content of the .bib file I've used:
Code:
MEMORY
; Name Address Size Type
MYXIP 81f00000 0013f000 RAMIMAGE
RAM 8c020000 00fe0000 RAM
CONFIG
COMPRESSION = ON
PROFILE = OFF
ROMFLAGS = 2
ROMSTART=81f00000
ROMSIZE=13f000
ROMWIDTH=32
DLLHIGHADDR=00b00000
MODULES
; Name Path Memory Type
; ------------------------- ------------------------------- ------ ----
outres.dll.0407.mui input\outres.dll.0407.mui MYXIP SHU
syncres.dll.0407.mui input\syncres.dll.0407.mui MYXIP SHU
tapres.dll.0407.mui input\tapres.dll.0407.mui MYXIP SHU
tshres.dll.0407.mui input\tshres.dll.0407.mui MYXIP SHU
wmplayer.exe.0407.mui input\wmplayer.exe.0407.mui MYXIP SHU
FILES
; Name Path Memory Type
; ------------------------- ------------------------------- ------ ----
Busy.0407.mid input\Busy.0407.mid MYXIP
mxip_lang_799.rgu input\mxip_lang_799.rgu MYXIP
ms_splash.gif input\ms_splash.gif MYXIP
carrier_splash.gif input\carrier_splash.gif MYXIP
- The MYXIP region in MEMORY section is a hole in the ROM I've found with calcgaps.pl.
- The RAM region is copied from the other sections (they all use the same)
- ROMSTART and ROMSIZE have to be the same values as defined in MYXIP
- DLLHIGHADDR has to be the !!!lowest!!! loading address found with dumprom (header: dlls=...-... ).
Example: If the lowest address found is "header: dlls=00b00000-00c90000 ..." then DLLHIGHADDR has to be 00b00000
Don't care about the warning the warning "Unable to do imports from ... to COREDLL.dll - will late bind". Thats because coredll is in another XIP.
John
P.S. Thanks a lot for all of your support.
DETAILED INFORMATION ABOUT THIS HACK CAN BE FOUND HERE:
http://smartphonerom.tripod.com (only download the "detailed information")

[APP][DISCONTINUED] TF3D manila mode9 editor

6Fg8 proudly presents:
m9editor
development discontinued
​
m9editor is your little helper when it comes to editing manila files. It lets you edit nearly all aspects of a mode9 file. Additionally it will assist you with graphics, allowing import, export and CFC compression. And It includes a directory viewer with information specific to manila files.
requirements:
m9editor is written in .net, so .netCF3.5 (currently SP1) must be installed (usually its already there). It has been developed and tested using Windows XP x64, but should run on any Windows platform.
usage:
see the manual contained in the download package (PDF)
making of / credits:
thanks to: D-MAN666, sztupy, nixx-X1, Chainfire, pcarvalho, showaco, guap, 12aon, xboxmod, NisseDILLIGAF, smotrs, for contributing knowledge, ideas, testing. Thank you guys (& gals of course) !
version history:
2009-03-10 - v3.3.0.1
BUGFIX: crashed when using selections
2009-03-09 - v3.3.0.0
- LuaConv is now included in the m9editor download package
- CHG: all sourcecode related functions use ".lua" as default extension
- NEW: compile and import function for lua script sourcefiles:
for mode9 files (embedded scripts):
m9editor remembers the filename of the source on a per-node basis and lets you subsequently edit the recently used file from the particular node. You may also attach/detach such a link manually. If a link is established, you may edit the sourcefile (check if you set the "SourceCodeViewer" option in m9editor.cfg) and recompile/import in a single step. When saving the mode9 file the links are saved too (mode9file.ann) and reloaded on next open.
for standalone scripts:
m9editor checks if a correspondig xxxxxxxx_manila.lua exists in the script directory. If yes, you are offered edit and compile options.
- NEW: editor context menu: view source for embedded scripts
- NEW: search previous button
- FIX: editing characterreference threw exception
- NEW: dirviewer context menu for mode9 files: compare with: compare two mode9 files without loading it first
- NEW: dirviewer context menu for all files: hex compare with: compare two files with hexview.
Only available if AptDiff is installed. Edit m9editor.cfg and insert the following line (use the path on your system):
AptDiffPath=c:\Program Files (x86)\Brother Technology\AptDiff 1.5\aptdiff.exe
If AptDiff is installed and configured in m9editor.cfg, text compares are done with AptDiff instead of ExamDiff
WARNING:
during directory analyze, if a lua script is found, m9editor checks for existence of <filename>.luac.lua. If its found it will be renamed to <filename>.lua. In directory viewer the line will be shown blue colored to indicate the presence of a sourcecode file.
during mode9 file load, if a embedded script is found, m9editor checks for existence of <filename>_<bytepos>.luac.lua. If its found it will be renamed to <filename>_<bytepos>.lua. In mode9 editor you will notice the link to the script sourcecode. Dont forget to save the mode9 file to make the link permanent (otherwise if you modify the structure, links might be lost due to changing byteposition).
WARNING2:
The pdf attached doesnt reflect the latest changes to m9editor, update follows.
2009-02-22 - v3.2.0.1
- fixed definitions for weather.mode9 and landscape.mode9 which rendered them unusable after modifying
2009-02-22
I removed the QtcConv and LuaConv tools because they lack of syntactical checks and the same functionality is available in m9editor.
EDIT: added the old LuaConv again, seems to work not that bad i thought it would
2009-02-21 - v3.2.0.0
- font changes for various fields to avoid overlapping
- if "manila files dir" is changed in settings, directoryview will be reloaded
- new: directory viewer context menu option for mode9 files: "extract embedded scripts", writes them to script directory
- changed interpretation of UV property (16.16 instead of INT32)
- picture view: new "save as PNG" and "save as QTC" buttons will save your graphic to a selectable directory
2009-02-19 - v3.1.0.1
bugfix for "save tree to file" function
2009-02-19 - v3.1.0.0
time for a new version
- m9editor now remembers last window/splitter positions and restores them on start and returning from minimize
- HOT compare feature: if a mode9 file is loaded, rightclick on the toplevel node to compare with another mode9 file, or save the treeview as a textfile. Compares are done with a freeware tool named "ExamDiff" from PrestoSoft.
2009-02-18 - v3.0.5.0
new context menu functions in directory viewer: delete, copy to
2009-02-17 - v3.0.4.0
content of XML, HTML displayed in lower right panel
evaluation of XML locales in directory viewer
manual is complete, and now a part of the package
2009-02-16 - v3.0.3.1
annotations now work for new tags as well, and will be copied between instances (+corrected a few flaws during display)
new: search within decompiled lua scripts
new: copy annotations between instances of m9editor
2009-02-11 - v3.0.2.0
bugfix: empty value crash
minor improvements
2009-02-10 - v3.0.1.0
some little improvements:
- sourcecode of embedded scripts and .luac files is shown whenever possible
- zoom percentage display for graphics display
2009-02-09 - v3.0.0.0
I've made a lot of changes to improve m9editor. The most important are:
- integration with sztupy's luadec (thanks and credits to sztupy for great decompiler and permission to use, and of course once again to D-MAN666 for providing knowledge)
- import of compiled lua scripts (ANSI or Unicode, autodetected)
- UI reworked: editor, directory viewer, image viewer combined in one window
planned for next releases of m9editor:
- copy files to/from PDA if connected via ActiveSync
- smart-select referenced graphic files in a mode9 file
- selfcheck for consistent mode9 writing
- analysis of lua script "require"'s for valid manila names
download stats:
v.3.2.0.1: 1554 views (wow! impressed!)

Guess I'll have to try that, seems great !
Your editor seems (as you describe it) only able to change values, but not the structure of the mode9 file, do you intend to change that ?
I have an ongoing project of mode9/xml converter in C, if I can be of some help, don't hesitate to ask

Hi Ximoon,
guess you belong to the "intended audience"
At the moment only changing values is allowed, correct. The main reason for this is that i'm unable to interpret and understand the complete structure, and to make sure that writing of m9 files produces valid files. If there's more info on that i could build it into the editor of course.
What i could imagine is a more generic approach, e.g. allow insertion/deletion of tags based on hex-values. I'll give that a thought.
Do you have any detailed info about the tag-structure? It can be seen in the tagfile but as hexvalues. Ive been able to interpret some of them, but the meaning of many others is unknown to me. So m9editor filters most of them for better reading.
Anyway, there's more to come with m9editor. I already built routines to disassemble lua scripts which run fine, but as its output is only relevant to p-code its pretty useless at this time. Also planned is a tool to allow image replacement. That could go to a point where we have a complete workbench for TF3D, including preview and the like. But thats far beyound my current scope.

Nice anyway !
I have no more information than you do, all I got is from the manilla thread where you posted earlier
But in the end, names of the tags doesn't really matter.
For LUA, I found some decompiler on the web, so I guess a tool to convert lua-unicode to lua-ascii and reverse could work to use those tools.
I guess I'll go on with my editor and see what I could do in the end, for two different approches could be constructive !

Must have been ChunkSpy, right? I found many more but mostly outdated and not working for Lua 5.1.
The Lua routines i've already written allow converting unicode to ascii, and save the scripts in ascii respectively. I'll have to recode them in .net as the first quick approach has been VB

Nice start, man
I was waiting for so long since somebody starts public Manila DevKit since i posted specs i reversed
Drop me a PM if you have any questions

Can't wait to use those lua tools

yeah, right D-MAN, most of that thing is based on your findings, my respect m8.
I'll PM you later, gotta go shopping now for newyears eve

6Fg8 said:
yeah, right D-MAN, most of that thing is based on your findings, my respect m8.
I'll PM you later, gotta go shopping now for newyears eve
Click to expand...
Click to collapse
Yah, sure, after NY eve when my hangover is over

oh oh, nice tool thank you

new version has been released, details here

got ean error if i want to open a file..
System.UnauthorizedAccessException: Der Zugriff auf den Pfad C:\Dokumente und Einstellungen\Administrator\Desktop\HTC\tf3d\2c684cd8_manila wurde verweigert.
bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
bei System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
bei System.IO.FileStream..ctor(String path, FileMode mode)
bei m9editor.m9efunctions.readm9(String fname, Boolean writedebug)
bei m9editor.m9editor.fileopen_Click(Object sender, EventArgs e)
bei System.Windows.Forms.Control.OnClick(EventArgs e)
bei System.Windows.Forms.Button.OnClick(EventArgs e)
bei System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
bei System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
bei System.Windows.Forms.Control.WndProc(Message& m)
bei System.Windows.Forms.ButtonBase.WndProc(Message& m)
bei System.Windows.Forms.Button.WndProc(Message& m)
bei System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
bei System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
bei System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
Click to expand...
Click to collapse
edit: i was write protected.....

schreda said:
got ean error if i want to open a file..
System.UnauthorizedAccessException: Der Zugriff auf den Pfad C:\Dokumente und Einstellungen\Administrator\Desktop\HTC\tf3d\2c684 cd8_manila wurde verweigert.
Click to expand...
Click to collapse
sorry, i dont have that file on my touch HD, cannot test. Are you sure its a mode9 file? Can you upload it here?

schreda said:
edit: i was write protected.....
Click to expand...
Click to collapse
that explains something

hmm.. i'm trying to port the analog clock from the Diamond to the HD.. i must change the Resolution i think but i got a bad result if i change the size var... ( i took the original file from the Hd to compare it

schreda said:
hmm.. i'm trying to port the analog clock from the Diamond to the HD.. i must change the Resolution i think but i got a bad result if i change the size var... ( i took the original file from the Hd to compare it
Click to expand...
Click to collapse
great idea, i'd appreciate to see a analog clock on the HD

new version has been released, details here

wow! finally someone is doing it
thanks and keep up the good work!!!
what i show in screenshot, is it portion of unparsed data?
can one edit them? i tried and failed cause i dont know what to change...
i'm missing: RectF, X, Y, Width, Height

pcarvalho said:
wow! finally someone is doing it
thanks and keep up the good work!!!
what i show in screenshot, is it portion of unparsed data?
can one edit them? i tried and failed cause i dont know what to change...
i'm missing: RectF, X, Y, Width, Height
Click to expand...
Click to collapse
its the value of the tag above, "Viewport". Currently i dont know how to parse that, as its binary and i dont understand the meaning. Could be 8 x Int16, or 4x Int32, or a mixture of that, and even then i wouldnt understand what these values mean. Obviously parameters for Viewport.
Yes, you can edit that, simply overwrite the hexvalues and press enter, it will be written.
Can you shed light into the purpose of these values? I'll be glad to parse them right.
BTW there is a baaad bug in v1.0.2.0 when writing m9, i'm currently correcting that.

6Fg8 said:
its the value of the tag above, "Viewport". Currently i dont know how to parse that, as its binary and i dont understand the meaning. Could be 8 x Int16, or 4x Int32, or a mixture of that, and even then i wouldnt understand what these values mean. Obviously parameters for Viewport.
Yes, you can edit that, simply overwrite the hexvalues and press enter, it will be written.
Can you shed light into the purpose of these values? I'll be glad to parse them right.
BTW there is a baaad bug in v1.0.2.0 when writing m9, i'm currently correcting that.
Click to expand...
Click to collapse
take your time my friend
those ones for Viewport i only know that they are:
<Property Name="Viewport" Type="RectF" X="0" Y="0" Width="31457280" Height="35127296" Animated="False"/>
cause i used D-MAN666's mode9parser, talk with him cause he might help, i hope

[REF][APP] Manila file names

I'm hoping this will be helpful to anyone looking to modify Manila.
Manila 2.5 has all new file paths so the file lists from older versions don't apply. I searched the forums but couldn't find any info on new file names, so I decided to try to put together a new list and this is the result.
This should be a solid base, and I hope that we can build up from here. Anyone who finds new file names please post your findings so we can get a more complete general reference as well as keep it up to date in case of any later changes to Manila.
[APP] Manila Name Finder:
This started out as just some scripts to help me search for names, but has evolved into a proper little program. It can search decompiled mode9 and lua scripts for manila names as well as reconstruct names from localization xmls.
It's not really the most user friendly thing, but it works. The included ReadMe.txt has some basic instructions for use. This tool is also included as part of this Manila Kitchen and that really is the best and easiest way to use it.
I'm also posting the source code. If a future version of Manila breaks the naming scheme again like Manila 2.5, this finder will need to be updated. I'll probably update it but in case I'm MIA someone will hopefully find this source code useful.
Download:
Manila Name Finder v1.0
MNF v1.0 source code (C++)
[REFERENCE] NAME LISTS: Manila 2.5 and 2.1: (a bit outdate right now - will update soon)
You can just paste the lists into m9editor's m9editor.names.txt to get the manila names displayed in m9editor or any manila kitchen which uses on the same m9editor.names.txt for manila names.
Note: Only the file list in the lower left corner of m9editor is affected by adding the new names to the list. The hashed names that appear when you mouse over a file in the editor window aren't affected by this. See the guide below to learn how to get hashed manila names from mode9 files.
Current status of Manila 2.5 list (stats based on Manila 2.5 from Duttys Leo R1 ROM)
Code:
mode9: 69 of 69 ....100.00%
luac: 297 of 318 .....93.39%
qtc: 925 of 1216 .....76.06%
xml: 1220 of 1239 .....98.46%
Current status of Manila 2.1 list (stats based on Manila 2.1.38680)
Code:
mode9: 51 of 51 ....100.00%
luac: 225 of 234 .....96.15%
qtc: 711 of 813 .....87.45%
xml: 1076 of 1081 .....99.57%
Note: Both name lists actually have more files then the stats show. The stats only count the names that translate to valid *_manila names.
Download:
Manila 2.5 Name List (8-Aug-09)
Manila 2.1 Name List
[GUIDE]: Manila 2.5 file names - what changed compared to 2.1 and earlier
The hash algorithm that was used in earlier Manila versions is unchanged in Manila 2.5. What has changed is the directory structure.
In Manila 2.1 all the mode9 files were located in \windows\htc\ (e.g. \windows\htc\home.mode9, \windows\htc\email.mode9), all the images in \windows\hts\assets\images and lua scripts in \windows\htc\scripts.
Manila 2.5 changes this by breaking it up into subdirectories. So home.mode9 is now located in \windows\htc\home\home.mode9. The same \windows\htc\home\ directory is shared by related mode9 files, such as citypicker.mode9 and worldclock.mode9 (full paths: \windows\htc\home\citypicker.mode9, \windows\htc\home\worldclock.mode9). Other mode9 files have different subdirectories e.g. calender.mode9 and calendar_picker.mode9 are located in \windows\htc\calendar\, while email.mode9 is \windows\htc\email\email.mode9 etc.
Now, any files referenced in the mode9 (images and scripts) are also in the same subdirectories. Because Manila uses relative paths, you'll see something like .\assets\images\%imagefilename%.qtc in a mode9 files.
The ".\" is the current directory. If you're editing \windows\htc\home\citypicker.mode9 then the current directory is \windows\htc\home\, so the full path of the image is \windows\htc\home\assets\images\%imagefilename%.qtc.
You might also see this: ..\common\assets\images\%imagefilename%.qtc. "..\" is up one directory. So again if you're editing \windows\htc\home\citypicker.mode9 up one dir from there is \windows\htc\, and the full image path becomes: \windows\htc\common\assets\images\%imagefilename%.qtc
Now that you have the full path you can use this small tool by NisseDILLIGAF to get the hashed name: http://forum.xda-developers.com/show...&postcount=271
2.5 vs. 2.1 compared - example:
Manila 2.1: \windows\htc\email.mode9
Manila 2.5: \windows\htc\email\email.mode9
In both versions the mode9 has a reference to .\Assets\Images\Email\envelopeback.qtc, but the current directory ".\" is different between 2.5 and 2.1, so that:
Manila 2.1 .\ = \windows\htc\
Manila 2.5 .\ = \windows\htc\email\
So the full file names are:
Manila 2.1: \windows\htc\Assets\Images\Email\envelopeback.qtc
Manila 2.5: \windows\htc\Email\Assets\Images\Email\envelopebac k.qtc
And after the hash:
2.1: 77FB7FAD_manila
2.5: 5C5B9C11_manila

You probably have a lot of these already but you never know,
http://rapidshare.com/files/261745557/m9editor.names.txt
I'm curious what you been by an automated way of naming? What method do you use do find the names? Anyway hope this helps, 12

Thanks 12aon, there are 57 new luac files that I didn't have. I'll update the first post.
To find the names I converted all the mode9 files to xml (using sztupy's kitchen) and then wrote a script for UltraEdit to find and copy any paths ending with .luac or .qtc from the mode9.xmls.
I've attached the script below if anyone needs it (only works with UltraEdit).
The process needs some manual intervention in that all collected paths start with ".\" or "..\". The "..\" is always "\windows\htc", but the ".\" is variable - e.g. for all paths found in "\windows\htc\home\%anyfile%.mode9" the .\ needs to be replaced with "\windows \htc\home\" while for "\windows\htc\people\%anyfile%.mode9" the .\ should be "\windows\htc\people\" etc.
I'm trying to do something similar to get the names from lua files.
EDIT: find_in_mode9.zip was attached here. It's redundant. See MNF in first post instead.

Very savvy! Smart, I went the long manual way and though, screw that!

Just finished searching the lua scripts: got 297 valid luac file names. First post updated.
I attached the UltraEdit script I used for the search, in case anyone might need it.
Funny thing is the search turned up more results (322 total) but they couldn't all be matched to manila files. Not sure if this is due to the pre-release state of Manila 2.5 or some bugs in my search script.
Anyway, 93.39% of lua scripts accounted for so I'm happy with that
EDIT: find_in_lua.zip was attached here. It's redundant. See MNF in first post instead.

you should run these on the 2.1 manila also there are a couple of names still missing there too, 12

Just went through the Manila 2.1 files. Got about 190 new names. Everything is attached in first post.
Also, I think I have a good way to reconstuct the Manila 2.5 xml names. Should have the script done soon.

thanks a lot cookiemonster
was looking for a way to fix pixellated backgrounds in leo roms
u r simply great !!
keep up the good work

what i do!
i found that adding a .\home for any file in the home,mode9 files gives the right name..so also in internet.. .\internet
same in others.
i open my files in m9editor add them manually and make my mods.
so the files i s ok then

Got 1220 xmls named. New list up.
As before the UltraEdit script I used is attached below. This one is a little different then the last two. Because the xmls aren't named specifically in any file, the script reconstructs the name from within the xml and then gets the path from a list of mode9 files. These scripts really aren't as refined as they should be, but they got the job done.
The qtc files are still missing quite a few names, but I think that this is mostly because Manila 2.5 i still in development i.e. most of the artwork is created but not yet implemented.
EDIT: reconstruct_xml.zip was attached here. It's redundant. See MNF in first post instead.

GREAT WORK!!!!... I have been looking for something like this. It is wonderful to see it here... and hopefully it will be turned into a wiki or somethin'.
Great Work,
SoBBie
does anyone know where to get the actual 2.5 oem packages for manila? I am playing with the beta from the XOR ROM.

OK Found a Leo Update with a manila version. dont think it is 2.6 but it is at least 2.5. I was tryingt o go through and convert all the paths to manila names in an excel (I would be happy to share if someone wants it, it is not complete yet) but I noticed that there are MANY MANY VGA files already there. even luac files like \windows\htc\Weather\Scripts\Weather\WeatherGizmo_VGA.luac
Does this mean that HTC anticipated this being ported to VGA and gave us a head start? That would be GREAT!!!!!!!!!!... Anyway any answer to this would be great... I am going to start by looking through the VGA stuff. I do not see a home_VGA file but not sure yet.
Thanks,
SoBBie

supersobbie said:
OK Found a Leo Update with a manila version. dont think it is 2.6 but it is at least 2.5. I was tryingt o go through and convert all the paths to manila names in an excel (I would be happy to share if someone wants it, it is not complete yet) but I noticed that there are MANY MANY VGA files already there. even luac files like \windows\htc\Weather\Scripts\Weather\WeatherGizmo_VGA.luac
Does this mean that HTC anticipated this being ported to VGA and gave us a head start? That would be GREAT!!!!!!!!!!... Anyway any answer to this would be great... I am going to start by looking through the VGA stuff. I do not see a home_VGA file but not sure yet.
Thanks,
SoBBie
Click to expand...
Click to collapse
I've noticed a lot of lua scripts in both Manila 2.5 and 2.1 that have some form of the following:
Code:
if _application.Store:GetStringValue(Lifetime_Permanent, "VGAManila") == "true" then
VGALayout()
About 90 lua scripts in Manila 2.1.38680 contain "VGAManila", and about 80 have it in Manila 2.5.duttysLeoR1.
HTC is making a new VGA device (Tachi) so that's probably the reason. Should make porting to VGA as easy as switching "VGAManila" to "true".

Co0kieMonster said:
I've noticed a lot of lua scripts in both Manila 2.5 and 2.1 that have some form of the following:
Code:
if _application.Store:GetStringValue(Lifetime_Permanent, "VGAManila") == "true" then
VGALayout()
About 90 lua scripts in Manila 2.1.38680 contain "VGAManila", and about 80 have it in Manila 2.5.duttysLeoR1.
HTC is making a new VGA device (Tachi) so that's probably the reason. Should make porting to VGA as easy as switching "VGAManila" to "true".
Click to expand...
Click to collapse
Co0kieMonster GreatNews!!!!!!!! There are now 2 people that love you in my household! of course one is a 2 yr old ;P Thanks I think I can take a look at some of those to see if I can port some of this to 2.1 maybe :\ it has been tough.
Thanks,
SoBBie
EDIT:
OK Now on to actually decompiling.... The people in this thread seem to be some of the best decompilers so I thought I would as the question here. I am trying to play with the AgendaView lua File 1ECF3290_manila. I downloaded the latest kitchen from 12 (3.00.06). I copy that file to the Tools folder and run the following two commands:
luadec 1ECF3290_manila > 1ECF3290_manila.lua
compare 1ECF3290_manila 1ECF3290_manila.lua 1>1ECF3290_manila.log.txt 2>1ECF3290_manila.error.txt
I end up with an empty error text and a log file that ends with:
Opcodes in original: 30
Same lines in both files: 0 0%
Same opcodes in files: 0 0%
Different
Global:
Opcodes in original: 1527
Same lines in both files: 0 0%
Same opcodes in files: 0 0%
How? does the luadec not work with 2.5 yet? am I doing somethin wrong? I, for some reason can not use the 01_Decompile.cmd (which is a separate conversation) so I took the actual commands from there and ran them manually. I could understand a few but EVERYTHING is different? Any help would be greatly appreciate. if this is the wrong place to have the discussion then please pm me.

Co0kieMonster said:
I've noticed a lot of lua scripts in both Manila 2.5 and 2.1 that have some form of the following:
Code:
if _application.Store:GetStringValue(Lifetime_Permanent, "VGAManila") == "true" then
VGALayout()
About 90 lua scripts in Manila 2.1.38680 contain "VGAManila", and about 80 have it in Manila 2.5.duttysLeoR1.
HTC is making a new VGA device (Tachi) so that's probably the reason. Should make porting to VGA as easy as switching "VGAManila" to "true".
Click to expand...
Click to collapse
Yes a tachi with landscapemode will be released but I don't if it's going to be 2.1 or 2.5/6. In the tachi files the code to switch between VGA and WVGA were two register entries, might be the same for the leo files.
supersobbie said:
Co0kieMonster GreatNews!!!!!!!! There are now 2 people that love you in my household! of course one is a 2 yr old ;P Thanks I think I can take a look at some of those to see if I can port some of this to 2.1 maybe :\ it has been tough.
Thanks,
SoBBie
EDIT:
OK Now on to actually decompiling.... The people in this thread seem to be some of the best decompilers so I thought I would as the question here. I am trying to play with the AgendaView lua File 1ECF3290_manila. I downloaded the latest kitchen from 12 (3.00.06). I copy that file to the Tools folder and run the following two commands:
luadec 1ECF3290_manila > 1ECF3290_manila.lua
compare 1ECF3290_manila 1ECF3290_manila.lua 1>1ECF3290_manila.log.txt 2>1ECF3290_manila.error.txt
I end up with an empty error text and a log file that ends with:
Opcodes in original: 30
Same lines in both files: 0 0%
Same opcodes in files: 0 0%
Different
Global:
Opcodes in original: 1527
Same lines in both files: 0 0%
Same opcodes in files: 0 0%
How? does the luadec not work with 2.5 yet? am I doing somethin wrong? I, for some reason can not use the 01_Decompile.cmd (which is a separate conversation) so I took the actual commands from there and ran them manually. I could understand a few but EVERYTHING is different? Any help would be greatly appreciate. if this is the wrong place to have the discussion then please pm me.
Click to expand...
Click to collapse
No I think you overlooked something, are you using my kitchen? In my kitchen there should be an error log that has a line that compare.exe can't get passed. Sometimes it's as easy as just removing a double end statement and sometimes you gotta dig into the disassembly output to see what the function in the script is actually trying to do. Worst case you'll be up against conditional statements. You just gave me a new idea to simplify my kitchen I'll give the output files after decompiling in my kitchen a number that identifies the order in which to handle them, good luck, 12

This should make searching for new names a bit easier
See first post for download.
[APP] Manila Name Finder:
This started out as just some scripts to help me search for names, but has evolved into a proper little program. It can search decompiled mode9 and lua scripts for manila names as well as reconstruct names from localization xmls.
It's not really the most user friendly thing, but it works. The included ReadMe.txt has some basic instructions for use. This tool is also included as part of this Manila Kitchen and that really is the best and easiest way to use it.
I'm also posting the source code. If a future version of Manila breaks the naming scheme again like Manila 2.5, this finder will need to be updated. I'll probably update it but in case I'm MIA someone will hopefully find this source code useful.

thanku very much!

Quick question:
Does the actual filename of the xxxxxxxx_manila relate to the name too? i.e. the xxxxxxxx number is a hexadecimal HASH of the internal name?
I'm asking because I'm writing some manila editing tutorials and at some point I will be creating manila files and I'd like to name them correctly (also without using another name already in use elsewhere).
EDIT: Ok the aptly named manilaHASH tool answered that nicely for me when I tried it. \windows\iamafish -> 557FE52A_manila
Thanks. Excellent work by the way.

and what about names started with " .\..\" or "..\..\" ?

Think I may have found something. I've been trying to port the MyFaves Page from Sense 2.1 to Sense 2.5. Been having problems because of the whole renaming issue. So, I edited the 26948339_manila & ManilaFull.xml to use the old Mode9 filepaths for the Page & it's icon. It loaded but the icon was missing. So i copied in the images from Sense 2.1 & it used them. I think the whole renaming thing is linked to those xml's if you use the old Mode9 filepaths, it looks for the old image files paths as well.
example:
When I used;
Code:
<Page Order="1" Name="myfaves.page" PackageName="HTC" Title="MyFaves" >
<ComponentReference Name="page" Mode9Path="[COLOR="#ff0000"]HTC\Myfaves\MyFaves.mode9[/COLOR]" Component="GizmoRoot" SmartComponent="true" />
<ComponentReference Name="icon_normal" Mode9Path="[COLOR="#ff0000"]HTC\Manila\myfavesicon.mode9[/COLOR]" Component="MyFaves_Off" />
<ComponentReference Name="icon_selected" Mode9Path="[COLOR="#ff0000"]HTC\Manila\myfavesicon.mode9[/COLOR]" Component="MyFaves_On" />
<ComponentReference Name="icon_preview" Mode9Path="[COLOR="Red"]HTC\Manila\myfavesicon.mode9[/COLOR]" Component="MyFaves_Preview" />
</Page>
it used the new Filepath Hashnames (\Windows\HTC\MyFaves\Assets\Images\MyFaves\VGA\"ImageName".qtc).
But, if I use;
Code:
<Page Order="1" Name="myfaves.page" PackageName="HTC" Title="MyFaves" >
<ComponentReference Name="page" Mode9Path="[COLOR="#ff0000"]HTC\MyFaves.mode9[/COLOR]" Component="GizmoRoot" SmartComponent="true" />
<ComponentReference Name="icon_normal" Mode9Path="[COLOR="#ff0000"]HTC\myfavesicon.mode9[/COLOR]" Component="MyFaves_Off" />
<ComponentReference Name="icon_selected" Mode9Path="[COLOR="#ff0000"]HTC\myfavesicon.mode9[/COLOR]" Component="MyFaves_On" />
<ComponentReference Name="icon_preview" Mode9Path="[COLOR="#ff0000"]HTC\myfavesicon.mode9[/COLOR]" Component="MyFaves_Preview" />
</Page>
It used the old Sense 2.1 Filepath Hashnames (\Windows\HTC\Assets\Images\MyFaves\VGA\"ImageName".qtc).
What this all basically means is, ".\" didn't change meaning. It means from the Mode9's filepath. So if Mode9 is in "\Windows\htc\", than ".\" = "\Windows\htc\". If the mode9 is in "\Windows\htc\Myfaves\", than ".\" means "\Windows\htc\Myfaves\".
EDIT: This does NOT seem to effect Script Filepaths...

[App] RGUber v1.21a, RGUOrder v1.4

Purpose:
This purpose of this VBScript is to process and organize data in RGU/REG files to remove duplicates, identify faulty entries, and move entries to ascending alphabetical order (the same way it is displayed in a registry editor).
Requirements:
Windows Scripting Host (included in most versions of windows)
rgucomp.exe and cereg400.dll located somewhere in your path (same folder as the script probably won't work if the script is run from another folder)
.reg and .rgu files are expected to be UTF-16LE with BOM
Usage:
Drag a .rgu, .reg, or .hv onto RGUber.vbs OR run "wscript.exe RGUber.vbs example.rgu"
Details:
When an rgu|reg file is specified, RGUber will:
1) create backup of input file
2) rename input file to boot.rgu
3) use rgucomp to convert it to *.hv
4) use rgucomp to convert new .hv to original rgu path\name
5) Reorder all keys in ascending alphabetical order and all values for each key in ascending alphabetical order with default value first
When an hv file is specified, RGUber will:
1) use rgucomp to convert it to *.rgu
2) Reorder all keys in ascending alphabetical order and all values for each key in ascending alphabetical order with default value first
Options
Open RGUber.vbs in your favorite text editor. All options are set at the beginning with (hopefully) meaningful descriptions.
Code:
'//Path to rgucomp (leave this as default if rgucomp.exe is located in your system path)
Const RGUCOMP = "rgucomp.exe"
'//Path to notepad, only needed if %OPENAFTER% is true
Const NOTEPAD = "notepad.exe"
'//The following options can be set to True/False or 0/1
'//Organize registry entries in ascending alphabetical order
Const REORDER = True
'//Open in %NOTEPAD% after conversion is done
Const OPENAFTER = False
'//Save any errors from rgu -> hv conversion
Const LOGERRORS = True
'//Save a backup copy of %INPUTRGU% as "%INPUTRGU%_Backup.rgu"
Const BACKUPRGU = True
Other info
If target file already exists, RGUber will ask if you want to overwrite.
Text files (the MS way) typically contain CRLF for next line. Output from rgucomp.exe contains many CRCRLF. RGUber removes the extra CR.
I have very few comments in the code. If requested, I will upload another copy with as many detailed comments as I can manage.
I tried to code this as efficiently as VBScript can possibly be. I kept getting errors when trying to run 'rgucomp.exe -b -nologo' so instead of running it directly, RGUber creates a bat file, executes it, then deletes it.
On my AMD Phenom 9600 with Vista64 and 3 SATA in Raid5, RGUber completes rgu->hv->rgu of 2084 lines in <3s
RGUber always saves output from rgu->hv conversion but deletes the file if there were no errors.
RGUber crashes on files with no reg entries (e.g. empty app.reg in an EXT package that does not add any registry entries)
Changelog:
v1.21a
Values are now sorted in alphabetical order for each key
v1.2
Replaced Organize function with one from RGUOrder
Lost ability to reorder values for each key (To be readded in next version)
v1.1
Fixed a bug with removing hashdata from output (RGUber would mix data from two keys under one)
Changed sorting algorith with a much faster one
v1.02
Added option to remove RegistryUpdate key from rgucomp output
Fixed a typo where RGUber was not removing the system attribute from input rgu files
v1.01
Fixed typo where RGUber was waiting for backup file instead of log file
Change 'Done' msgbox to one that shows beginning time and ending time
v1.0
Initial Release
RGUOrder v1.4
This script will only reorder the contents of an rgu without processing with RGUComp, thereby keeping comments and delete key entries. RGUComp/cereg400.dll are not needed to use this app.
Changelog:
v1.4
Fixed a bug where if the original rgu did not end with a new line then the last entry after being sorted would be lost.
Fixed two bugs where only the first 25 tabs and first 25 spaces would be removed before sorting (This did not affect data integrity or performance, but the checksum would be different each time you run the output back through RGUOrder until all the original tabs/spaces were removed)
Added code to prevent multiple entries of the same key from being reordered
Fixed other miscellaneous bugs/oddities introduced with v1.3
v1.3
Added code to add a delete key for each subkey of a deleted key so that when reordered, the key deletion isn't broken
v1.2
Fixed a bug where the last key processed was being concatenated to another with no CRLF producing an invalid rgu file

I'm not sure if this relates to your app but I have a small question:
If a dumped a rom (raw, not kitchen type) and removed several apps/programs but did not clean the registry (very tedious), will this help me clean it up (remove dead paths, etc)?
And if so, how will it know just by dragging the .hv file? I mean how will your app know if a registry entry does not have the app/program included in the rom anymore?
Please forgive me if my question does not relate to your app

There is no way for my app to know, it isnt that smart
It would take an extensive app/database to know which keys are related to which apps.

Thanks for this post

updated to v1.1
v1.02 had a bug in the code which removed hash data from output which made it mix data from the key before it with the key after it
If I ever get around to updating again, I will use hvedit instead of rgucomp

I get an error.
Script: D:\RGUber.vbs
Line: 136
Char: 2
Error: File not found
Code: 800A0035
Source: Microsoft VBScript runtime error
Any reason why?
I attach the file i want to sort alphabetically.

I have no idea
It worked for me with no problem (file attached)
Please tell me the location of RGUber.vbs and of 51329f91-0017-4364-bcff-e032c5d45b01.rgu

Great application bro!!
Only limitation is that I have to put reg400.dll and rgucomp in C:\windows

c_shekhar said:
Great application bro!!
Only limitation is that I have to put reg400.dll and rgucomp in C:\windows
Click to expand...
Click to collapse
yeah, I tried to get around that but I didn't find anything feasible with vbscript :-/
Actually, they don't have to go in C:\windows
I reinstall windows regularly so I keep as many apps portable as I can. I have a bin folder on another partition that I add to the system path variable after a new install for stuff like this.

selyb said:
yeah, I tried to get around that but I didn't find anything feasible with vbscript :-/
Actually, they don't have to go in C:\windows
I reinstall windows regularly so I keep as many apps portable as I can. I have a bin folder on another partition that I add to the system path variable after a new install for stuff like this.
Click to expand...
Click to collapse
can you elaborate this a bit more. Because I too would like a similar arranfements...

My C:\ partition has Vista64
My F:\ partition has all my documents, downloads, music, movies, etc and a folder F:\bin\
F:\bin contains >100 downloaded command line programs and vbs scripts that I have written including
RGUber.vbs
lame.exe
rgucomp.exe
cereg400.dll
FixVTS.exe
faad.exe
nuerecmod.exe
Tag.exe
find Advanced System Properties (I can't remember how, it's different for XP/Vista/7) go to the Advanced tab and hit the Environment Variables button
Under system variables, scroll down to 'Path', double click it. This defines your 'system path'. It contains a list of folders separated by semicolon ";". At the end, add a semicolon and the path to the folder you want to add (e.g. ;F:\bin) after that, hit ok. XP may need to reboot to reflect the change but I'm not sure. Vista and 7 are affected immediately.
With this setup, you can open a command prompt in any folder on your computer and type "RGUber.vbs xyz.rgu" and it would work as if all the files are in that folder.

Thanks a lot bro!!!
I am grateful...

I'd really like to use this, but unfortunately I get this error regardless of the app.reg I drag onto the script:
Script: C:\RGUber\RGUber.vbs
Line: 232
Char: 3
Error: The system cannot find the path specified.
Code: 80070003
Source: (null)
Thanks if you can advise.

Quetzecotyl said:
I'd really like to use this, but unfortunately I get this error regardless of the app.reg I drag onto the script:
Script: C:\RGUber\RGUber.vbs
Line: 232
Char: 3
Error: The system cannot find the path specified.
Code: 80070003
Source: (null)
Thanks if you can advise.
Click to expand...
Click to collapse
Hmmm... this line asks the system for what is in the %temp% variable and attempts to change the working directory to the result.
Open RGUber.vbs in notepad and go to line 232
Modify
Code:
WSH.CurrentDirectory = WSH.Environment("SYSTEM")("temp")
to
Code:
WSH.CurrentDirectory = "C:\RGUber\"
then try again

Works great after your fix, selyb. Thank you for this useful app and your many helpful contributions to the Kaiser forums.

Quetzecotyl said:
Works great after your fix, selyb. Thank you for this useful app and your many helpful contributions to the Kaiser forums.
Click to expand...
Click to collapse
Yeah, I may relocate from Kaiser forums to Rhodium. I have an AT&T Tilt 2 in the mail to me ATM

Grats on getting a Rhodium. Found a question after using it for a while. This is just one example of such behavior, but why does:
Code:
[HKEY_CURRENT_USER\Software\HTC\TaskManager\ExclusiveList\System]
"CMBandSwitching.exe"=dword:0
get turned into:
Code:
"CMBandSwitching.exe"=dword:0
How do I make it regard CURRENT_USER keys?

Quetzecotyl said:
Grats on getting a Rhodium. Found a question after using it for a while. This is just one example of such behavior, but why does:
Code:
[HKEY_CURRENT_USER\Software\HTC\TaskManager\ExclusiveList\System]
"CMBandSwitching.exe"=dword:0
get turned into:
Code:
"CMBandSwitching.exe"=dword:0
How do I make it regard CURRENT_USER keys?
Click to expand...
Click to collapse
I had this problem with an earlier version. If you are using v1.1 then please attach the original rgu/reg. I have tried and I can't seem to reproduce it since I fixed it already.

Please, replace rgucomp with hvedit . I really need your help because rgucomp doesn't work for me . Thanks in advance .

tomcug said:
Please, replace rgucomp with hvedit . I really need your help because rgucomp doesn't work for me . Thanks in advance .
Click to expand...
Click to collapse
why doesn't rgucomp work? I would be surprised to learn that hvedit will work when rgucomp won't.

How to reconstruct a binary identical I9000XWJP6 kernel image, and more

The idea of this exercise is (at least) to get a stable starting point for kernel development. The thing which is currently missing is a proper working .config. I have reconstructed it using differential analysis and in the process hoped to find which components have actually been activated and to uncover changes (or Easter eggs) in the sources which have not been advertised. Having a working and identical I9000XWJP6 kernel also means that open development can continue from the current official public release. From there the things possible are only limited by your imagination.
The following is a walk-through on how to build the kernel, description of pitfalls that will cause changes in .config to break, and some annotations on discoveries made in the process.
The things you need are:
Mandatory:
- I9000XWJP6 zImage : from your favorite location
- Source tree : opensource•samsung•com the GT-I9000 OpenSource Froyo Update JPM.zip
- Sourcery G++: www•codesourcery•com/sgpp/lite/arm/portal/release1039
- Tweak-Kit : <attached>
Optional:
- Arm enabled GCC and binutils, including a development libbfd.
- Lots of your favorite beverage
The md5sum of the zImage should be: 26e9d5d206baf1515144c6b8de6f10d2
It is critical that the Sourcery G++ version is 2009q3-67.
The Tweak-Kit contains the following components:
Readme.txt - You're reading it
mkvmlinux.cc - convert zImage to vmlinux and extract the init ramdisk image
I9000XWJP6_defconfig - default .config
stamp.patch - set date/time and such to original
style.patch - fix style related warnings
prototype.patch - fix prototype related warnings
error.patch - Recoverable errors
houston.patch - Unrecoverable errors
shadow.patch - Fixate data structures
I9000XWJP6.h - Fixated macros
I9000XWJP6.c - Entry point stubs
SOME THINGS I ENCOUNTERED IN THE PROCESS:
a) What I absolutely did not expect was that I found two different encodings of the build timestamp. I could deduce that the timezone was central Europe. I had the assumption it would be Asia or America.
b) What was to be expected is that the source tree is incomplete. The directories drivers/fsr and fs/rfs are missing. You can still compile the kernel as the missing files are used to build modules. Problems start when you change the config. Doing so will change entry points and data structures and your kernel might die a horrible death when it loads modules who are unaware on these changes. There is a workaround which I will explain later.
c) The weirdest thing I encountered were the functions enable_hlt() and disable_hlt(). The are located deep in the unwind tables, a section not intended for code. I spent many hours trying to figure out how they got there or why but I still have no clue. I found exactly only one way to reproduce this behaviour and it is certainly not due to a typo, accident or ignorance.
d) The kernel is not a production but a debug version. It has nearly all tracking/tracing/debug bells and whistles switched on. If the energy required to maintain the statistics where to emit light, you could use your Galaxy as a Christmas tree. Function profiling is enabled and has a considerable negative effect on performance, code is not optimized for size but speed, and unwind tables have been enabled which are not used. These have a really bad impact on footprint size. I really hope that the same compiler and settings are not used for the Android layer. Changing the config into a production version will not work (and crash) as the non-native modules expect the debugging hooks which will no longer exist. But the same workaround as above can be used.
e) The functionality of the power management domains have been optimized to oblivion due to the excessive placing of code disabling comments in large parts of the clock, power management and mach-aries.c. Maybe because the Galaxy hardware is too different than the evaluation boards, or the hardware is buggy and disabling the code makes it less unstable, or there was just not enough time to get the code working. Anyway, at this moment I have no oversight into what degree the absence of power domains influence battery usage.
f) When I started examining the binary code I was puzzled by snippets of code I could not reproduce. Even worse, I encountered snippets that were just questionable. Unusual instruction sequences, and resister usage. Thinking I bumped into a GCC bug, I started debugging the compiler and even tweaked instruction scheduling weights but with no satisfying outcome. I know that GCC is very stubborn with regard to saving and clobbering registers in/across function calls and the code I saw was just incorrect. I knew a different compiler was used and I suddenly realized that it may be more different than what first meets the eye. The culprit turned out to be Sourcery G++. It is a private maintained branch of GCC for reasons I have not investigated. Even the Sourcery assembler is tainted as it played a nasty trick on me with the enable_hlt/disable_hlt thing. I do not like the code I see and I am aiming into getting the sources stock GCC friendly with a working kernel. However, GCC and Sourcery generate code which seem difficult to mix, but I'm getting closer.
g) Compiler warnings. Many of the Samsung sources generate warnings, something I really dislike. In my opinion a warning is emitted for a piece of code which can be interpreted in several ways, leaving the compiler to choose which. Usually it will choose the wrong one. Most warnings were related to coding style shortcuts, a couple of incorrect function prototype resulting in functions that should return int to return random or falsely ignoring return values. There were also a couple of nasties like deference of uninitialized pointers, accessing out-of-bound data and mixing clock data-structures of different types. Included are a number of patches to fix them.
h) I looked deeper into why GCC and Sourcery won't mix and discovered that they have different implementations with regard to constant definition within enum declarations. Google points to the staring point "GCC bug 30260" where is written that the behaviour of enumeration constants has changed to becoming signed int. I have noticed that even explicit unsigned values will change to signed.
Here is an example of what is going wrong:
Take following declaration
Code:
enum rt_class_t { RT_TABLE_MAX=0xFFFFFFFF }
. GCC will consider RT_TABLE_MAX to be -1, and Sourcery will consider it 4294967295. Now, in net/ipv4/fib_rules.c there is this code snippet
Code:
for (u32 id = 1; id <= RT_TABLE_MAX; id++)
GCC will skip the loop, and Sourcery will have a hard time doing nothing.
There are more examples like calculating the location of physical memory or signed/unsigned comparisons. The compiler switches -fwrapv and -fstrict-overflow might influence things, but it general the behaviour is hardcoded and both compilers have a different flavour. I think it would be wiser to choose the GCC flavour as it is more widespread and thus better tested (and fixed).
i) GCC. I noticed that early versions of kernels compiled with GCC would not start. At first I thought it was because of Sourcery /GCC code generating differences. After a number of buxfixes (in error.patch) I suddenly noticed that the GCC kernel is working. My phone is running a GCC compiled production configured kernel for nearly a week.
j) "Houston, we've had a problem" with the light sensor. One of the compiler warnings brought me to the file drivers/sensor/optical/gp2a.c. There within are located two routines which read the light and proximity sensor. They seem copy-pasted identical, however the sensor value types are different as the proximity value is a char and the light intensity a double. What is more convenient than to simply change the data type of the supplied buffer in the function prototype. Now headache starts as the semantics of the read (and write) call say that the unit size is byte. So returning "1" indicates that only the first byte of the sensor value is copied. Also, there is no bounds/access checking so supplying an invalid pointer to the call will crash the kernel. So, assuming this is all one big mistake, I redesigned the function to do better (see houston.patch) and built a new kernel with it. To my utter surprise my battery charge extended from <24 hours to 2 days and 20 hours.
However... I also noticed that my backlight intensity level was constant at it's lowest although the setting was set to auto. I needed to know how the caller invokes the call, but after an extensive search of internet and android sources it is still something I have not found. Heuristics show that the reading the light sensor is called with a buffer length of 1, and the returned value is only accepted when returning a 1 and that the sensor value type is a double (8 bytes). This is wrong: read() semantics require that you supply a length of 8, and expect a return value of 8. This may be the base of many light sensor issues I found when Googling.
Anyway, I returned the code to it's original faulty behaviour, and being illuminated I disabled the auto backlight intensity and changed it to it's lowest setting to enjoy a longer life between battery charges.
TO CREATE YOUR KERNEL:
1) Prepare a working environment
1a) Unpack Sourcery G++. No installation needed, unpacking is sufficient
1b) Unpack the Samsung sources and cd to the location of the top-level Makefile.
1c) Unpack the zImage and the contents of the Tweak-Kit to the same location
1d) Make sure the zImage is called zImage.I9000XWJP6
2) The ramdisk image is required and can be extracted from zImage.I9000XWJP6
2a) Create an uncompressed image Image.I9000XWJP6
Code:
gcc scripts/binoffset.c -o scripts/binoffset
ofs=`scripts/binoffset zImage.I9000XWJP6 0x1f 0x8b 0x08 0x00 2>/dev/null`
dd ibs=$ofs skip=1 <zImage.I9000XWJP6 | gzip -c -d >Image.I9000XWJP6
2b) The Tweak-Kit includes mkvmlinux which converts the uncompressed binary image into a bfd object. You need an Arm enabled libbfd to get it working. This does not get installed by default so you need to deeplink into binutils. mkvmlinux locates and decodes the kallsym data and econstructs the symbol table. It then uses the values of __initramfs_start/end to extract the initramfs. If you are not bothered with the hassle, just use dd with hardcoded values.
Code:
g++ mkvmlinux.cc -o mkvmlinux -lbfd -liberty -lz [-I and -L that deeplink into binutils]
./mkvmlinux Image.I9000XWJP6 vmlinux.I9000XWJP6 -r initramfs.cpio
or
Code:
dd if=Image.I9000XWJP6 of=initramfs.cpio bs=1 count=2739712 skip=165568
3) Patch date/time and other environmental issues to the moment of original creation
Code:
patch -p1 <stamp.patch
4) Make your computer happy
Code:
# edit Makefile line 184 and update the macro CROSS_COMPILE=
cp I9000XWJP6_defconfig arch/arm/configs/I9000XWJP6_defconfig
make I9000XWJP6_defconfig
make
5) Verify that the kernel is identical
Code:
diff zImage.I9000XWJP6 arch/arm/boot/zImage
AND NOW FOR SOMETHING COMPLETELY DIFFERENT...
Tweaking the configuration will build you a new kernel but when your Galaxy powers on it will either die silently (hang) or experience a horrible death (reboot). The problem is that there are modules built from sources located in the removed directories drivers/fsr and fs/rfs. These modules were compiled with a specific data structure layout and entry points. These will surely change when re-configuring. The way to keep the non-native modules happy is to keep the structures and entry points intact.
The structure layout is influenced by the CONFIG_ macros. The structures can be fixed to reflect the state of the I9000XWJP6 kernel by replacing the CONFIG_ macro's by something that does not change after reconfiguration. For that I use a collection of 'shadow' macro's which have SHADOW_ as prefix. Because the data structures cannot expand, you cannot (easily) enable configure functionality which require extra fields in the data structures. Reducing functionality is highly seldom a problem.
If changing kernel functionality results in removal of entry points, then stubs are required for those entry points needed by the non-native modules
There are automated methods to verify that a new kernel abides to the above constraints. For the data structures the compiler must generate gstabs debug information. This is human readable and includes detailed structure descriptions. This information should be identical across re-configuration. However, the scripts get confused by anonymous structs which are by product of "typedef struct {" constructions. These need to be named, something shadow.patch also does.
The kernel modules have easily-readable symbol tables containing needed kernel entry points. These should all be present in all re-configured kernels. Validation tests that fail emit enough information to further fix data structures and entry points. The Tweak-Kit contains two files: I9000XWJP6.h containing the SHADOW_ macro's and I9000XWJP6.c for the stubs. Both were constructed in an on-demand basis for the reconfiguration I am currently using and both serve as good examples on what to do when validation fails.
Before reconfiguring, rebuild the kernel for usage as a validation checkpoint.
1) Undo the timestamp patches
Code:
patch -R -p1 <stamp.patch
2) Fix the warnings
Code:
patch -p1 <style.patch # style related issues
patch -p1 <prototype.patch # prototype related issues
patch -p1 <error.patch # bug fixing
3) Apply datastructure fixation, entrypoint stubbing and Makefile tweaking
Code:
patch -p1 <shadow.patch
cp I9000XWJP6.c arch/arm/plat-samsung
4) Before recompiling everything, you need to issue "make clean" first. However, the missing directories will now pose a problem as "make clean" will include their Makefiles and will fail if it can't. Just create empties to keep the build happy.
Code:
mkdir -p drivers/fsr fs/rfs
touch drivers/fsr/Makefile fs/rfs/Makefile
5) Optionally change the Makefile to point to your favorite compiler/toolchain. Please note that I am using GCC 4.4.5. GCC 4.5.1 is bumping into problems I haven't looked into yet.
Code:
# edit Makefile line 184 and update the macro CROSS_COMPILE
6) This build will generate gstab debug information. Unexpectingly, this might bite when combined with function profiling, so disable that. But do not CONFIG_FUNCTION_TRACER yet as that does more.
Code:
# edit Makefile line 553, disable the line containing KBUILD_CFLAGS += -pg
7) Unpack the initramfs image. The directory /lib/modules needs to be examined/updated
Code:
mkdir initramfs.dir
cd initramfs.dir
cpio -i --make-directories --preserve-modification-time --no-absolute-filenames <../initramfs.cpio
cd ..
8) Repack initfs as a tarball, as make clean will erase all the modules
Code:
tar cf initramfs.tar initramfs.dir
9) The initramfs image will contain new kernel modules. Make sure a new version will get generated.
Code:
# in .config line 80 point to the unpacked initram location
CONFIG_INITRAMFS_SOURCE="initramfs.dir"
# in .config lines 86-89, select your favourite compression
CONFIG_INITRAMFS_COMPRESSION_NONE=N
CONFIG_INITRAMFS_COMPRESSION_GZIP=Y
10) Build a new kernel
Code:
# not cleaning will confuse the verification
make clean
make CONFIG_DEBUG_INFO=y
# install the modules
tar xf initramfs.tar
cp `find drivers -name '*.ko'` initramfs.dir/lib/modules
# rebuild with a fresh new ramdisk image
rm usr/initramfs_data.cpio*
make CONFIG_DEBUG_INFO=y
11) Checkpoint structure/entrypoint information. This is architecture independent.
Code:
# extract structures. They are the entries with :T
objdump -G vmlinux | awk '{ print $7 }' | grep :T | sed 's/([^)]*)/()/g' | sed 's/=\*()//g' | sort -u > gstabs.ckp
# extract the entrypoints
nm vmlinux | grep 'r __ksymtab_' | awk '{ print $3 }' | sort > ksymtab.ckp
12) Do a test-run. Pack zImage and flash with Odin. If your Galaxy is up and running, I strongly suggest you make a backup of your environment. If you later change something and it breaks, then this is the best place to restart.
Code:
cp arch/arm/boot/zImage .
tar cf I9000XWJP6-2.6.32.9-test.tar zImage
13) Make your re-configuration. I really suggest you do not make too many changes in one go because it gives more work when the structure/entrypoint verification fails.
Code:
# re-configure. For this exercise, change the kernel to a more production version
CONFIG_CC_OPTIMIZE_FOR_SIZE=Y
CONFIG_DM_DEBUG=N
CONFIG_S3C_KEYPAD_DEBUG=N
CONFIG_DEBUG_FS=N
CONFIG_DEBUG_KERNEL=N
CONFIG_LATENCYTOP=N
CONFIG_FTRACE=N
CONFIG_ARM_UNWIND=N
CONFIG_DEBUG_USER=N
14) Build a new kernel
Code:
# not cleaning will confuse the verification
make clean
make CONFIG_DEBUG_INFO=y
# install the modules
tar xf initramfs.tar
cp `find drivers -name '*.ko'` initramfs.dir/lib/modules
# rebuild with a fresh new ramdisk image
rm usr/initramfs_data.cpio*
make CONFIG_DEBUG_INFO=y
You get "struct has no member named" errors if you have enabled subsystems that require data structures to change which are incompatible with the non-native modules.
15) Verify structure/entrypoint checkpoint
Code:
# extract/verify structures
objdump -G vmlinux | awk '{ print $7 }' | grep :T | sed 's/([^)]*)/()/g' | sed 's/=\*()//g' | sort -u > gstabs.t
# new/changed structures are tagged with '+'. Display only the changed ones
diff -U0 gstabs.ckp gstabs.t | grep '+' | grep ':T' | sed 's/+//' | sed 's/:T.*/:T/' | while read s; do
grep -q "$s" gstabs.ckp
if [ $? -eq 0 ]; then
echo $s;
fi
done
# extract/verify entrypoints
nm vmlinux | grep 'r __ksymtab_' | awk '{ print $3 }' | sort >ksymtab.t
# extract all entrypoints needed by the modules
nm `find initramfs.dir/ -name '*.ko'` >allkosym
# some symbols are referenced in other modules. Cross-reference and remove from list
grep ' U ' allkosym | sort -u | awk '{ print $2 }' | while read s; do
if ! egrep -q "^[0-9a-f]* . $s\$" allkosym; then
echo $s;
fi
done > allusym
# check that kernel has entrypoints for all final undefined symbols
cat allusym | while read s; do
if ! egrep -q "^__ksymtab_$s\$" ksymtab.t; then
echo $s;
fi
done
Both scripts will generate output if you have enabled subsystems that require datastructures to change. These do not necessarily have to be data structures needed by non-native modules. However, missing entrypoints are those used by the modules. If it's about datastructures, your best chance is to lookup the data type and see if there any #ifdef CONFIG_ macros that need to be changed into #ifdef SHADOW_. If it's a missing entry point, you need to add a stub in I9000XWJP6.c.
16) Do a test-run. Pack zImage and flash with Odin.
Code:
cp arch/arm/boot/zImage .
tar cf I9000XWJP6-2.6.32.9-test.tar zImage
17) If you want more, jump to step 13
18) When you are really done, rebuild a final and fresh kernel and initramfs image with debugging stuff removed. The -gstabs compiler switch slightly influences code generation.
Code:
# not cleaning will confuse the verification
make clean
make
# install the modules
tar xf initramfs.tar
cp `find drivers -name '*.ko'` initramfs.dir/lib/modules
# rebuild with a fresh new ramdisk image
rm usr/initramfs_data.cpio*
make
My uncompressed image has now shrunk from 14700623 to 11822559 bytes.
Happy Hacking...
[...and now to find a better workaround for those non-native modules.]

WoW, Nice work !! very good info for beginners like me

thx a lot for this tut and i've learnt a lot
btw, seems there r some typos or something is missing. i did it with (XXJPO):
Hexabit said:
make I9000XWJP6_defconfig
Click to expand...
Click to collapse
make defconfig I9000XWJP6_defconfig
- modified include/linux/a.out.h by removing the 2nd def for SEGMENT
- changed the boolean to lowercase for .config
PS i use the cpio extracted by myself coz i couldnt enable libbfd on my ubuntu x64

Good tips. Thanks.

Really insightful i hope the dev take all the tweaks into consideration and make a new optimized kernel

good job here!

I think it's the most amazing first post ever! It should be sticked or kept somewhere safe.

Awesome first post. Will have to work through this.

Great post with very interesting findings!
I'm no expert, so maybe my question is a bit silly:
Is Samsung's published code just a buggy and incomplete pre-release debug version? Then how can e.g. Voodoo get a good working kernel?
Or is the official firmware really built of this, so possibly full of strange bugs and missing optimizations?