Related
Update November 11, 2009
CFC and CFC GUI have been updated to 0.60 and 0.60.35 respectively. CFC runtime files have been updated. This thread has also been update to add information about the new file formats.
Changes:
- (CFC GUI) Support for new filenames (and categories) used in TF3D v2.5+
- Support for new file formats (read only, "replace" write is always in old format) used in TF3D v2.5+
- Support for new compressed formats that correspond to the new formats mentioned above
- (CFC GUI) If you are using this with 2.5, please do read the 2.5 specific notes (in the CFC GUI thread).
- At this time I STRONLY recommend patching Manila files manually instead of using the auto-patching! See the tech thread referenced below.
Update June 5, 2009
CFC and CFC GUI have been updated to 0.55 and 0.55.25 respectively. The CFC runtime files have not been updated. Changes:
- Better compression ratio (backwards compatible with old runtime)
- UltraHQ en/de-coding support (CFC)
- "Patch Manila on device" function has been improved to work-around the issue of Manila no longer starting that some people have (CFC-GUI)
Update April 20, 2009
CFC and CFC GUI have been updated. Many changes! For the changelog see the CFC GUI thead. The CFC runtime has also been updated to v0.50, so there is a new libgles_mn file required. See the cfc-support-dll zip attached a few posts down.
Update Feb 26, 2009
CFC and CFC GUI have been updated to 0.46.15. Fixed an issue with PNGs without alpha channel.
Update Jan 15, 2009
CFC Live Patch 0.45.01 has been released. You can run this tool on your phone to make your current Manila installation compatible with CFC compression.
See this post.
Update Jan 14, 2009
CFC GUI 0.45.15 released. No changes to CFC core, only GUI stuff. Attached download updated.
CFC GUI thread: http://forum.xda-developers.com/showthread.php?t=470798
Changelog: http://forum.xda-developers.com/showpost.php?p=3164604&postcount=3
Update Jan 13, 2009
CFC 0.45 and CFC GUI 0.45.12 released.
CFC (core) changes:
- Heavily modified encoding algorithm. It is often slower but the quality should be much better. Please read this post!
CFC GUI changes:
- No longer freaks out if the wrong file attributes are set on some files
- Added background color selection
- Added tool to patch a complete Manila package for CFC compatibility and optimization
Also, non-technical discussion of CFC GUI (only) should go to this thread:
http://forum.xda-developers.com/showthread.php?t=470798
Update Jan 12, 2009
Yes I know, too many updates today!
CFC 0.42 attached. Was something weird going on with the encoding sometimes. Furthermore it seems like the encoding works great on the original files, but it drops the ball here and there with other files. Going to look into that ASAP and finetune the encoding algorithm!
Most important about this update is, CFC now also comes with a GUI!
Update Jan 12, 2009
CFC has been updated to version 0.4. Added features are:
- QTC -> PNG (+- 30% faster than 0.4b2)
- PNG -> QTC
- CFC -> QTC (finally)
- QTC padding
- QTC trimming
With QTC <-> PNG conversion now available from CFC, it seems the Compressonator is no longer needed!
Further EDIT: All posts updated.
Further EDIT: 0.41 added... There was a small bug in CFC compression in 0.4, it didn't always set PayloadSize correctly, which could create errors with padding/trimming.
Update Jan 8, 2009
CFC has been updated to version 0.3. It can now fully handle the RGB format as well. CFC compression has been slightly optimized. QTC and ATC headers are now completely written correctly.
This now also allows for larger than original images
Also lots of info in the first 6 posts has been updated to reflect these changes and add information.
Update Jan 7, 2009
Thanks to myself and D-MAN666, it seems the QTC format is now completely known!
Also today brings CFC compression for (W)VGA devices, if chefs choose to implement it. The needed stuff is here.
The CFC tool itself still needs an update (0.3 ?) to handle QTC/ATC_RGB conversion to ATC/ATC_RGB conversion (and back) and to decompress the CFC files. Donor headers won't be needed anymore then either in some cases (but they will remain handy in others!), and with that some parts of the first 5 posts will have to be rewritten as well (sigh...)
Note that some other parts of the first few posts are marked with changes. Look for the red letters.
- end of updates -
Intro
As some of you may have seen, me, djboo, pcarvalho, "he who shall not be named" and several other enthousiasts have ported TouchFlo3D (to be specific, the version that came on my Raphael) to QVGA. A large part of this effort involved hacking into Manila/Mode9 and even OpenGL ES itself to get it operating decently, but after that, a lot of effort went into optimizing, which was largely done by scaling the images. While working on this, I encovered a lot of information and wrote quite a bit of code to 'get it done'. As always, a lot of it ultimately redundant, but we did pull it off! (barring some issues that are driver related).
Acknowledgements
Before continuing, people need to be acknowledged for their parts in this. I could hardly have done it alone. A lot of these credits go out to people involved in the TF3D QVGA porting, but also drivers porting, they all had a hand in this information being 'discovered', and hence are mentioned here.
- djboo Keeping all this stuff going. I looked at this stuff once when herg did an attempt, never really got involved, but because of him did get involved this time around. As he seems to be forgotten in the credits here and there, he's #1 in this thread!
- pcarvalho A bit of competition that led to great stuff. In the end, our intended methods of porting complement eachother nicely - 'my' part got it going, but the QVGA port didn't really shine 'til we did the other things 'his way' as well.
- "he who shall not be named" The anonymous HTC-CA hacker, about whom probably everybody knows who he is (it aint me ) Did some cracking work on this too.
- The P3D Team A bunch of them did a lot of testing
- D-MAN666 Mentioned last but certiainly not least! Cracked the file format first, and generally found out and published a hell of a lot of information. Also the author of Manila Editor.
Requirements
For all this, you will need and/or want the following:
- "The Compressonator", image conversion/compression/viewer tool by ATI/AMD. You may need to sign up at the AMD website, but it is free of charge and I haven't received any spam from them yet. Update: The version of Compressonator on AMD's site is no longer able to do ATC. The correct version of Compressonator is attached to this post. Update: The Compressonator is now no longer strictly needed due to CFC being able to do the QTC en/decoding.
- "Manila Editor", Manila image editor. You will not be using this for the actual good stuff, but you may be using this for testing things quickly, and you will definitely want to use it for finding the files you actually want to replace. Update: no longer needed, use the CFC GUI instead!
- CFC, (attached), tool by yours truely to convert between QTC, ATC, CFC and PNG formats
- Knowledge of the Windows command line - Though CFC now comes with a GUI as well, yay!
- The files attached to this post. These are all the images from Manila (the version that came with my Raphael) converted to PNG by "The Compressonator". It's kind of a ***** to do, so if you want to save yourself some trouble, just see that post. VGA as well as (rescaled using Lanczos3) QVGA images are attached to that post. Update: this image packs need to be updated, will do some time
CFC download
Notice: you do not need the compressonator files to use CFC. They are just here in case you want to do things the old-school way
( < 0.60 : 3408 downloads )
Textures, ATC, CTES, QTC, CFC (tech background)
The imageon 3D chip in our devices support texture compression, and Manila (Mode9) uses this. The format used is a special format created by AMD/ATI for low size and lower power use on mobile devices, called ATC (ATI Texture Compression).
There are three ATC formats:
- ATC_RGB: 4 bits per pixel (4 bits RGB)
- ATC_RGBA_EXPLICIT_ALPHA: 8 bits per pixel (4 bits RGB + 4 bits Alpha)
- ATC_RGBA_INTERPOLATED_ALPHA: 8 bits per pixel (not sure on the format)
Almost all images used in Manila are of the 'ATC_RBGA_EXPLICIT_ALPHA' variant, and this article will focus on these. ATC_RGB is also used for a small number of images, though I have not further investigated its format.
The image data for these formats are stored in one of the following file formats:
- ATC: The file format generally used by AMD/ATI
- CTES: ATC related, some weirdness, see below. Seems to be forward compatible with ATC, but not backwards.
- QTC: Qualcomm adapted version of ATC, used by Manila/Mode9
The formats are very similar, though we will focus only on ATC/QTC, that's all we need.
Image data (ATC_RGBA_EXPLICIT_ALPHA) - Updated January 12, 2009
A lot of the original information comes from D-MAN666's posts here.
I will skip over the headers (32 bytes), they are listed below for ATC and QTC specifically. This is about the actual image.
The image is divided into blocks of 4x4 pixels. An 8x8 image would be stored like this: (A, B, C and D are 'pixel blocks')
AAAABBBB
AAAABBBB
AAAABBBB
AAAABBBB
CCCCDDDD
CCCCDDDD
CCCCDDDD
CCCCDDDD
A 4x4 pixel block is 16 pixels and 16 bytes. So in effect, you can see it as 8 bits per pixel. An image is always stored using these 4x4 pixel blocks. A 5x5 images would thus use 4 blocks!
bytes 0-7 are alpha values for each pixel, 4 bits per pixel (4 bits * 16 = 64 bits = 8 bytes) - this is not present for the ATC_RGB format. Scale these to the 0..255 range by multiplying each alpha value by 17.
bytes 8-11 are color keys, there are two keys of 16 bits (2 bytes). The keys are stored like this:
word 1: XRRRRRGG GGGBBBBB (1-bit method, 5-bit R, G, B)
word 2: RRRRRGGG GGGBBBBB (5-bit R, 6-bit G, 5-bit B)
Where X is the decoding method to use, there are two.
bytes 12-15 are mixdown values, 2 bits per pixel (2 bits * 16 = 32 bits = 4 bytes). The per-pixel mixdown value, combined with the color keys mentioned above determine the actual color that is outputted. You must scale the scolor keys to the 0..255 range and apply a formula to them.
Code:
if HasAlpha then begin // skip for ATC_RGB
sIn.Read(dw, 4); // read dword
for i := 0 to 7 do begin
alpha[i] := (dw and $F) * 17;
dw := dw shr 4;
end;
sIn.Read(dw, 4); // read dword
for i := 8 to 15 do begin
alpha[i] := (dw and $F) * 17;
dw := dw shr 4;
end;
// alpha[0..15] now contain the scaled 4x4 pixel block alpha values
end;
sIn.Read(w, 2); // read a word, key1
Method := (w shr 15);
Keys[iR, 0] := ((w and $7C00) shr 10) * (255/31);
Keys[iG, 0] := ((w and $03E0) shr 5) * (255/31);
Keys[iB, 0] := (w and $001F) * (255/31);
sIn.Read(w, 2); // read a word, key2
Keys[iR, 1] := ((w and $F800) shr 11) * (255/31);
Keys[iG, 1] := ((w and $07E0) shr 5) * (255/63);
Keys[iB, 1] := (w and $001F) * (255/31);
sIn.Read(mixdown, 4); // read a dword, mixdown values
for i := 0 to 15 do begin
pixels[i] := (mixdown and $3);
mixdown := mixdown shr 2;
end;
// pixels[0..15] now contain the still-encoded 4x4 pixel block mixdown values
When thinking about the color keys and mixdown values, think of the keys as a color-range start and end value. The mixdown values decide which value to pick inside the range. (for each R,G,B)
For example, let's take a key1 of 10 and a key2 of 40 for Green. Then the decoded Green values would be:
Code:
mixdown 00 01 02 03
value 10 20 30 40
This is only true, however, when the 'method' bit (X) is 0. Full decoding of both methods:
Code:
for i := 0 to 15 do begin
a := alpha[i];
if (method = 0) then begin
r := Round( Keys[iR, 0] + ((pixels[i] / 3) * (Keys[iR, 1] - Keys[iR, 0])) );
g := Round( Keys[iG, 0] + ((pixels[i] / 3) * (Keys[iG, 1] - Keys[iG, 0])) );
b := Round( Keys[iB, 0] + ((pixels[i] / 3) * (Keys[iB, 1] - Keys[iB, 0])) );
end else begin
case pixels[i] of
0: begin
r := 0;
g := 0;
b := 0;
end;
1: begin
r := Round( Keys[iR, 0] - ((1/4) * Keys[iR, 1]) );
g := Round( Keys[iG, 0] - ((1/4) * Keys[iG, 1]) );
b := Round( Keys[iB, 0] - ((1/4) * Keys[iB, 1]) );
end;
2: begin
r := Round( Keys[iR, 0] );
g := Round( Keys[iG, 0] );
b := Round( Keys[iB, 0] );
end;
3: begin
r := Round( Keys[iR, 1] );
g := Round( Keys[iG, 1] );
b := Round( Keys[iB, 1] );
end;
end;
end;
end;
Both methods have various way of formulating them. I thought the ways I have chosen here are clearest for how it works.
Update Jan 7, 2009
Image data (ATC_RGB)
The image data here is exactly the same as ATC_RGBA_EXPLICIT_ALPHA, except that the alpha bits aren't includes. So, each 16-pixel block becomes 8 bytes instead of 16, as bytes 0-7 from ATC_RGBA_EXPLICIT_ALPHA are not there. This means 4 bits per pixel.
- end of update -
ATC, CTES, QTC
This image data seems to be the same across all formats - and it should be, as this data is sent directly to the 3D chip. You would not want to have to process it first.
Let's first pick out CTES, as I have very little to say about it. It seems to be nearly the same as ATC and QTC, however, for some reason, "The Compressonator" will output CTES files we can use as ATC, but will not read our own Manila-based ATC's in CTES mode (only in ATC mode). What's up with that? I don't know. Perhaps one of you will figure it out.
QTC header
Code:
Magic: DWORD; // 0x31435451 : "QTC1"
Magic2: DWORD; // always 1 ?
Width: DWORD;
Height: DWORD;
Format: DWORD; // 0x14, 0x15
Dummy1: DWORD; // formerly known as Unknown1, may be 0 - Jan 7, 2009
PayloadSize: DWORD; // formerly known as Unknown2 - Jan 7, 2009
Dummy3: DWORD; // formerly known as Unknown3, may be 0 - Jan 7, 2009
The meaning of the unknowns has not been deciphered yet. Setting them to weird values does muck-up the decoding of the images, however, they do not seem to be actually sent to the 3D chip. Or perhaps I just have not found where and when!
For format, 0x14 is used for ATC_RGB_EXPLICIT_ALPHA. The small number of images that use 0x15, I suspect, are ATC_RGB. Either way, they do not decode using the ATC_RGB_EXPLICIT_ALPHA method and I know ATC_RGB is used some places, so it would make some sense to make this assumption.
Update Jan 7, 2009
Unknown2 has been replaced by PayloadSize, thanks to myself, D-MAN666 and eidolen.
The PayloadSize is the number of bytes after the header that contain content.
For images of type 0x14 (ATC_RGB_EXPLICIT_ALPHA) this is: Width * Height, where both Width and Height are multiples of 4, due to how the format itself works, in other words: (RoundUp(Width / 4) * 4) * (RoundUp(Height / 4) * 4).
For images of type 0x15 (ATC_RGB) this is half of type 0x14, because ATC_RGB uses 4 bits per pixel instead of 8. The multiples of 4 rule still stands, so the PayloadSize is: Round(((RoundUp(Width / 4) * 4) * (RoundUp(Height / 4) * 4)) / 2)
Note that all Manila image files (at least the ones I have) are padded to be a multiple of 512 bytes in size. This is probably a speed optimization for when these files are cooked into a ROM.
Dummy1 and Dummy3 (aptly renamed from Unknown1 and Unknown3) seem to be irrelevant. After we figured out how PayloadSize (Unknown2) was relevant, we tried blanking out Dummy1 and Dummy3 with 0's, and TF3D still works without a hitch. The original values do not seem to be related to the dimensions nor the payload size, and they are not sent to the graphics chip either.
- end of update -
Update November 6, 2009
Manila 2.5 uses 4 additional file formats:
0x01 - 8888 RGBA, 32bpp
0x02 - 888 RGB, 24bpp (I have not encountered an actual image in this format, so processing may be faulty by CFC and CFC GUI)
0x03 - 565 RGB, 16bpp
0x05 - 4444 RGBA, 16bpp
- end of update -
ATC header
Updated 08/Jan/2009
Code:
Magic: DWORD; // 0xCCC40002
Width: DWORD;
Height: DWORD;
Format: DWORD; // ATC_FORMAT, 0x01 for RGB, 0x02 for RGBA_EXPLICIT_ALPHA, 08/Jan/2009
Magic3: DWORD; // 0x20 ... mucks up colors... not clear?
Magic4: DWORD; // 0x01, 08/Jan/2009
Magic5: DWORD; // 0x01, 08/Jan/2009
FormatMagic: DWORD; // 0x8C92 for RGB, 0x8C93 for RGBA_EXPLICIT_ALPHA, 08/Jan/2009
- end of updated content -
CFC format + historic Compressonator editing
CFC format
I use the CFC format (yes, that's why the tool is called CFC) for the Manila QVGA port. It saves a lot of space and even seems to improve performance a bit. It uses standard gzip/zlib compression on the QTC image data (which compresses to about 20% on average) and hides the compressed data inside the QTC file itself. Decompression of this is over 5 MB/s on our devices, but images are only a few KB each. The proxy libgles_cm is what actually decodes this and sends the decompressed data to the 3D chip.
CFC adjusts the QTC header to the proper values. Beware when doing this yourself that Mode9 uses these values internally as well). The image data ('payload') is replaced as follows:
Code:
Magic: DWORD; // 0x31434643 : 'CFC1'
Format: DWORD; // CFC_FORMAT...
Width: DWORD;
Height: DWORD;
CompressedSize: DWORD;
UnCompressedSize: DWORD;
... compressed data ...
Format can be one of the following:
Code:
CFC_FORMAT_COMPRESSED_QTC_RGBA_EXPLICIT_ALPHA = 0x3001; // 0x14 from QTC
CFC_FORMAT_COMPRESSED_QTC_RGB = 0x3002; // 0x15 from QTC - used since CFC 0.3
CFC_FORMAT_COMPRESSED_RGBA = 0x3101; // April 20, 2009 - RGBA format - used since CFC 0.5
CFC_FORMAT_COMPRESSED_RGB = 0x3102; // April 20, 2009 - RGB format - used since CFC 0.5
Width and height are included for historic reasons, and it also opens up the possibility to do some weird mods. RGBA format is included for possibly allowing use of uncompressed textures for Manila support on non-HTC/Qualcomm/ATI/AMD based devices.
Update April 20, 2009
(A)RGB formats are gzip/zlib compressed just as QTC/CFC variants and require the CFC 0.50 runtime files. The uncompressed data is actually stored as (height x width x) BGR(A) (from x86 viewpoint) as this is the format the graphics chip can handle.
Update November 6, 2009
The following formats have been added to CFC. Note that the QTC header always says RGB or RGBA_EXPLICIT_ALPHA. This actually allows the new formats to be used on older Manila versions that do not directly support them, if you are using CFC 0.60 runtimes.
Code:
// CFC 0.60 additional formats
CFC_FORMAT_COMPRESSED_QTC_8_8_8_8 = 0x3003;
CFC_FORMAT_COMPRESSED_QTC_X_8_8_8 = 0x3004;
CFC_FORMAT_COMPRESSED_QTC_5_6_5 = 0x3005;
CFC_FORMAT_COMPRESSED_QTC_4_4_4_4 = 0x3007;
Notice: the text below in this post is here for historic reasons. It is no longer completely relevant
Right, well I did tell you to get "The Compressonator" from AMD's site, right? You should have done this by now. You should also have the CFC tool attached to the first post of this thread.
Say we want to manipulate an image from Manila. First we need to find out which image it is. Easiest way to find them is to use Manila Editor (also linked in first post), so you get the 'magic' filename.
You may want to ask why we dont simply use Manila Editor for doing these things, simply put, the image quality from The Compressonator is better than the current version of Manila Editor. Also, we can do stuff in batch in The Compressonator.
Manila to PNG (single)
Now, say this image is 7d3f1247_manila (the globe on the internet page), we use the CFC tool to convert it to ATC format:
Code:
cfc -qa 7d3f1247_manila 7d3f1247_manila.atc
You can open this file in The Compressonator. It may look a bit weird, because alpha is not displayed. Right click on the image and select "Show RGBA", there, that looks better.
Something you will not directly notice with the globe image, but you will with other images, is that the image is UPSIDE DOWN. You will need to flip the image over if you want to put it back into Manila. For some reason I haven't figured out yet, decoding goes upside down, but encoding needs to be the correct side up.
Now we may wish to edit this file, so we save it as PNG: File -> Save Original.
Open it in photoshop, flip it vertically, and save it.
Manila to PNG (batch)
We will need to use batch mode to convert back to Manila anyways, so lets just start using it for converting it to PNG as well (for some reason doing it non-batch doesn't work right).
This will assume you have a bunch of .atc files in a directory. Batch converting Manila (QTC) files to ATC files is also possible with CFC:
Code:
cfc -qaf orgfiles atcfiles
Assuming you have your original Manila files in the folder 'orgfiles' and created a new empty directory 'atcfiles'.
Open The Compressonator, and go to File -> Batch Compress (or press F4). Navigate to your folder containing all the ATC files, set the "Files of type" box to "ATC Textures (*.ATC...)". set the "Output File Format" to PNG and "Output Format" to "ARGB8888". Punch the "Compress All" button and wait a bit.
Note that some files will not decompress correctly and crash The Compressonator. You will have to look at the crash dump to find out which file was the culprit and remove it from your batch directory. IIRC, there are about 10 files that have this issue, so be prepared for 10 minutes of infuriating work.
You MUST set "Output directory" to "Use Input Directory", or you will not be able to decompress more than one file!
In the end, you will have a large bunch of PNG files. Note that these PNG files are also available already done for you, see the link in the first post of this thread.
Files known to crash The Compressonator: 08/Jan/2009
Code:
00ad7edb_manila.atc
056e5c7f_manila.atc
063f5858_manila.atc
0c175082_manila.atc
2255b55f_manila.atc
24720929_manila.atc
39064485_manila.atc
4a209508_manila.atc
PNG to Manila (single + batch)
Even for single files, we are using the batch function, as there seems to be an issue with doing this in The Compressonator normally.
The operation is exactly the same, but for single you select the file and press "Compress", and for many files you do not select a file and press "Compress All".
Note that as previously mentioned, decoding ATC to PNG files results in the PNG's being upside down, but to make ATC files from PNG files the correct side needs to be up!
This time around we set "Files of type" to "PNG Textures" (duh) and "Output File Format" to "CTES Textures". As previously mentioned CTES is compatible with ATC, but ATC is not compatible with CTES. You won't notice this though.
The magic is the "Output Format" setting. Set it to "CTES Texture Compression" and hit the "Options" button. In the "Compress Texture" dialog that pops up, select "ATC RGBA Explicit Alpha (8 bits per pixel)", or "ATC RGB (4 bits per pixel)", depending on which format you want, hit OK, and you're there. Hit "Compress" or "Compress All", and wait 'til it's done.
You MUST set "Output directory" to "Use Input Directory", or you will not be able to decompress more than one file!
Now we want to convert these CTES/ATC files back to Manila files, and for this, again, we use the CFC tool:
Code:
cfc -aqf atcfolder qtcfolder orgfolder
You can also use -aq instead of -aqf for a single file. Note that the CFC tool does NOT change filenames, so you have some renaming to do.
Update 08/Jan/2009
With CFC 0.3, donor headers are not longer necessary, and have become an optional parameter.
- end of update -
Rescaling to QVGA
Converting Manila images to QVGA is pretty simple. Just use the techniques described above.
What you want to do is scale ONLY images 32x32 and larger, and you will want to divide the width and height exactly by two. That's all there is to it.
If you have a bunch of PNG files you want to scale, the CFC tool can even do this for you, including the needed vertical flip:
Code:
cfc -nsf vgapngfolder qvgapngfolder
This will rescale using the Lanczos3 algorithm.
CFC Compression
The QVGA port supports the CFC format as mentioned above. This can save a lot of space and is the preferred way of using textures for the QVGA port.
To compress your QVGA QTC files to CFC:
Code:
cfc -cf qvgaqtcfolder qvgacfcfolder
CFC tool (commandline)
As CFC offers a lot of options, many of them related to the pre-0.4 way of converting images, I'll take a very short time to explain the most relevant CFC options, what they do, and when/why you should use them. Most of these are available in the GUI as well.
Convert QTCs to PNGs
Code:
cfc -qp in-filename out-filename
cfc -qpf in-folder out-folder
Since CFC 0.4 this should be the preferred way to convert Manila's QTC files to PNG's (it's pretty fast and saves a lot of steps and trouble compared to using other CFC options and the Compressonator). These images come out with correct side up, so no longer is there a need to flip them manually with CFC or Photoshop or whatever.
Note: This does not handle CFC compressed QTC's, you will have to decompress those first!
Convert PNGs to QTCs
Code:
cfc -pq in-filename out-filename
cfc -pqf in-folder out-folder
Obviously if you want to convert QTCs to PNGs there's a good chance you also wish to do the reverse. Introduced in CFC 0.4b2. CFC will automatically detect if RGBA_EXPLICIT_ALPHA or RGB is the most optimal QTC format to use.
Quality rivals the Compressonator, but CFC is quite a bit slower (though the saved number of steps save you more time).
Note: The output are not CFC compressed, you will have to do that manually.
Compress QTCs to CFCs
Code:
cfc -c in-filename out-filename
cfc -cf in-folder out-folder
Compressed QTCs (CFCs) are usually much smaller than the original QTCs, and provide a performance boost as well. However, you will need a patched Manila to be able to use CFCs. The QVGA port is patched to do this, and instructions on how to other Manila versions can be patched are included a few posts below this one.
Decompress CFCs to QTCs
Code:
cfc -d in-filename out-filename
cfc -df in-folder out-folder
This should speak for itself
Trim QTCs
Code:
cfc -t in-filename out-filename
cfc -tf in-folder out-folder
Saves some space on your hard-disk, removes unnecessary data from the QTC files (also works on CFCs).
Pad QTCs
Code:
cfc -p in-filename out-filename
cfc -pf in-folder out-folder
This makes the QTCs (and CFCs) a multiple of 512 in bytes in size. HTC originally did this with all their images. It seems to improve performance when Manila is cooked in.
Scale PNGs VGA->QVGA
Code:
cfc -ps in-filename out-filename
cfc -psf in-folder out-folder
For the theme and Manila porters. Note that Manila in QVGA can handle VGA textures (and vice versa) just fine, however, using the correct image size for the Manila resolution does improve performance quite a bit.
Complete QVGA port example
Updated 13/Jan/2009
You can now do this whole thing with a single push of a button in the CFC GUI: Tools, Scale QVGA -> VGA
- end of update -
Updated 12/Jan/2009
The text below has been updated to reflect changes with CFC 0.4
- end of update -
This is the complete rundown of how I converted all the VGA images from Manila to QVGA images, the same method can be applied to themes and what not.
First, create a directory somewhere, and put the CFC.exe in it. Then go to the command line and change directory to this new directory.
We will want to create a bunch of directories:
1) Setup
Code:
mkdir org png pngscale qtc cfc out
Dump all your original manila files in org (in this case, the entire manila, but you could just dump your theme in there instead)
2) Convert QTC to PNG
Code:
cfc -qpf org png
3) Scale PNG images from VGA to QVGA
Because CFC will not scale very small images, we need to make sure the output folder (pngscale) has all the images we want first. The ones that are scaled will be overwritten by CFC:
Code:
copy png\*.png pngscale\*.png /y
Do the scaling:
Code:
cfc -psf png pngscale
4) Convert PNG back to QTC
Code:
cfc -pqf pngscale qtc
5) Convert QTC to CFC (optional)
Code:
cfc -cf qtc cfc
6) Pad QTC/CFC (optional, for cooking)
Code:
cfc -pf cfc out
2-6 in one go:
Note that this does not include making the directory and placing your Manila in the org folder.
Code:
cfc -qpf org png
copy png\*.png pngscale\*.png /y
cfc -psf png pngscale
cfc -pqf pngscale qtc
cfc -cf qtc cfc
cfc -pf cfc out
--
Et voila, most of our images have now been rescaled to QVGA, are compressed for optimum filesize, and located in the qtc, cfc, or out folder, depending on which steps you skipped.
Also note we retouched some images by hand for the QVGA version.
CFC for VGA and WVGA
!IMPORTANT! Updated 11.11.2009: CFC GUI updated. Support DLLs updated to CFC 0.60!
!IMPORTANT! Updated April 20, 2009: CFC Live Patch discontinued!
I have recompiled the libgles proxy file originally made for QVGA to a version that only handles the CFC compression, and should work on any 'normal' Manila 3D version, like the ones found on the Touch Diamond, Pro and HD.
If you cooks/chefs/whomever implement this, you can reduce TF3D's size footprint by half (7 mb smaller in my test). This also makes a positive performance difference, as the on-the-fly decompression of the images is actually faster than the flashdisk access.
Patching TF3D to be able to use CFC compression also allows theme makers to make faster and smaller themes.
Instructions to modify TF3D VGA/WVGA - for users NEW!:
FOR v2.5 AND NEWER PATCH MANUALLY!
- Get CFC GUI, attached to the first post of this topic.
- Connect your device using ActiveSync
- Use the "Device->Patch Manila on device" feature
Instructions to modify TF3D VGA/WVGA - for chefs:
FOR v2.5 AND NEWER PATCH MANUALLY!
- Get CFC GUI, attached to the first post of this topic.
- Run CFC GUI and select the folder where your Manila package is stored
- Make sure you got a backup of said folder
- Go to Tools -> Patch Manila
The following things will be done to your Manila package:
- All image files will be CFC compressed (lossless, faster)
- All image files will be padded to be a multiple of 512 bytes in size (faster)
- All image files will be set to System/Hidden/Archive file attributes
- Manila.exe will be patched to use libgles_mn.dll
- Manil2.exe will be patched to use libgles_mn.dll (depending on Manila version)
- Mode9.dll will be patched to use libgles_mn.dll
- libgles_mn.dll will be placed in the package
- zlib_mn.dll will be placed in the package
If no errors occur, there is nothing else you need to do, aside from cooking the package
Instructions to modify TF3D VGA/WVGA - by hand
Attached is a zip file (cfc-support-dlls) with the two DLLs you need: libgles_mn.dll and zlib_mn.dll . These must be placed in your \Windows folder.
Next, hex edit Manila.exe, Manil2.exe and Mode9.dll to use libgles_mn.dll instead of libgles_cm.dll. Just search for "libgles_cm.dll" in the files and replace it with "libgles_mn.dll". These values may appear multiple times in the file! Make sure to search and replace for both ANSI and UNICODE variants! Your Manila version may not have both Manila and Manil2, or only one of them may contain the "libgles_cm.dll" string. This is normal.
You should also modify HKLM\Software\OEM\MASD\Manila and append _CFC to the version string. This so people can recognize if their installation supports CFC.
This will only add support for CFC to your Manila install, it will not make images and such actually use CFC. But you can now support themes that do use CFC.
Note to people who used an older version of this patch
You don't have to do it again, but the automated tool makes sure everything is done right. If you still have zlib1.dll from the old patch on your device, do not remove it unless you want to break the "Teeter" game.
You can run the automated tool over a package you used the older patch on - it will clean it up.
Wow. I can't wait. So is it possible then to run TF3D on an older VGA device? Or is this to help us decode the themes. Isn't james making a image converter from vga to qvga?
Kraize92 said:
Wow. I can't wait. So is it possible then to run TF3D on an older VGA device? Or is this to help us decode the themes. Isn't james making a image converter from vga to qvga?
Click to expand...
Click to collapse
everything has been converted to qvga
CraZyLiLbOy said:
everything has been converted to qvga
Click to expand...
Click to collapse
True, but a tool would still be nice because the skins of tf3d are all in vga. If we had a converter, I could apply the theme and everything, and then convert it to qvga.
Kraize92 said:
True, but a tool would still be nice because the skins of tf3d are all in vga. If we had a converter, I could apply the theme and everything, and then convert it to qvga.
Click to expand...
Click to collapse
You don't have to convert anything to qvga. All touchflo 3d themes work on our qvga devices. You are using touchflo 3d interface from the diamond. You can apply any themes you want for the touchflo 3d. I know the diamond is a vga device but it still works. I'm saying this because I've tried it and it works, mark my words
The QVGA port was made specifically in mind with that VGA stuff will just work - and they will. However, when I finish this guide and you have read it, you will understand exactly how you can significantly optimize the QVGA version - and make the VGA version better
I don't know man but I think I fall in love with the touchflo 3d. Thanks to Chainfire and djboo.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Great TUT very interesting. thanks for sharing your tools
yesssss thank you for your great tuto,
2Chainfire
Didn't understand how to convert single PNG to manila's VGA file
Could u give me more examples?
any progress???
Any progress... on what? This tutorial is finished. TF3D-QVGA is finished. All we're waiting for is a new driver release that fixes the power issues.
i dont mean to sound like a complete idiot, but will this ever be in cab form?
i have no idea what most of what you are talking about means in this tutorial, I am just legitimately curious if it is even possible.
also if that is not possible, wouldnt the old touchflo3d porting-to-qvga cab be necessary along with this tutorial? maybe i am just seriously confused and have no idea what is going on
disregard if this is too much ignorance in one post!
thanks
htctoucher
htctoucher said:
i dont mean to sound like a complete idiot, but will this ever be in cab form?
i have no idea what most of what you are talking about means in this tutorial, I am just legitimately curious if it is even possible.
also if that is not possible, wouldnt the old touchflo3d porting-to-qvga cab be necessary along with this tutorial? maybe i am just seriously confused and have no idea what is going on
disregard if this is too much ignorance in one post!
thanks
htctoucher
Click to expand...
Click to collapse
I'm not quite sure what you are after with this CAB you ask for. If you are looking for TF3D for QVGA, a CAB is available (but there are still some issues with the required 3D drivers for the QVGA devices).
The modifed TF3D executable files work fine on QVGA with the original graphics. This thread however explains the method to do pixel-perfect TF3D image manipulation (for themers, porters, etc) in general, which was up until the documentation of this thread impossible (read: unknown). Methods like using Manila Editor work, but degrade image quality significantly more than the method described here.
However, this thread also documents converting the (originally) VGA TF3D images to the corresponding QVGA sizes, and even adding an extra 5x compression to them. This is not necessary to do if you have the modified TF3D executables for QVGA (it will use the VGA graphics files just fine), however, it certainly makes TF3D a hell of a lot faster and significantly reduces the size it requires on disk and the memory use (both of which are more limited on the QVGA devices compared to the VGA ones). It includes this information because this method was 'discovered' in the process of porting TF3D to run on QVGA devices. Similarly, porting existing TF3D VGA themes to QVGA resolutions will increase the speed of TF3D on QVGA devices over using the original theme's VGA graphics files.
Note that the TF3D-QVGA CAB ( http://www.htcclassaction.org/download/TF3D-QVGA.cab ) already includes the optimized QVGA graphics, and some of them were even retouched by hand (not mentioned in this threaD).
I hope this explains what you want to know
Hi I have managed to convert a manila file to ATC format using cfc.exe but now I cant open it using The Compressonator. If I go to open (there is no ATC file type listed in 'Files of type' drop down menu), select All Files (*.*) and try to open the converted .atc file, it does not load. I am using The Compressonator 1.50 which seems to have been updated recently (18-12-08) so it might be an issue with the new version. If it is that can you please upload or direct me to the old version or else if I am doing something wrong, please help.
Thank you
AF241
Hive and registry file editor 2.0.112
A command line utility to work with hv files.
Full Unicode support, and more.
!!! WARNING !!!
Do not use CeRegEditor (cereg) to work with .hv !!!
!!! WARNING !!!
Usage:
HVEdit.exe <rgu>|<dump>|<hive>|<hvm> [-r] [<outputdir>]
<rgu> Build boot.hv from rgu file.
<dump> Build default.hv and user.hv from project folder.
<hive> Output hive file contents to the <hivename.hvm> text file.
<hvm> Build hive file from hvm file.
-r Don't insert *.rgu hash value to the hive.
Release history:
---------------------
V. 2.0.112
+ Some changes.
V. 2.0.035
+ Extract hive to the .hvm text file.
+ Build hive from .hvm file.
V. 1.0.000 (Initial Release)
Don't need Unicode??
and I'm wondering whether the "REGEDIT4" is needed.
btxakita said:
Don't need Unicode??
Click to expand...
Click to collapse
Who spoke?
btxakita said:
and I'm wondering whether the "REGEDIT4" is needed
Click to expand...
Click to collapse
Yes.
NEW Release
V. 2.0.035
Added new functions:
+ Extract hive to the .hvm text file in Unicode format.
+ Build hive from .hvm file.
AnDim
AnDim said:
NEW Release
V. 2.0.035
Added new functions:
+ Extract hive to the .hvm text file in Unicode format.
+ Build hive from .hvm file.
AnDim
Click to expand...
Click to collapse
A dump question I'm sure, I started cooking recently and I'm wondering what I could accomplish with this application. In what scenario would I use it? Thanks, 12
Thanks. One problem I had w/ rgucomp is missing unicode support.
12aon said:
A dump question I'm sure, I started cooking recently and I'm wondering what I could accomplish with this application. In what scenario would I use it? Thanks, 12
Click to expand...
Click to collapse
What exactly is not clear?
Thanks AnDim, this makes GSM -> CDMA SYS porting much easier.
AnDim said:
What exactly is not clear?
Click to expand...
Click to collapse
Well I think I'm starting to understand. As I always have had the luxury of initflashfiles.txt or app.reg and never edited the .hv manually. It was unclear to me why I would need this or what it would add to a kitchen that supports those features (initflashfiles.txt and app.reg)
So does this app have a function in such a kitchen, I mean is there anything I will need to do manually that can't be done otherwise or is this more like a general app so that the hives on any phone can be more easily changed?
Thanks 12
@AnDim, PMed you.
Have anyone used this program? It's not working well for me. I use it fo rgu to hv convert but when i compare the output hv file with cereg alot different files
prpetrov said:
I use it fo rgu to hv convert but when i compare the output hv file with cereg alot different files
Click to expand...
Click to collapse
!!! WARNING !!!
Do not use CeRegEditor (cereg) to work with .hv !!!
how does project folder need to be set up to build default and user hv's?
BUG: Default value is imported and exported incorrectly
If I convert an rgu which contains
Code:
@="one"
"Default"="two"
hvedit overwirtes the default value with "two"
Request: option to specify output file
selyb said:
how does project folder need to be set up to build default and user hv's?
Click to expand...
Click to collapse
HVEdit.exe <My project folder>
Don`t need to be setup. Rgu´s, packages.sof and so on, can be in any sub folders.
selyb said:
BUG: Default value is imported and exported incorrectly
If I convert an rgu which contains
Code:
@="one"
"Default"="two"
hvedit overwirtes the default value with "two"
Click to expand...
Click to collapse
Hey man .... Key name "Default" == Key name @
selyb said:
Request: option to specify output file
Click to expand...
Click to collapse
??? f.e.
AnDim
sorry, I misunderstood. Until now, I didn't know you could use "Default"=
selyb said:
Request: option to specify output file
Click to expand...
Click to collapse
What I mean is like
Code:
hvedit myhive.hv myhive.txt
so hvedit would export myhive.hv to myhive.txt
selyb said:
What I mean is like
Code:
hvedit myhive.hv myhive.txt
so hvedit would export myhive.hv to myhive.txt
Click to expand...
Click to collapse
Usage:
HVEdit.exe <rgu>|<dump>|<hive>|<hvm> [-r] [<outputdir>]
<rgu> Build boot.hv from rgu file.
<dump> Build default.hv and user.hv from project folder.
<hive> Output hive file contents to the <hivename.hvm> text file.
<hvm> Build hive file from hvm file.
-r Don't insert *.rgu hash value to the hive.
AnDim
AnDim said:
Usage:
HVEdit.exe <rgu>|<dump>|<hive>|<hvm> [-r] [<outputdir>]
<rgu> Build boot.hv from rgu file.
<dump> Build default.hv and user.hv from project folder.
<hive> Output hive file contents to the <hivename.hvm> text file.
<hvm> Build hive file from hvm file.
-r Don't insert *.rgu hash value to the hive.
AnDim
Click to expand...
Click to collapse
Hi,
thanks for this software it works great, if you now how to put this commands.
My experience was this:
D:\>hvedit.exe hive d:\hive
or:
D:\>hvedit.exe hive d:\hive\user.hv
I'm getting this:
(HVEdit) !ERROR: failed building DEFAULT hives.
When I copy HVEdit.exe and cereg400.dll to d:\hive and run this:
D:\hive>hvedit.exe d:\hive\user.hv
I'm getting this:
Admsoft (R) HVEdit Version 2.0.035
Hive and registry file editor.
(C) 2009 AnDim, XDA-Developers.
(HVEdit) Extracting user.hv ...
(HVEdit) Done! and only 21.kB of 69kB .hv file
When I run it on system.hv like this:
D:\hive>hvedit.exe d:\hive\system.hv
I'm getting this:
(HVEdit) Extracting system.hv ...
ERROR: C:\MacB\private\winceos\COREOS\filesys\reg\reghive\.\registry.c line 172: Reg: Invalid reghandle 0x20000a61
and 144kB .hvm file of 105kB .hv
I didn't mention it but I was trying to convert .hv to .hvm
Maybe my dump has got some errors in it.
Any idea what is wrong, why the user.hvm is so small?
Thank you AnDim for great software. It is quite difficult to find out utility for ce hive.
After use for a period I would to to report some small problem.
1. Long multi string was trimed. (Maybe fixed length variable in code) See Ex.
Registry key in my rgu file is
REGEDIT4
[HKEY_LOCAL_MACHINE\System\StorageManager\Filters\fsreplxfilt]
"DirsToExclude"=multi_sz:"\\mxip_initdb.vol",
"\\mxip_system.vol",
"\\mxip_notify.vol",
"\\mxip_lang.vol",
"\\cemail.vol",
"\\pim.vol",
"\\Documents and Settings\\default\\user.hv",
"\\Documents and Settings\\system.hv",
"\\Windows\\Profiles\\guest\\Temporary Internet Files\\",
"\\Windows\\Profiles\\guest\\Cookies\\",
"\\Windows\\Profiles\\guest\\History\\",
"\\Windows\\Activesync\\",
"\\Windows\\messaging\\",
"\\Temp\\"
After change to boot.hv and extract back to boot.hvm the Registry is as hereunder.
; Admsoft (R) HVEdit Version 2.0.035
; Hive and registry file editor.
; (C) 2009 AnDim, XDA-Developers.
;
;
; Source file: boot.hv
; Size: 28672 byte
; Hash: e9e4f2edd12c78eeed7aceeaea22c36f
; Boothash: 1ef4fb8f2b109c0757709e7f46c71a20
; Build tool: HVEdit v2.0
[HKEY_CLASSES_ROOT]
[HKEY_CURRENT_USER]
[HKEY_LOCAL_MACHINE\System\ObjectStore\RegistryUpdate]
"boot.rgu"=hex:1E,F4,FB,8F,2B,10,9C,07,57,70,9E,7F,46,C7,1A,20
[HKEY_LOCAL_MACHINE\System\StorageManager\Filters\fsreplxfilt]
"DirsToExclude"=multi_sz:"\\mxip_initdb.vol",
"\\mxip_system.vol",
"\\mxip_notify.vol",
"\\mxip_lang.vol",
"\\cemail.vol",
"\\pim.vol",
"\\Documents and Settings\\default\\user.hv",
"\\Documents and Settings\\system.hv",
"\\Windows\\Profiles\\guest\\Temporary Internet Files\\",
"\\Windows\\Profiles\\guest\\Cookies\\",
"\\Windows\\Profiles\\guest\\History\\",
"\\Windows\\Activesync\\"
[HKEY_USERS]
; (HVEdit) Done!
2.The size of hv file is larger than original. (This is not effect functional just file size is increase from 732kb to 744kb for instance.)
After investigate, there was extra length on data key which have length just fit 32 bits size. (Maybe when adjust cell size to fit 32 bits)
Hope my comment will be some benefit for you for next version.
Regards,
AnDim said:
Hive and registry file editor 2.0.035 (C) 2009 AnDim
A command line utility to work with hv files.
Full Unicode support, and more.
!!! WARNING !!!
Do not use CeRegEditor (cereg) to work with .hv !!!
!!! WARNING !!!
Usage:
HVEdit.exe <rgu>|<dump>|<hive>|<hvm> [-r] [<outputdir>]
<rgu> Build boot.hv from rgu file.
<dump> Build default.hv and user.hv from project folder.
<hive> Output hive file contents to the <hivename.hvm> text file.
<hvm> Build hive file from hvm file.
-r Don't insert *.rgu hash value to the hive.
Release history:
---------------------
V. 2.0.035
+ Extract hive to the .hvm text file.
+ Build hive from .hvm file.
V. 1.0.000 (Initial Release)
(C) 2009 AnDim
Click to expand...
Click to collapse
This seems to be JUST what I am looking for .... but I am having problems trying to use I suspect is simply a case of finger-trouble on my part, so will greatly appreciate any help available from AnDim or any other users who can point me in the right direction, please.
I think I correctly understand the command line syntax and have run HVEdit to unpack a default.hv registry file taken from a dumped ROM (Toshiba TG01). I then made some basisc edits to the resulting .hmv file and used HVEdit again to recompile back to .hv.
When the ROM is recompiled and flashed it refused to boot. So I went back and tried using HVEdit to convert to .hmv and back without making any changes. Same problem.
I have attached the .bat files i used to run for reference....View attachment 305797
I have been reading about a need to insert REGEDIT4 at the start of the file; have tried doing so but that just results in an error.
Any guidance will be greatly appreciated
Request Response !!
Is anyone able to offer any assistance on this question please
Help will be much appreciated !
kevinpwhite said:
This seems to be JUST what I am looking for .... but I am having problems trying to use I suspect is simply a case of finger-trouble on my part, so will greatly appreciate any help available from AnDim or any other users who can point me in the right direction, please.
I think I correctly understand the command line syntax and have run HVEdit to unpack a default.hv registry file taken from a dumped ROM (Toshiba TG01). I then made some basisc edits to the resulting .hmv file and used HVEdit again to recompile back to .hv.
When the ROM is recompiled and flashed it refused to boot. So I went back and tried using HVEdit to convert to .hmv and back without making any changes. Same problem.
I have attached the .bat files i used to run for reference....View attachment 305797
I have been reading about a need to insert REGEDIT4 at the start of the file; have tried doing so but that just results in an error.
Any guidance will be greatly appreciated
Click to expand...
Click to collapse
Intro:
This is an all-in-one tool for decompiling, compiling and comparing lua scripts found in Manila (TouchFLO 3D / Sense).
All this is a continuation of sztupy's original work: Lua 5.1 tools.
General:
LuaTool consists of 4 parts: Lua decompiler, Lua compiler, Lua compare utility and a Manila file type detection utility.
LuaDec 3.2 - Lua decompiler
Notes on latest version:
Major overhaul of the local finding algorithm. Most lua scripts can now be fully decompiled without a problem.
Manila 2.5.1921 has a total of 703 scripts (including embedded scripts). LuaDec can fully decompile 663 files. That's a success rate of 94.31%.
General notes:
LuaDec automatically checks if the output file was decompiled successfully.
If it wasn't, LuaDec will also output the disassembly and compare file.
In case the decompile was 100% good, LuaDec will only output the standard .lua file as before.
LuaC 1.2 - Lua compiler
Binary function replacement:
LuaC can directly replace functions in compiled luac files. This can be useful if the luac file can't be fully decompiled, but only a small part of the file needs to be edited. Some more info on function replacement.
Continue statement:
The "continue" statement has been added to the Lua Compiler.
Lua doesn't officially support continue statements, but it looks like HTC added it for their needs, so I'm following their lead.
Usage and versions:
Code:
LuaTool 1.2 by Co0kieMonster
Usage: LuaTool <task_select> [task_options] <task_input>
Tasks:
/decompile (or /d) -- Lua Decompiler
/compile (or /c) -- Lua Compiler
/compare (or /cr) -- Lua Compare utility
/detect (or /dt) -- Manila file type detect utility
LuaDec 3.2
Usage: LuaTool /decompile [options] <inputfile>
Available Options:
-o <filename> specify output file name
-dis don't decompile, just disassemble
-f <number> decompile/disassemble only function number (0 = global block)
LuaC 1.2
Usage: LuaTool /compile [options] <inputfile>
Available Options:
-o <filename> specify output file name
-s strip debug information
-r <n> <luac_file> replace function <n> in <luac_file> with <inputfile>
LuaCompare 1.2.1
Usage: LuaTool /compare [options] <original.luac> <newfile.lua(c)>
Available Options:
-o <filename> specify output file name
-s side by side file comparison
-du disable underline
ManilaDetect
Usage: LuaTool /detect <inputfile>
LuaTool changelog:
# LuaTool v1.2
-updated LuaDec to v3.2, LuaC to v1.2 and LuaCompare to v1.2.1
# LuaTool v1.1
-updated LuaDec to v3.1, LuaC to v1.1 and LuaCompare to v1.2
LuaDec changelog:
# LuaDec v3.2
-Local guesser improvements
---major overhaul - gives much better results
-Conditionals handling improvements
---fixed elseif not being recognised in some cases
---added partial support for complex inline boolean assingment
-General improvements
---fixed single function decompile
---fixed table assignments where there are more then 16 values
---better error handling
# LuaDec v3.1
-Conditionals handling improvements
---wrote a brand new algorithm for handling complex logic expressions
---fixed falsely detected generic for loops
---fixed misplaced if end, because of end-to-break optimization
-Local guesser improvements
---declarations at CALL before RETURN
-General improvements
---fixed indents not behaving properly in some cases
---fixed LOADNIL assignments where the destinations are local variables
---decompiler now displays success rate after decompile
---added SETLIST handling
# LuaDec v3.0.4
-General improvements:
---added back error messages
---fixed variable arguments handling
---fixed multiple inline assignments
---fixed a rare if ending misplacement
-Local guesser improvements at:
---inline bool assignments
---table in table situations
---TAILCALLs
---CALLs which return multiple results
---locals declared just before TEST ops
---SETTABLE where b isn't a constant
# LuaDec v3.0
-core rewrite and cleanup
-more accurate especially with conditionals and loops
-some miscellaneous accuracy improvements
-added extra info to script header (date, time, file name and manila name)
-LuaCompare updated to v1.0.1 (compatibility)
# LuaDec v2.1
- Less crashing:
--- added a failsafe for crashing on bad registers
--- fixed crash on SETUPVAL
--- fixed crash on SETLIST
- Better conditional handling:
--- fixed handling of deeper nested else and elseif
--- fixed handling of empty if-end and else-end blocks
--- added break handling
- Better table handling:
--- fixed inline table assignments
--- fixed handling of numerically indexed tables
- Adjustments to local guesser:
--- fixed guessing for inline table assignments
--- fixed guessing for SETGLOBAL and SETUPVAL at PC 1
LuaC changelog:
# LuaC v1.2
-added binary function replacement
# LuaC v1.1
-added "continue" statement
LuaCompare changelog:
# LuaCompare v1.2.1
-small change to support single function decompile
# LuaCompare v1.2
-pre-compare disassembly is now done internally instead of writing to disk and reading
-added a console message with match percentage
# LuaCompare v1.1
-initial version integrated in LuaTool
Go co0kiemonster! You da man!
boy oh boy ... cant believe that, thanks
time to get back to the keyboard and do some hack0r's stuff
see you guys
I like the new compare output a lot! Saves some lines in the manilatool.cmd as well. Do you plan on updating all the ruby tools or just the compare?
Muchos gracias
12aon said:
Do you plan on updating all the ruby tools or just the compare?
Click to expand...
Click to collapse
Probably all (except luadecguess, which is redundant because luadec has an internal guesser since version 2.0). But I hadn't planned on doing it any time soon - right now, luadec is keeping me pretty busy. I'm doing a semi-rewrite of it in order to inject some OOP love (port to C++) and then hopefully make a proper conditionals and loops engine.
I don't mind OOP love . Hey I somebody came with this idea about luadec but as it turned out I misunderstood him. He was actually talking about the m9editor. Nevertheless the idea is good. You tell me if it's doable.
Wouldn't it be a good idea to include the full manila name in the lines of code as well (If known). Going a bit further might it not be an even better idea to include some more diagnostic info there.
Thing I can think of are manila version (although I can't imagine a foolproof method), date, full manila path name maybe some diagnostics.
You know I'm going to keep you occupied right?
12aon said:
Wouldn't it be a good idea to include the full manila name in the lines of code as well (If known). Going a bit further might it not be an even better idea to include some more diagnostic info there.
Thing I can think of are manila version (although I can't imagine a foolproof method), date, full manila path name maybe some diagnostics.
Click to expand...
Click to collapse
Full manila name and date aren't a problem. I'll add them in the next release.
Manila version would have to be set by the user so that's a bit problematic. But it would be great to have. I'll try to think of good way to add it.
As for diagnostics: Did you mean adding something other than the "-- DECOMPILER ERROR: ... " lines, or just making those lines a bit more useful?
12aon said:
You know I'm going to keep you occupied right?
Click to expand...
Click to collapse
I'm counting on it
Co0kieMonster said:
Full manila name and date aren't a problem. I'll add them in the next release.
Manila version would have to be set by the user so that's a bit problematic. But it would be great to have. I'll try to think of good way to add it.
As for diagnostics: Did you mean adding something other than the "-- DECOMPILER ERROR: ... " lines, or just making those lines a bit more useful?
I'm counting on it
Click to expand...
Click to collapse
The version number can be found in a package here:
Code:
[HKEY_LOCAL_MACHINE\Software\HTC\Manila]
"Version"="2.1.19193517.0"
That's either the .reg or .rgu file
It can also sometimes be found in the package name. But these things are very unpredictable. In that sense it could only be used as an extra. I don't know if any of the exe's in the package hold the info.
By diagnostics I was referring to my lack to come up with anything else. I hoped your developer instincts would lead you to add in the rest for me. But now that I think of it maybe something amount of errors in the script or amount of opcodes, maybe the number of functions. I don't know why, or how it would be useful so probably just leave out that part. Unless you disagree of course,
12
12aon said:
You know I'm going to keep you occupied right?
Click to expand...
Click to collapse
LOL 12 has a new toy!
I guess it would be dumb to ask if you intend to use this in your Manila kitchen! LOL
Asphyx said:
LOL 12 has a new toy!
I guess it would be dumb to ask if you intend to use this in your Manila kitchen! LOL
Click to expand...
Click to collapse
It is already part of the kitchen , co0kie has been helping us for a while now. He is the one who added the lua scheme to notepad2
Ive been trying to use this on the lua files in the sprint hero but no matter what i try i get the error "Bad header in precompiled chunk"
Any thoughts/ideas?
You sure hero's got lua files? Would you mind sharing them?
12
pentace said:
Ive been trying to use this on the lua files in the sprint hero but no matter what i try i get the error "Bad header in precompiled chunk"
Any thoughts/ideas?
Click to expand...
Click to collapse
Might be a different encoding.
Can you upload a few of the files so I can check it out?
Version 3.0 is up
Some info:
Version 3.0 is a complete rewrite of LuaDec. It's more accurate then 2.1, especially when large loops are involved. It might just need a little bit more tweaking but conditional and loop handling is almost perfect. The next big thing to tackle is local guessing, and that will come in a later version.
LuaDec has also generally been cleaned up, so no more obsolete command line switches or memory leaks.
It can also retrieve the full manila name and add it to the file header. E.g.: if you decompile 0bd9db81_manila, LuaDec will add \windows\htc\people\scripts\people\peoplegroupdeta il.luac to the decompiled script header for better reference. For this to work you need to have the m9editor.names.txt file in the same folder as LuaDec.
Now that I've done this rewrite I should be able to accelerate development. And there are some cool new feature coming in future versions.
Decompile Luaplugins for lightroom
Hi,
I just wondering if it is possible to use this to decompile any lua files, the one i'm looking for is decompiling lightroom plugins
skrollster said:
Hi,
I just wondering if it is possible to use this to decompile any lua files, the one i'm looking for is decompiling lightroom plugins
Click to expand...
Click to collapse
LuaDec has been tuned specifically to HTC's Lua variant. Theoretically it should decompile any Lua 5.1 scripts, but it might be incompatible with the character and number encodings of non-HTC scripts. I'm not sure about the specifics, since those adaptation were done before my development efforts - see here for some of the details: http://forum.xda-developers.com/showpost.php?p=3466886&postcount=249
You can always give it a try and see what happens. It can't hurt
Co0kieMonster said:
LuaDec has been tuned specifically to HTC's Lua variant. Theoretically it should decompile any Lua 5.1 scripts, but it might be incompatible with the character and number encodings of non-HTC scripts. I'm not sure about the specifics, since those adaptation were done before my development efforts - see here for some of the details: http://forum.xda-developers.com/showpost.php?p=3466886&postcount=249
You can always give it a try and see what happens. It can't hurt
Click to expand...
Click to collapse
It just gave me an almost blank file, the only thing in it was some stuff i guess you add to all files
skrollster said:
It just gave me an almost blank file, the only thing in it was some stuff i guess you add to all files
Click to expand...
Click to collapse
Yeah, that's definitely because of the different encodings. Sorry, but I guess it's not going to work.
Too bad really, is it possible to create a decompiler for the encoding used for adobes applications? if so, is it much work to change it?
I'm not sure. Upload one or two lua files so I can take a look.
Purpose:
This purpose of this VBScript is to process and organize data in RGU/REG files to remove duplicates, identify faulty entries, and move entries to ascending alphabetical order (the same way it is displayed in a registry editor).
Requirements:
Windows Scripting Host (included in most versions of windows)
rgucomp.exe and cereg400.dll located somewhere in your path (same folder as the script probably won't work if the script is run from another folder)
.reg and .rgu files are expected to be UTF-16LE with BOM
Usage:
Drag a .rgu, .reg, or .hv onto RGUber.vbs OR run "wscript.exe RGUber.vbs example.rgu"
Details:
When an rgu|reg file is specified, RGUber will:
1) create backup of input file
2) rename input file to boot.rgu
3) use rgucomp to convert it to *.hv
4) use rgucomp to convert new .hv to original rgu path\name
5) Reorder all keys in ascending alphabetical order and all values for each key in ascending alphabetical order with default value first
When an hv file is specified, RGUber will:
1) use rgucomp to convert it to *.rgu
2) Reorder all keys in ascending alphabetical order and all values for each key in ascending alphabetical order with default value first
Options
Open RGUber.vbs in your favorite text editor. All options are set at the beginning with (hopefully) meaningful descriptions.
Code:
'//Path to rgucomp (leave this as default if rgucomp.exe is located in your system path)
Const RGUCOMP = "rgucomp.exe"
'//Path to notepad, only needed if %OPENAFTER% is true
Const NOTEPAD = "notepad.exe"
'//The following options can be set to True/False or 0/1
'//Organize registry entries in ascending alphabetical order
Const REORDER = True
'//Open in %NOTEPAD% after conversion is done
Const OPENAFTER = False
'//Save any errors from rgu -> hv conversion
Const LOGERRORS = True
'//Save a backup copy of %INPUTRGU% as "%INPUTRGU%_Backup.rgu"
Const BACKUPRGU = True
Other info
If target file already exists, RGUber will ask if you want to overwrite.
Text files (the MS way) typically contain CRLF for next line. Output from rgucomp.exe contains many CRCRLF. RGUber removes the extra CR.
I have very few comments in the code. If requested, I will upload another copy with as many detailed comments as I can manage.
I tried to code this as efficiently as VBScript can possibly be. I kept getting errors when trying to run 'rgucomp.exe -b -nologo' so instead of running it directly, RGUber creates a bat file, executes it, then deletes it.
On my AMD Phenom 9600 with Vista64 and 3 SATA in Raid5, RGUber completes rgu->hv->rgu of 2084 lines in <3s
RGUber always saves output from rgu->hv conversion but deletes the file if there were no errors.
RGUber crashes on files with no reg entries (e.g. empty app.reg in an EXT package that does not add any registry entries)
Changelog:
v1.21a
Values are now sorted in alphabetical order for each key
v1.2
Replaced Organize function with one from RGUOrder
Lost ability to reorder values for each key (To be readded in next version)
v1.1
Fixed a bug with removing hashdata from output (RGUber would mix data from two keys under one)
Changed sorting algorith with a much faster one
v1.02
Added option to remove RegistryUpdate key from rgucomp output
Fixed a typo where RGUber was not removing the system attribute from input rgu files
v1.01
Fixed typo where RGUber was waiting for backup file instead of log file
Change 'Done' msgbox to one that shows beginning time and ending time
v1.0
Initial Release
RGUOrder v1.4
This script will only reorder the contents of an rgu without processing with RGUComp, thereby keeping comments and delete key entries. RGUComp/cereg400.dll are not needed to use this app.
Changelog:
v1.4
Fixed a bug where if the original rgu did not end with a new line then the last entry after being sorted would be lost.
Fixed two bugs where only the first 25 tabs and first 25 spaces would be removed before sorting (This did not affect data integrity or performance, but the checksum would be different each time you run the output back through RGUOrder until all the original tabs/spaces were removed)
Added code to prevent multiple entries of the same key from being reordered
Fixed other miscellaneous bugs/oddities introduced with v1.3
v1.3
Added code to add a delete key for each subkey of a deleted key so that when reordered, the key deletion isn't broken
v1.2
Fixed a bug where the last key processed was being concatenated to another with no CRLF producing an invalid rgu file
I'm not sure if this relates to your app but I have a small question:
If a dumped a rom (raw, not kitchen type) and removed several apps/programs but did not clean the registry (very tedious), will this help me clean it up (remove dead paths, etc)?
And if so, how will it know just by dragging the .hv file? I mean how will your app know if a registry entry does not have the app/program included in the rom anymore?
Please forgive me if my question does not relate to your app
There is no way for my app to know, it isnt that smart
It would take an extensive app/database to know which keys are related to which apps.
Thanks for this post
updated to v1.1
v1.02 had a bug in the code which removed hash data from output which made it mix data from the key before it with the key after it
If I ever get around to updating again, I will use hvedit instead of rgucomp
I get an error.
Script: D:\RGUber.vbs
Line: 136
Char: 2
Error: File not found
Code: 800A0035
Source: Microsoft VBScript runtime error
Any reason why?
I attach the file i want to sort alphabetically.
I have no idea
It worked for me with no problem (file attached)
Please tell me the location of RGUber.vbs and of 51329f91-0017-4364-bcff-e032c5d45b01.rgu
Great application bro!!
Only limitation is that I have to put reg400.dll and rgucomp in C:\windows
c_shekhar said:
Great application bro!!
Only limitation is that I have to put reg400.dll and rgucomp in C:\windows
Click to expand...
Click to collapse
yeah, I tried to get around that but I didn't find anything feasible with vbscript :-/
Actually, they don't have to go in C:\windows
I reinstall windows regularly so I keep as many apps portable as I can. I have a bin folder on another partition that I add to the system path variable after a new install for stuff like this.
selyb said:
yeah, I tried to get around that but I didn't find anything feasible with vbscript :-/
Actually, they don't have to go in C:\windows
I reinstall windows regularly so I keep as many apps portable as I can. I have a bin folder on another partition that I add to the system path variable after a new install for stuff like this.
Click to expand...
Click to collapse
can you elaborate this a bit more. Because I too would like a similar arranfements...
My C:\ partition has Vista64
My F:\ partition has all my documents, downloads, music, movies, etc and a folder F:\bin\
F:\bin contains >100 downloaded command line programs and vbs scripts that I have written including
RGUber.vbs
lame.exe
rgucomp.exe
cereg400.dll
FixVTS.exe
faad.exe
nuerecmod.exe
Tag.exe
find Advanced System Properties (I can't remember how, it's different for XP/Vista/7) go to the Advanced tab and hit the Environment Variables button
Under system variables, scroll down to 'Path', double click it. This defines your 'system path'. It contains a list of folders separated by semicolon ";". At the end, add a semicolon and the path to the folder you want to add (e.g. ;F:\bin) after that, hit ok. XP may need to reboot to reflect the change but I'm not sure. Vista and 7 are affected immediately.
With this setup, you can open a command prompt in any folder on your computer and type "RGUber.vbs xyz.rgu" and it would work as if all the files are in that folder.
Thanks a lot bro!!!
I am grateful...
I'd really like to use this, but unfortunately I get this error regardless of the app.reg I drag onto the script:
Script: C:\RGUber\RGUber.vbs
Line: 232
Char: 3
Error: The system cannot find the path specified.
Code: 80070003
Source: (null)
Thanks if you can advise.
Quetzecotyl said:
I'd really like to use this, but unfortunately I get this error regardless of the app.reg I drag onto the script:
Script: C:\RGUber\RGUber.vbs
Line: 232
Char: 3
Error: The system cannot find the path specified.
Code: 80070003
Source: (null)
Thanks if you can advise.
Click to expand...
Click to collapse
Hmmm... this line asks the system for what is in the %temp% variable and attempts to change the working directory to the result.
Open RGUber.vbs in notepad and go to line 232
Modify
Code:
WSH.CurrentDirectory = WSH.Environment("SYSTEM")("temp")
to
Code:
WSH.CurrentDirectory = "C:\RGUber\"
then try again
Works great after your fix, selyb. Thank you for this useful app and your many helpful contributions to the Kaiser forums.
Quetzecotyl said:
Works great after your fix, selyb. Thank you for this useful app and your many helpful contributions to the Kaiser forums.
Click to expand...
Click to collapse
Yeah, I may relocate from Kaiser forums to Rhodium. I have an AT&T Tilt 2 in the mail to me ATM
Grats on getting a Rhodium. Found a question after using it for a while. This is just one example of such behavior, but why does:
Code:
[HKEY_CURRENT_USER\Software\HTC\TaskManager\ExclusiveList\System]
"CMBandSwitching.exe"=dword:0
get turned into:
Code:
"CMBandSwitching.exe"=dword:0
How do I make it regard CURRENT_USER keys?
Quetzecotyl said:
Grats on getting a Rhodium. Found a question after using it for a while. This is just one example of such behavior, but why does:
Code:
[HKEY_CURRENT_USER\Software\HTC\TaskManager\ExclusiveList\System]
"CMBandSwitching.exe"=dword:0
get turned into:
Code:
"CMBandSwitching.exe"=dword:0
How do I make it regard CURRENT_USER keys?
Click to expand...
Click to collapse
I had this problem with an earlier version. If you are using v1.1 then please attach the original rgu/reg. I have tried and I can't seem to reproduce it since I fixed it already.
Please, replace rgucomp with hvedit . I really need your help because rgucomp doesn't work for me . Thanks in advance .
tomcug said:
Please, replace rgucomp with hvedit . I really need your help because rgucomp doesn't work for me . Thanks in advance .
Click to expand...
Click to collapse
why doesn't rgucomp work? I would be surprised to learn that hvedit will work when rgucomp won't.
[2012/06/03] IMPORTANT UPDATE HERE
Hi hackers,
This is meant as a little update on one of the projects I've been working on. I'm kinda stuck now. I have a suspicion of what the problem is. I thought that maybe if I write a post about it, me or someone else will have an idea on how to get this working.
The goal is to run native homebrew executables on WP7
This has not been done yet. All apps are Silverlight apps that are compiled as DLL and run by Taskhost.exe with least privileges. All other executables are signed by Microsoft. Executables that are compiled as ARM executable cannot be started.
The angle is to create a certificate that allows to sign a WP7 executable. Then add that to the appropriate certificate store. Create an executable. Sign it with the private key. Load it onto a WP7 device. Copy it to the Windows folder. Use an OEM driver to launch the executable.
First I did research on the certificate stores. I can now with certainty state that there are 4 certificate stores:
- CA
- Root
- My
- Code Integrity
After a lot of research I finally got complete read/write access to all of these stores. The Code Integrity store contains all the certificates that are used by the Loader Verifier to verify the executable that is being launched. When the device is launched for the first time, the certificates that are in \Windows\ciroots.p7b are installed to that certificate store. These certificates have these properties:
Key Usage = 0x86 = Digital Signature, Certificate Signing, Off-line CRL Signing, CRL Signing
Entended Key Usage = Code Signing (1.3.6.1.5.5.7.3.3) + Unknown key usage (1.3.6.1.4.1.311.10.3.14)
So I used OpenSSL to create such an certificate (with private key) for myself. And I installed the certificate in the Code Integrity store.
I then used VS2008 to create a completely barebone executable (ARMv4 Console app with only Sleep(-1) in the Main). I signed it with SignTool from Microsoft.
I loaded the executable to my device and I copied it to the \Windows folder (I think the policies restrict executing to only from that folder, but I'm not sure about that).
I use the Samsung driver to launch the executable, because I need at least Standard Rights to launch an executable. The Samsung driver has Elevated Rights. My own app has only Least Privileges. Using the Samsung driver does not return any success or fail codes. But looking at the Running Processes list, I don't see my Test.exe running. It should be, because the main thread is put to sleep infinitely.
So why is this not working?
Well, I have a guess. I think it's the policies that bind the certificates in the Code Integrity store to the different accounts/chambers. In the \Windows folder there are a lot of policy xml-files. On fist boot, these are merged into PolicyCommit.xml and then compiled to policydb.vol. When the Loader Verifier (lvmod.dll) loads an executable, it queries the policies to determine access rights and chamber for that executable. The policies that matter in this context are defined in 8314B832-8D03-444f-9A2A-1EF6FADCC3B8.policy.xml. It's an xml-file that basically says this:
Code:
Microsoft Mobile Device Privileged PCA - ced778d7bb4cb41d26c40328cc9c0397926b4eea - not used in this context
Microsoft Mobile Device TCB PCA - 88bcaec267ef8b366c6e6215ac4028e7a1be2deb - honored by System Identity Group
Microsoft Mobile Device Unprivileged PCA - 1c8229f5c8d6e256bdcb427cc5521ec2f8ff011a - honored by Standard Right Identity Group
Microsoft Mobile Device VSD PCA - 91b318116f8897d2860733fdf757b93345373574 - not used in this context
VeriSign Mobile Root Authority for Microsoft - 069dbcca9590d1b5ed7c73de65795348e58d4ae3 - honored by LPC Identity Group
I should find a way to add a policy with my certificate in it. Any ideas?
Ciao,
Heathcliff74
If you are able to re-sign an executable that is already in the ROM, i would try that, so you know the problem isn't within the native code, but only with the signing. Or maybe the other way round which would be awesome.
regards
Flow WP7 said:
If you are able to re-sign an executable that is already in the ROM, i would try that, so you know the problem isn't within the native code, but only with the signing. Or maybe the other way round which would be awesome.
regards
Click to expand...
Click to collapse
That's a good idea. I must say that I don't have much faith in the current RecMod tools for WP7 right now. I am able to get the binaries recmodded so that I can disassemble them correctly. But I don't think they can be easily launched. But there are executables that are on the rom as complete binaries, instead of rom-modules. To begin with, I have to select one that does not need much privileges to run and try to sign that one and then run it.
I'm really busy with work right now, so I think I won't be able to try it until the day after tomorrow. But I will try it and will let know how that went.
Thanks!
Decompiled taskhost.exe, so it gets more easy for us to see if its able to make taskhost to start another exe for us. Lots of code tho (C code).
taskhost.c (276 KB) in attachments.
edit: Oh, WOW, this really shows how to call those anonymous methods without call signature "Hello" (signature: "??z_Hello_?mze")
Hmm, pretty much about the pause part?
Code:
if ( v10 )
{
a7 = sub_178E7(v10);
if ( a7 >= 0 )
{
a7 = sub_180A5(v7, v7 + 64);
if ( a7 >= 0 )
{
a7 = ThemeInitialize(v7 + 136);
if ( a7 >= 0 )
{
v11 = sub_1862B(v13, v7);
EnableHostAutoDehydration(v11 == 3);
v16 = 0;
a7 = InitializeEmClientEx(&a2, 0, &v16);
if ( a7 >= 0 )
{
a7 = RegisterPausedHostCallback(sub_19D0D, 0);
if ( a7 >= 0 )
{
a7 = RegisterResumingHostCallback(sub_19D31, 0);
if ( a7 >= 0 )
{
if ( v11 != 3
|| (a7 = RegisterDehydrateHostCallback(sub_19D76, 0), a7 >= 0)
&& (a7 = RegisterFreezeHostCallback(sub_19D97, 0), a7 >= 0) )
{
a7 = RegisterExitHostCallback(sub_19D55, 0);
if ( a7 >= 0 )
a7 = sub_17C0A(*(_DWORD *)(v7 + 128), 0);
}
}
}
}
}
}
}
}
UIX framework entry-point (exe)
Code:
int __cdecl sub_11114(int a1, int a2, int a3)
{
int v4; // [sp+0h] [bp-38h]@1
char Dst; // [sp+4h] [bp-34h]@1
int v6; // [sp+8h] [bp-30h]@1
int v7; // [sp+Ch] [bp-2Ch]@1
int v8; // [sp+18h] [bp-20h]@1
int v9; // [sp+28h] [bp-10h]@1
v4 = 0;
memset(&Dst, 0, 0x34u);
v8 = a3;
v6 = (int)L"res://FlightModeUXDLL!FlightMode.uix";
v7 = (int)L"FMMain";
v9 = 2;
RunApplication(&v4);
return dword_12034;
}
C++ converted
Code:
UIXApplicationInfo app;
app { ... }
RunApplication(&app);
struct UIXApplicationInfo
{
int UNK_v4 = 0;
char Dst = {0};
char* uixFile;
char* uixEntryPoint;
int UNK_v8;
int UNK_v9 = 2;
}
Then just figure out the UIX part (or test the existing "res://FlightModeUXDLL!FlightMode.uix" if it launches, if so, we made it).
___
Found this in mango dump:
> Uninstall provxml
Code:
<!-- Uninstall Xbox LIVE Extras App -->
<characteristic type="AppInstall">
<nocharacteristic type="{0c17d153-b5d5-df11-a844-00237de2db9e}"/>
</characteristic>
Is there a reason you can't just use COM interop to run native code? Check out this thread for a discussion covering the technique: http://forum.xda-developers.com/showthread.php?t=820455
athompson said:
Is there a reason you can't just use COM interop to run native code? Check out this thread for a discussion covering the technique: http://forum.xda-developers.com/showthread.php?t=820455
Click to expand...
Click to collapse
Hello "co-founder of native code on WP7"
I'm fully aware of the possibility of native code through COM. I use it for example in the WP7 Root Tools. But I just wanted to take it a step further. Running native executables give a lot more freedom. Not being bound to the watchdog, getting higher privileges and running in the background for instance. But there's a whole lot more. So that's why I started research on it. Thanks anyway. You helped making native code possible on WP7.
Ciao,
Heathcliff74
The taskhost.exe is our RAM, because our app run in it, giving us full RAM access inside our "viritual ram". So that means we own all strings, int, floats etc. Then rewrite the ram to change strings in mscorlib. The checksum if an exe has been modified is only checked at startup, without checking if we modify the dll at runtime.
My purpose with this is that some function's call external apps, where we rewrite the args going in to the function. Just find an exploitable function and modify it after JIT has been there one before generating the pre ram, that we modify and call yet again but with the modified ram values behind.
Marshal.Copy, my friends, there.
[SecurityFuckingSafeCritical]
(byte[] source, IntPtr destination, int length)
> Interopservices leaked dll (\windows)
destination = our ram ptr to modify.
fiinix said:
The taskhost.exe is our RAM, because our app run in it, giving us full RAM access inside our "viritual ram". So that means we own all strings, int, floats etc. Then rewrite the ram to change strings in mscorlib. The checksum if an exe has been modified is only checked at startup, without checking if we modify the dll at runtime.
My purpose with this is that some function's call external apps, where we rewrite the args going in to the function. Just find an exploitable function and modify it after JIT has been there one before generating the pre ram, that we modify and call yet again but with the modified ram values behind.
Marshal.Copy, my friends, there.
[SecurityFuckingSafeCritical]
(byte[] source, IntPtr destination, int length)
> Interopservices leaked dll (\windows)
destination = our ram ptr to modify.
Click to expand...
Click to collapse
Hmmm. 10 Points for inventiveness But I don't think it's going to work. Even if you could find a function where the executable is passed as argument you still don't have enough privileges. Most code will have the path to the executable hardcoded instead of an argument. And you will still run under TaskHost with Least Privileges. And you need to have at least Standard Privileges or higher to launch most executables with CreateProcess() or ShellExecuteEx().
Sent from my OMNIA7 using XDA Windows Phone 7 App
Heathcliff74 said:
Hmmm. 10 Points for inventiveness But I don't think it's going to work. Even if you could find a function where the executable is passed as argument you still don't have enough privileges. Most code will have the path to the executable hardcoded instead of an argument. And you will still run under TaskHost with Least Privileges. And you need to have at least Standard Privileges or higher to launch most executables with CreateProcess() or ShellExecuteEx().
Sent from my OMNIA7 using XDA Windows Phone 7 App
Click to expand...
Click to collapse
"And you will still run under TaskHost with Least Privileges"
I know, i dont need standard rights to do it. Because i call a mscorlib function that is trusted code. I think you saw my idea wrong, let me show you.
[mscorlib, SecuritySafeCritical]
public static void example(string str)
{
string mscorlibStr = "you cant change my value ";
Debug.WriteLine(mscorlibStr + str);
}
This is where we modify "mscorlibStr" in ram and the function is still trusted code. But its doing something totally different from that it would do.
fiinix said:
"And you will still run under TaskHost with Least Privileges"
I know, i dont need standard rights to do it. Because i call a mscorlib function that is trusted code. I think you saw my idea wrong, let me show you.
[mscorlib, SecuritySafeCritical]
public static void example(string str)
{
string mscorlibStr = "you cant change my value ";
Debug.WriteLine(mscorlibStr + str);
}
This is where we modify "mscorlibStr" in ram and the function is still trusted code. But its doing something totally different from that it would do.
Click to expand...
Click to collapse
I really hate to break it for you. But the [SecuritySafeCritical] is indeed trusted code, but it will still check your privileges. All the API functions that do system modifications like that, do the security checks. Read the note under SecuritySafeCriticalAttribute here. Also read this; same problem. You are in process TaskHost.exe and it is launched in LPC (Least Privilege Chamber), so every CeImpersonateToken() to do the important stuff will fail and return an error code. I also wouldn't know how you would modify the stack-frame of a function that you call. Seems impossible to me, because at the moment you call the function, that stack-frame has not been allocated yet.
Anyway, although I don't think that is going to work in any way, I absolutely don't want to discourage you, because my experience is that when you try enough, sooner or later you will find an exploit
Ciao,
Heathcliff74
Currently installing "Windows Embeded Compact 7", because this lousy ARMv4 compiler (from WM5-6) maybe generates wrong ARM op-codes (WP7 runs ARMv7), therefore it says "Invalid program signature" (or what error it was).
Maybe ARMv7 is'nt even backwards compatibility with ARMv4.
By compiling with the ARMv7 compiler from WEM7, it will probably (hope) generate a valid exe.
Thats it..
edit:
*Research
"Armv7 is the processor instruction set used starting with the S5L8920 in the iPhone 3GS and in all subsequent devices. Processors that support Armv7 instructions are backward compatible with Armv6 instructions, but attempting to run binaries compiled for Arm7 on older, Armv6 processors will result in the error: "Bad CPU type in executable"."
Source: http://theiphonewiki.com/wiki/index.php?title=Armv7
___
"As I said in the past, the ARMv6 CTR was kept backwards compatible with
> > > earlier versions of the ARM architecture (and ARM tried to keep it like
> > > this as much as possible). With ARMv7, you have multiple levels of cache
> > > and different types (e.g. ASID-tagged VIVT I-cache). There is no way you
> > > could encode the useful information while keeping the same (and only)
> > > register, hence the the need for a new register."
Source: http://www.spinics.net/lists/arm-kernel/msg58813.html
As i see this (^), all ARMv > 6 == no backwards
ARMv6 had backwards to 4
ARMv7 >> ARMv6 compatibility, not more.
_
Problem officer even running ARMv4???
>On a non ARMv4 backwards compatibility CPU.
Profit!!
__
[ExeX.exe] (the one that i recompiled to a state: "this has to work")(ARMv4)
Decompilation:
Code:
; Attributes: bp-based frame
EXPORT start
start
var_20= -0x20
oldR4= -0x1C
oldR5= -0x18
oldR6= -0x14
oldR7= -0x10
oldR11= -0xC
oldSP= -8
oldLR= -4
MOV R12, SP
STMFD SP!, {R4-R7,R11,R12,LR}
ADD R11, SP, #0x1C
SUB SP, SP, #4
MOV R4, R3
MOV R5, R2
MOV R6, R1
MOV R7, R0
.
Next up, decompile a ARMv7 from a raw device. (how, someone has one)
fiinix said:
Next up, decompile a ARMv7 from a raw device. (how, someone has one)
Click to expand...
Click to collapse
I think you'll find what you're looking for here: http://forum.xda-developers.com/showthread.php?t=681659 in the dump of the IMAGEFS. What did you use to decompile it? IDA Pro, or a different thing?
athompson said:
I think you'll find what you're looking for here: http://forum.xda-developers.com/showthread.php?t=681659 in the dump of the IMAGEFS. What did you use to decompile it? IDA Pro, or a different thing?
Click to expand...
Click to collapse
IDA Pro, yes. Ill see if i can dump that "nbh" (used to nb0), and extract a fully operable exe that is not corrupted.
fiinix said:
IDA Pro, yes. Ill see if i can dump that "nbh" (used to nb0), and extract a fully operable exe that is not corrupted.
Click to expand...
Click to collapse
First use Andim's WP7 Rom Tools to extract the rommodules. Remember to always dump a folder, not a single file.
Then use Denomitor's version of Recmod and follow the instructions in the post. That works most of the time.
Going forward
Currently building the WP7 ARMv7 commandline, getting closer.
Current cmd (not working, no need to help):
Code:
"C:\WINCE700\sdk\bin\i386\arm\cl.exe" /Od /D "_DEBUG" /D "_WIN32_WCE=0x700" /D "UNDER_CE" /D "ZUNE_HD" /D "WINCE" /D "DEBUG" /D "_WINDOWS" /D "ARM" /D "_ARM_" /D "_UNICODE" /D "UNICODE" /D "_CRT_SECURE_NO_WARNINGS" /Gm /EHsc /MTd /Gy /fp:fast /GR- /Fo"C:\Users\Steven VM\Desktop\ARMv7\Build\Debug/" /Fd"C:\Users\Steven VM\Desktop\ARMv7\Build\Debug/vc80.pdb" /W3 /c /Zi /TP /QRfpe- /QRarch7 "C:\Users\Steven VM\Desktop\ARMv7\main.cpp"
/QRarch7 is the ARMv7.
edit:
HOORRY SHEEAT
generated:
> main.obj
> vc80.idb
> vc80.pdb
, feels soo good:
main.exe is there.
IDA Pro says "ARM AND THUMB MODE SWITCH INSTRUCTIONS", just like others.
Code:
; Input MD5 : B50E8D8395DE7CA2419464DC3CE0BC74
; File Name : C:\Users\Steven\Desktop\burn\main.exe
; Format : Portable executable for ARMI (PE)
; Imagebase : 10000
; Section 1. (virtual address 00001000)
; Virtual size : 00000018 ( 24.)
; Section size in file : 00000200 ( 512.)
; Offset to raw data for section: 00000400
; Flags 60000020: Text Executable Readable
; Alignment : default
; Processor : ARM
; Target assembler: Generic assembler for ARM
; Byte sex : Little endian
; Segment type: Pure code
AREA .text, CODE, READWRITE, ALIGN=4
; ORG 0x11000
CODE32
EXPORT start
start
var_4= -4
SUB SP, SP, #4
MOV R3, #1
STR R3, [SP,#4+var_4]
LDR R0, [SP,#4+var_4]
ADD SP, SP, #4
BX LR
; End of function start
Made an empty entry point as from above ^:
Code:
int wWinMainCRTStartup()
{
return 1;
}
PE Explorer (main.exe):
Machine: THUMB
Operating System Version: 7.0
Image Version: 7.0
Subsystem Version: 7.0
Subsystem: WinCE GUI
**** so CLOSE!
Successful copied "main.exe" and "ExeX.exe" to "\Windows", where i have the right to launch them remotely.
Method:
WP7Process p = device.LaunchEXE(@"main.exe", "");
main.exe (no signing, ARMv7):
System.UnauthorizedAccessException: Access is denied.
WP7Process p = device.LaunchEXE(@"ExeX.exe", "");
ExeX.exe (signed with CA/ROOT custom, ARMv4):
System.Runtime.InteropServices.COMException (0x800704EC): This program is blocked by group policy. For more information, contact your system administrator.
There IS different things going on! Something is missing, but what
edit:
Signed main.exe with custom XDA ROOT certificate (ARMv7):
signtool.exe sign /sha1 "[CertChomp]" "main.exe"
> Now main.exe also gets "This program is blocked by group policy. For more information, contact your system administrator."
Ill see if i can add it to startup list , if it boot from there.
edit 2:
Nope gonna hijack "fieldtestapp.exe" with my app because policy says:
Risky-mode.Activate();
Backup(fieldtestapp.exe, backupPath);
Copy(main.exe, > fieldtestapp.exe);
"LOADERVERIFIER_ROUTE_BY_NAME"
"LOADERVERIFIER_EXE_AUTHZ_INROM_ROOT"
<Rule Description="Route fieldtestapp.exe" ResourceIri="$(LOADERVERIFIER_ROUTE_BY_NAME)/PRIMARY/WINDOWS/FIELDTESTAPP.EXE" SpeakerAccountId="$(SYSTEM_USER_NAME)" PriorityCategoryId="PRIORITY_LOW">
<Authorize>
<Match AccountId="$(FIELDTESTAPP_EXE_SID)" AuthorizationIds="LV_ACCESS_EXECUTE" />
</Authorize>
</Rule>
<Rule Description="Authorize fieldtestapp.exe be loadable to $(FIELDTESTAPP_EXE_SID) and chambers" ResourceIri="$(LOADERVERIFIER_EXE_AUTHZ_INROM_ROOT)/WINDOWS/FIELDTESTAPP.EXE" SpeakerAccountId="$(SYSTEM_USER_NAME)" PriorityCategoryId="PRIORITY_STANDARD">
<Authorize>
<Match AccountId="$(FIELDTESTAPP_EXE_SID)" AuthorizationIds="LV_ACCESS_EXECUTE,LV_ACCESS_LOAD" />
</Authorize>
</Rule>
edit 3:
Seems like "fieldtestapp.exe" is ROM locked. Need to try out some other targets.
edit 4:
Target acquired "ProximitySensorDisable.exe" > "ProximitySensorDisableBackup.exe"
Successful copy == no ROM lock.
edit 5:
There exists two types of talking to the LoadVerifier (the: This program is blocked by group policy.):
Direct exe name OR special certificate
How we do:
> Direct exe (hijack exe)
How we cant do (SHA1) (Nope, ain't gonna happen):
> We certainly dont have Microsofts certificate so this way is a nodo, haha lol, no do way.
(1: direct exe name) /LOADERVERIFIER/GLOBAL/AUTHORIZATION/PE_AUTHZ/NONE/NONE/PRIMARY/WINDOWS/CFGHOST.EXE
(2: static/pre certificates) /LOADERVERIFIER/GLOBAL/CERTIFICATES/HASH/SHA1/91B318116F8897D2860733FDF757B93345373574
edit 6:
Yep, loads of edits, just for you.
Allowed exe's to run (sorted a-z) (direct exe) (pre cert removed):
Code:
ACCESSIBILITYCPL.EXE
ACCOUNTSMANAGER.EXE
ALARMS.EXE
APPCHECKERSHIM.EXE
APPPREINSTALLER.EXE
AUTODATACONFIG.EXE
AUTOSIM.EXE
AUTOTIMEUPDATE.EXE
BRIGHTNESSCPL.EXE
BTUXCPL.EXE
CALENDARAPP.EXE
CALLSETTINGSHOST.EXE
CALNOT.EXE
CALUPD.EXE
CAM_FW_UPDATE_UI.EXE
CELLUXCPL.EXE
CERTINSTALLER.EXE
CFGHOST.EXE
CFLAUNCHER.EXE
CHDIALERHOST.EXE
CIPHASE2.EXE
CLIENTSHUTDOWN3.EXE
CLOCKNOT.EXE
CMACCEPT3.EXE
COLDINIT.EXE
COMMSVC.EXE
COMPOSITOR.EXE
CONFIGDM.EXE
CONFIGXML.EXE
CONMANCLIENT3.EXE
CONTACTS.EXE
CPROG.EXE
DATETIMECPL.EXE
DCVSSWITCH.EXE
DEPOTCOPY.EXE
DEVICEFEEDBACKCPL.EXE
DEVICEREG.EXE
DIAGPORTCHANGETEST.EXE
DLLHOST.EXE
DMSCHEDULERCALLBACK.EXE
DMSRV.EXE
DMSTOOLS.EXE
DUACLIENT.EXE
DW.EXE
EDM3.EXE
EMAIL.EXE
EMAILSETUP.EXE
ENDPOINT.EXE
FCROUTERCMDTEST.EXE
FIELDTESTAPP.EXE
FLIGHTMODE.EXE
GAMESUX.EXE
IEXPLORE.EXE
INITIATEDMSESSION.EXE
INVALIDLICENSEUXLAUNCHER.EXE
KEYBOARDCPL.EXE
LASSCREDENTIALEXPIRATIONCHECK.EXE
LASSRESTARTER.EXE
LIVETOKEN.EXE
LOCKCPL.EXE
LOOPBACKTEST.EXE
MEDIAGROVEL.EXE
MEUX.EXE
MITSMAN.EXE
MMSPRPROXY.EXE
MMSTRANSHOST.EXE
MULTIMEDIALAUNCHER.EXE
MYPHONECPL.EXE
MYPHONETASKSRUNTIME.EXE
NATIVEINSTALLERHOST.EXE
OFFICEURL.EXE
OMADMCLIENT.EXE
OMADMPRC.EXE
OMHUB.EXE
ONBOOTSQM.EXE
ONENOTEMOBILE.EXE
OOBE.EXE
PACMANINSTALLER.EXE
PHOTOENT.EXE
PHOTOENTCAPTURE.EXE
PHOTOUPLOADER.EXE
PPT.EXE
PWORD.EXE
PWRLOGCTRL.EXE
PXL.EXE
RAPICONFIG.EXE
REGIONCPL.EXE
RMACTIVATE.EXE
SAPISVR.EXE
SECSIMTKIT.EXE
SERVICESD.EXE
SERVICESSTART.EXE
SETTELEPORTMODE.EXE
SETTINGS3.EXE
SHORTMSG.EXE
SICLNT.EXE
SIGNALEVENT.EXE
SIREPSERVERAPPDEV.EXE
SMSETTINGS.EXE
SMSTRANSPORT.EXE
SOUNDCPL.EXE
SPEECHCPL.EXE
SPMC.EXE
SQMEVENT.EXE
SSUPDATE.EXE
TASKHOST.EXE
TELSHELL.EXE
TESTSHOW.EXE
THEMECPL.EXE
TOGGLEBROWSERHIBERNATION.EXE
TOGGLEDOG.EXE
UDEVICE.EXE
UIF.EXE
UNIFIEDPAIR.EXE
USBMGR.EXE
WEBSEARCH.EXE
WIFIUXSPLASH.EXE
WLANEXT.EXE
WLIDSETUP.EXE
WWANDATAMGR.EXE
XDRMREMOTESERV.EXE
ZIPVIEW.EXE
ZMFTASKLAUNCH.EXE
How code (yes i know its super un-optimized, fast put together):
Code:
var doc = XDocument.Load(File.OpenRead("SamsungOmnia7_BasePolicy_webserver.xml"));
var ea = doc.Elements().ToArray()[0].Elements()
.Where(x => x.Name.LocalName == "Rule")
.Where(x => x.Attributes("ResourceIri").Count() > 0)
.Where(x =>
{
var r = x.Attribute("ResourceIri").Value;
return r.Contains("LOADERVERIFIER") && r.ToLower().Contains(".exe") && !r.Contains("CERTIFICATES");
})
.Select(x =>
{
var v = x.Attribute("ResourceIri").Value;
var l = v.LastIndexOf('/');
return v.Substring(l + 1);
})
.Distinct()
.OrderBy(x => x)
.ToArray();
edit 7:
yeah, lol i say too.
Unprotected exe (FCRouterCmdTest.exe)
> c:\Project Work\SGH-i707(Cetus)\FCRouterCmdTest\Windows Mobile 6 Professional SDK (ARMV4I)\Release\FCRouterCmdTest.pdb
mfw samsung use "Windows Mobile 6 Professional SDK (ARMV4I)"
Wow, this truly was a big step today
Done hacking today.
"After a day, there comes another day"
@fiinix,
You did a lot of testing. Good job, man.
A few comments:
0x800704ec "blocked by group policy" is THE error of the new WP7 security model. It is basically telling you to go f*ck yourself. Everything you do without enough privileges or capabilities results in this error.
The two ways of policies, exe-path and cert-hash, is result of difference between rom-modules and executables that are signed and added as a file. Rom-modules are not even normal files. You can't open and read them. They are executable sections that are mapped in rom-address-space. You can only call loadlibrary() and createprocess() on them. Since they are only executable sections, they don't have a signature, like a normal executable file would have. Therefore they are referred to with an exe-path. You may safely assume that every path to an executable in the policy files is referring to a rom-module and can't be overwritten in any way (except by cooking your own rom - who is going to unlock our bootloaders?!?) Other than that, there are a few signing certs that Microsoft has. Signing the different executables with different privileges and accordingly a different cert. Their hashes are in the policies.
Using ARMv7 isn't going to add much I'm afraid. Although it may make a difference in the exe-header. But you've seen tools that were really old, remember And they were signed to have TCB access. And they were compiled for ARMv4. So it should not make much difference.
I did some testing with certificates myself yesterday. Up until Zune totally went bezerk on it. I don't know what happened, but after removing my own cooked certs it all seems normal again. Zune started using 100% cpu on verifying certs and dropping my connection all the time. Help! So I haven't made much progress. I will try again later. Hope it will go better. And I will try to resign an existing executable, as Flow WP7 suggested.
According to policy on my omnia (webserver dumped) there seems to exist two typed of HDD, one ROM hard coded and one that points to internal sd card. It seems that all exe and dll on the sd are not "protected" and therefore can be hijacked.
Seems like ARMv4 will be enough, but to be on the safe side i compile with both, to have more chance getting it work.
Zune, hmm, did not seem to like you, maybe Microsoft DDOS'ed you lol
"Sent from my fingers on my phone", don't expect way too long text
XxXPachaXxX said:
Excuse my ignorance...I'm a noob...This hack may also work on LG devices?
Click to expand...
Click to collapse
At the moment fiinix and I are both working on Samsungs and we use a couple of Samsung-specific exploit to get deeper in the system and getting a better understanding of the system. The ultimate goal is to find exploits that will work for all devices. But we're not at that stage yet. Hacking is research, a lot of trying and being lucky sometimes. Just bear with us
Ciao,
Heathcliff74