Related
***THIS POST IS NOT COMPLETE, I WILL UPDATE MORE LATER***
First, an introduction:
The Image Update system allows the OEM (us! ) to issue updates to a "Live" filesystem - without disrupting user data. This allows, for example, a buggy driver to be updated after the phone has been shipped, or a software package to be updated to the latest version, with minimal knowledge on the user's part. The system validates all updates against an internal list of certificates, and refuses the update if a match is not obtained. This system can also be used to deploy entirely new software to the device (such as support for another locale, input method editor, application support for a new feature the carrier is rolling out, etc.)
Potential usage scenarios for this system
A central server could be maintained for all SYS/OEM updates - each ROM Chef would need to maintain a list of original packages, any updated package(s), and download URL's for each updated package. The user would then receive these updates through the built-in AutoUpdate facility in Windows Mobile, which can check periodically, or on-demand. Each Chef could maintain seperate download servers from the update server to minimize server load.
Alternatively, a chef could provide .cab.pkg updates in his or her ROM thread, on their own web site, etc., and the user could download these and install them at will. These packages can optionally be authenticated to be coming from the Chef, if the Chef wants to ensure updates are coming from him only. A public certificate could also be used to allow users to issue updates as well.
The more technical Summary
Image Update allows an OEM to issue updates to the OEM's, XIP, SYS, (possibly) Radio, or any combination of these. The update can be pushed to the user via a specially formatted SMS or by manual execution. There are at least 2 levels of certificate checking involved in the process, I believe against \SYS\Metadata\DefaultCerts.dat. The system reboots into the ULDR to apply the update, because the filesystem cannot be modified while actively mounted. The ULDR provides a minimal operating enviornment to facilitate this.
How does a Chef need to prepare a ROM for Image Updating?
The Chef would need to use a ROM Kitchen that leaves the .dsm and .rgu file structure intact (i.e. an "unprotected" ROM) - All .dsm's in this ROM would need to be properly formatted with Package Name, versioning info, etc. during the cooking process, in order to facilitate version checking, etc. Each .dsm would need to be signed with a certificate the Chef would use to validate the update as coming from him or her. (Alternatively a public certificate could be used like SDKCerts if the Chef chooses not to maintain direct control over updates, and allow other users to create updates as well)
The Chef also needs to ensure not to Reduce the ULDR partition or remove it; this would cripple the Update Loader and prevent the Image Update system from functioning.
The .cab.pkg format
At the root of a package, the .dsm defines the Package structure (all files, registry entries, etc.) It contains version info, certificates, and other data. A ROM consists of a number of these packages, in an area of flash memory that is not user-writable. When someone wants to issue an update using the ImageUpdate system, they create a matching .dsm, same guid, with a newer version number, same internal package name, same processor ID, os version, etc., there is also a flag that can be marked as an update package or a new package - in this .dsm they define the files that will make up the new, updated package. Here they can add or remove files. One of the files defined by the package is optionally an .rgu, and this defines the registry entries associated with the package. Also optionally included is a provxml to facilitate file operations (copy/replace/delete/rename/etc.) and further registry or metabase operations. This collection of files is rolled up into a ".cab.pkg" archive by a program like cabarc. Once in a pkg.cab format, the package is signed.
A .cab.pkg is referred to as a "Canonical Package" and multiple Canonical Packages can be rolled up into a single "Super Package" to facilitate updating multiple Packages at the same time.
Inside this .cab.pkg is where the "MNGE" file format comes in to play. Essentially, this format is Microsoft's way of storing an XIP Module in the filesystem. (Their equivalent of our imageinfo.bin + s000, s001, etc.) - The "MNGE" format is simply a 4-byte MNGE header, followed by the imageinfo.bin, followed by the s00x sections attached to the end. When the Image Update system processes a file in this MNGE format, it is added to the IMGFS as an XIP Module.
Deploying a .cab.pkg to a device
Now there are several ways to deploy this package to the device, the primary way is an OMA-DM SMS message, which triggers the Image Update system to initiate a download from a server defined in the SMS message. The server can be either a normal HTTP server or a secure HTTPS server. The Image Update system will prompt the user to plug in the usb cable to download over the users home connection if the user has not already authorized the Auto Update system to utilize their GPRS data connection.
Another way for a .cab.pkg update to be pushed to the system is simply through a file copy operation, be it ActiveSync, SD Card, Bluetooth, or otherwise. Once on the device, the .cab.pkg is executed by the user the same way a .cab would be.
The Update Agent
Initiated by either a completed download from push SMS, or user-executed, the "Update Agent" program (which is part of the \SYS\FWUPDATE Package) attempts to validate the Certificates, Package dependancies, and other info contained in the .cab.pkg and .dsm. Once validated, the "Update Agent" sets a flag that the bootloader reads, the flag is a boolean, off = boot into normal OS, on = boot into ULDR - so then the system reboots, the flag is read, and you load into...
The Update Loader
The "Update Loader" or "ULDR" which is a minimal kernel configuration, that provides just enough driver support to display info on screen, respond to user input, and read/write from the internal flash (NAND or NOR)
From here the ULDR does further validation on the .cab.pkg, and applies it to the filesystem. If there are any modules in the package it dynamically relocates the memory map to make sure there are no overlaps. This is why it's important that reloc's not be removed from your ROM - ULDR will fail in this case.
The End Result
Once the ULDR has completed updating, the device is again rebooted, back into the full system, where the now-updated packages are now a part of the IMGFS, any updated files are processed (.rgu, .provxml, etc.) - The package is now a full part of the ROM.
The new .dsm replaces the old .dsm (along with the other files in the package) and now a future update will be checked against this new package.
If the update was pushed via OMA-DM SMS, or AutoUpdate, the device Pushes a notification to the OMA-DM server notifying it of the update status.
What's missing right now to implent the ImageUpdate system?
We need a Kitchen that's properly configured to allow us to create versioning info, proper package names, and insert this along with a certificate (or multiple certificates) into the .dsm's.
We also need the Kitchen to be able to modify \SYS\Metadata\DefaultCerts.dat with the certificates used, so that it passes authentication. Alternatively the authentication checking could be patched out. (this one is easily doable at build-time)
We need a program that can convert from a standard file to an MNGE format, so we can implement modules in our .cab.pkg's. (done it seems, thanks ervius!)
We (optionally) need a properly configured web server that supports HTTP/HTTPS, can communicate the proper xml configuration data, and can be updated with new packages by Chefs. (this one's a ways off)
We (optionally) need a program to convert from MNGE format to a standard file to facilitate extracting modules from .cab.pkg's. (working hard on that)
I've attached a .cab.pkg for NetCF2. Open up mscoree.dll in a hex editor, and check out the MNGE header. This file becomes a module once processed by the ImageUpdate system. Note that all the executables (.exe/.dll/.mui) that become modules contain this MNGE header. All executables that are inserted into ROM as files keep their normal MZ file header. The first major step here will be in being able to convert between MZ<-->MNGE freely NetCF2 is a well known package that can be found in any stock ROM, so with this we have a good baseline to work with.
http://rapidshare.com/files/238295848/netcf.cab.pkg
--Reserved--
More Technical Specifications
The basic ImageUpdate Layout consists of:
[IPL] -- [MBR] -- [ULDR] -- [NK] -- [IMGFS] -- [TFAT]
[IPL] is the "Initial Program Loader" that handles basic init functions and determines if control should be handed over to ULDR, or NK through a flag set by UpdateBin.exe - the IPL is not contained within a partition. The IPL is copied entirely to RAM and executed from there. IPL loads NK into RAM, and also handles any decompression of NK if it's required - some SmartPhone's ive seen use SRPX compression for the NK partition. Once NK is copied to RAM it is then executed. The IPL is handled seperately from the other parts of the operating system, and is not flashed during a normal update.
[MBR] is the "Master Boot Record" and contains partition tables for the below components - it points to NK so when IPL loads the MBR, control is handed over to NK. The MBR contains information on where each partition is located on the flash (memory address), the size of the partition(s), and the type of each partition. The MBR is referenced from many components on the device such as IPL and ULDR in order to facilitate handoff of control between ULDR and NK. The MBR also serves as a boundry between the IPL which is not part of the regular partition structure, and the rest of the flash, which is part of the partitioning structure.
[ULDR] is the "Update Loader" and provides a basic WinMo system so that file operations can be done on the IMGFS partition while it's unmounted. The Update Loader is even able to update itself - during operation it is loaded entirely to RAM. On development workstations the ULDR supports a KITL connection, that can be used to load updates directly from the "Release" folder. It seems it may be possible through this method to flash a new image to the device, possibly opening up the ability to flash to devices that have not yet been flashed with "HardSPL"
[NK] is the "Kernel Partition" - or what we know as xip.bin - This component is updatable by ImageUpdate, and has a pre-defined "free space buffer" with room to grow, which defaults to 512KB. This partition holds only the kernel and drivers necessary to bring up the rest of the filesystem, from which the rest of WinMo is loaded. The Kernel Partition uses the same "Package" format as the IMGFS and is updatable in the same manner.
[IMGFS] is the "System Partition" - running the Image Update filesystem. This component is updatable by ImageUpdate, and has a pre-defined "free space buffer" with room to grow, which defaults to 9.5MB. The IMGFS uses the "Package" format to further split its components.
[TFAT] is the "Transaction Safe FAT File System" which is where all user-writable data goes.
In most Device Designs, there's a single NOR or NAND chip used for flash. This is important as due to the typical layout above, both NK and IMGFS must have a pre-defined amount of free space - because TFAT is the last partition on the drive, and cannot be shifted once flashed to the device. It's possible for the partition layout to be setup differently (Partitions in different order) to help alleviate that problem. The ImageUpdate system would really shine on a device with 2 flash chips, a NOR chip dedicated to the ImageUpdate partition and a NAND chip dedicated to the TFAT, but no OEM has created such a design yet.
Packages
Package Types
There are 3 different types of packages, Canonical, Update, and Super.
Canonical contains the entire contents of the package. It is used for a first-time package install, and if there are any major updates to be issued that would require the complete package. The file extension is .cab.pkg
Update contains a binary delta between a package already on the device, and the updated version of that package. In this manner the limited space is conserved (i.e. if a package change was a simple registry entry - no need to replace the 5mb of .dll and .exe in that package, just alter the .rgu with the new data. These packages are also referred to as "Delta" packages. The concept is similar to the unix implementation of Diff/Patch. The file extension is .cab.pku
Super contains a collection of update and/or canonical packages. This is very useful when you are attempting to bring in a new package that has dependencies on other packages - rather than reboot into ULDR for each individual package in the proper dependency order, they can all be introduced at once. Every package contained inside a super package is validated, and if one fails, the remaining valid updates may still be applied, as dependencies allow. A super package is simply an un-compressed .cab containing other packages, renamed to .cab.pks
The package layout itself is quite basic, it consists of a .dsm which contains all versioning info, association info, and dependency info. It also contains a list of all modules and files inside the package, and a certificate store of all approved certificates that will be allowed to update that package. Alongside the .dsm is an optional .rgu, which defines the registry settings associated with that package. Also optional is a .provxml file, which can be: mxip_[packagename]_[version].provxml, mxipcold_[packagename]_[version].provxml, or mxipupdate_[packagename]_[version].provxml. mxip and mxipcold are effectively treated the same, executed only on a cold boot. mxipupdate_ provxml's will be executed any time that package is updated, in addition to a cold boot - so if you are adding new .cab.pkgs and wish the .provxml to be executed immediately, it would need to be mxipupdate. This may not be desirable in some cases, such as when the provxml might override a user preference - in that case you would only want it to run on a cold boot, in order to avoid "strange" behavior on the user's side of things.
There is a "shadow order" defined in the .dsm as well - this controls what "priority" .rgu's are compiled together into the device registry hive - a package that shadows another package will override any .rgu entries that shadowed package may contain. This is important to consider when utilizing .cab.pkgs in order to obtain your desired end registry. This shadow order also applies to provxmls inside the package - a package that shadows another package will override its provxml settings as well.
The user registry hive is always top-most in the shadow order (except in the case of an mxipupdate_ provxml) - so any changes to .rgu registry settings will not override a user-changed registry setting. (example: You had foobar set to 5 in your initial deployment. At some point after flashing to his/her device, the user modified the registry, changing the value of foobar to 6. Your new .cab.pkg contains an .rgu changing foobar to 7 - on device, foobar will remain set to 6, as the user registry is higher in the shadow order than the .rgu) - in the case of an mxipupdate_ provxml these will override user settings.
So theoretically if we get this working we can apply updates to ANY portion of a rom via the .cab.pkg system. To XIP,SYS,or OEM without flashing a new rom. Essentially with this system working we would never need to flash again unless a major corruption occured?
Ive always been confused as to why autoupdate was included in roms still. I guess this sheds some light on it. I know that several individuals have toyed with OTA updates in the past. This could make that and a whole lot more a reality.
It seems to me silly that we aren't exploiting the MS autoupdate feature already...
I have no idea how to get it to work, but I'd love to have it working!
Yes, the .cab.pkg is the key to this whole thing, I already know how to redirect the Windows Mobile Auto Update client to look at another server, and i've studied the connection, it's a simple HTTPS connection, WinMo sends the server a manifest of all the .dsm's contained within your ROM, with version info, then the server checks against it's internal list of packages, if it finds an update, it pushes a URL to the device, which then triggers a download (it requests you to plug in the activesync cable if you've not checked the box to 'use my data connection for updates') - once the .cab.pkg is downloaded, it's checked against the signatures on the system, once verified the system reboots into the ULDR, and the update is applied.
I've attached a NetCF2 package to my first post, I can't get it to deploy on my ROM (fails during validation step) but it contains the modules in MNGE format, if we can decipher that format there's a whole bunch of goodies that will become available...
Also these packages can even be used to update the radio rom, it seems. So essentially everything but the Bootloader/ULDR can be updated with .cab.pkgs. It even looks like we can resize existing packages (i.e. remove files or modules from the ROM entirely) - this is something we can't do at all right now without a flash!
Da_G said:
Yes, the .cab.pkg is the key to this whole thing, I already know how to redirect the Windows Mobile Auto Update client to look at another server, and i've studied the connection, it's a simple HTTPS connection, WinMo sends the server a manifest of all the .dsm's contained within your ROM, with version info, then the server checks against it's internal list of packages, if it finds an update, it pushes a URL to the device, which then triggers a download (it requests you to plug in the activesync cable if you've not checked the box to 'use my data connection for updates') - once the .cab.pkg is downloaded, it's checked against the signatures on the system, once verified the system reboots into the ULDR, and the update is applied.
I've attached a NetCF2 package to my first post, I can't get it to deploy on my ROM (fails during validation step) but it contains the modules in MNGE format, if we can decipher that format there's a whole bunch of goodies that will become available...
Click to expand...
Click to collapse
So in theory, there could be a central place for SYS/XIP packages, where as and when new XIP/SYS updates come out, they can be uploaded and pushed to every device?
Have you worked out how to create the cab.pkg files, or is the one you've attached one taken from platform builder?
Just a thought:
Could the MNGE headered files not be replaced by files from a converted module, thus getting around the problem of what the hell the MNGE format does? Sure, we lose the ability to have modules instead of files, but it does bring more immediate benefits to your findings...
EDIT:
The MNGE headered files are smaller than the MZ equivalents... Are they simply a compressed version?
Yep, not pushed though as that needs to be triggered via an OMA-DM SMS message, and it's not practical for someone to maintain a database of all our numbers for such a purpose.. but easily though settings - autoupdate
I am able to extract files from .cab.pkg with winrar and 7zip, not able to create them just yet.. working on that. This one came from a blue birdy.
The MNGE headered files could indeed be replaced by a converted module, but in this case, there's a different reason for needing to convert from MNGE -> MZ,
It appears to me as though the file size difference had to do with the PE executable headers that are missing..
Da_G said:
Yep, not pushed though as that needs to be triggered via an OMA-DM SMS message, and it's not practical for someone to maintain a database of all our numbers for such a purpose.. but easily though settings - autoupdate
I am able to extract files from .cab.pkg with winrar and 7zip, not able to create them just yet.. working on that. This one came from a blue birdy.
The MNGE headered files could indeed be replaced by a converted module, but in this case, there's a different reason for needing to convert from MNGE -> MZ,
It appears to me as though the file size difference had to do with the PE executable headers that are missing..
Click to expand...
Click to collapse
Oh right. It's not hard to just check for updates every so often.
I just ran Cab2OEM on the cab.pkg files, and it extracts fine. So cab.pkg files are just cab files in terms of compression.
Is that because there are more up to date MNGE file versions than the MZ equivalents?
Is it just a case of replacing the file headers? *opens up hex edit*
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Da_G said:
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Click to expand...
Click to collapse
Where are you getting your MNGE files from? Or does your blue birdy wish to remain anonymous?
EDIT:
Looking at the hex, apart from the file header, the main difference seems to be that the whitespace has been removed in the MNGE version...
l3v5y said:
Where are you getting your MNGE files from? Or does your blue birdy wish to remain anonymous?
Click to expand...
Click to collapse
Do you know what a "hint" is ? I'd say quit asking... If he wanted to say who the birdy was, he would. Thank you!
Da_G said:
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Click to expand...
Click to collapse
put cabarc.exe intoa folder, then create a new subfolder called "package", put inside all files you need for package (dsm and rgu also!)
open in cabarc.exe root folder a dos prompt and write:
cabarc.exe N new_pack.cab.
pkg package\*.*
some seconds and you'll have the .pkg file ready, but nothing I know on how to install by the rom!
Thanks for the input ervius!
Installing into the ROM is simple, .cab.pkg is treated similar to a .cab by Windows Mobile, simply copying to device and clicking on it in file explorer allows you to install - device will authenticate signature, then do some further checking (i think checking on current packages in device by .dsm) - then once validated reboot into ULDR to apply update.
ervius said:
put cabarc.exe intoa folder, then create a new subfolder called "package", put inside all files you need for package (dsm and rgu also!)
open in cabarc.exe root folder a dos prompt and write:
cabarc.exe N new_pack.cab.
pkg package\*.*
some seconds and you'll have the .pkg file ready, but nothing I know on how to install by the rom!
Click to expand...
Click to collapse
for posted netcf example, the optimum is:
cabarc.exe - 20 N new_pack.cab.
pkg package\*.*
so, header (-s 20 reserve space for sign!), is same size, but how and with , sign it, I don't know!
Da_G said:
Thanks for the input ervius!
Installing into the ROM is simple, .cab.pkg is treated similar to a .cab by Windows Mobile, simply copying to device and clicking on it in file explorer allows you to install - device will authenticate signature, then do some further checking (i think checking on current packages in device by .dsm) - then once validated reboot into ULDR to apply update.
Click to expand...
Click to collapse
shure, but now maybe we have to fight against right sign code!?!?
bye!
Yes, I think the ROM will need be cooked with additional certs, in \SYS\Metadata\DefaultCerts.dat - these appear to be the certs that are being checked against. So we can replace with SDKCerts, sign .cab.pkgs with that, should be good!
Da_G said:
Yes, I think the ROM will need be cooked with additional certs, in \SYS\Metadata\DefaultCerts.dat - these appear to be the certs that are being checked against. So we can replace with SDKCerts, sign .cab.pkgs with that, should be good!
Click to expand...
Click to collapse
ok, go to work then, I ', yet thiniing about oldstyle buildos with this all new features of visualkitchen without platformrebuilder if someone wants use oldstyle (maybe I'm at my goal!)
ervius said:
ok, go to work then, I ', yet thiniing about oldstyle buildos with this all new features of visualkitchen without platformrebuilder if someone wants use oldstyle (maybe I'm at my goal!)
Click to expand...
Click to collapse
Can we not look at removing the signing check the same way cmonex did for the kernel? Or is that the same signing check?
Just so I am getting this right? I could cook a ROM with a custom DaveShaw https update server IP and then provide automatic updates to my ROM, bug fixes, new build release, normal CABs, etc. all using Windows Update from my website??
That would be damn useful, no more re-flashes
Dave
probably to create MNGE from module folder we just need
copy /b imageinfo.bin + S000 + S001 + ... module.dll
and add MNGE header to the beginning of the file.
but maybe i am wrong and this will not work in all cases.
at least when i converted in this way dll module from 21725 to MNGE and compared it to the same file which was originally in MNGE format, there were 0 differences.
MODERATORS: Please humor me on this... instructional writing is NOT something I'm good at, so I spend many hours writing these pages. It will take time - many weeks - for the entire "how to" to be completed. I think it'll be worth it.
PLEASE DO NOT REPLY TO THIS POST. I've created a separate thread for discussion and Q&A:
http://forum.xda-developers.com/showthread.php?p=34234642
This is NOT a new wiz-bang overvolting underclocking souped up power ranger wonder-kernel. If you're looking for a kernel to load on your phone for daily use, this thread isn't for you.
This is a thread for developing a kernel that works in the Samsung Galaxy Note II for AT&T (SGH-i317.) That previous sentence is carefully phrased: The thread is for DEVELOPING the thread. It's not a "download this kernel" thread. As a matter of fact, I will NOT be posting a precompiled kernel or installer for the kernel.
I'm hoping to try and share my experience making these things so that YOU can do it on your own... without my help. In that regard, this will likely start as a "HOWTO" type of thread, and hopefully move into actual development discussion. On the other hand, it is NOT a classroom.
My intended audience includes existing developers with little android experience, or even experienced android developers that would like to tinker with kernels. I'm sure even non-developers who want to learn might be able to pick something up from this. I fully expect that I'll be writing things that people already know - and much of this will be boring "review" for many readers. Remember that while "you" might already know something, someone else reading it might not. (This is something I often forget myself. When it happens, please remind me.)
There are a couple things you should know about me...
First and foremost, you have to understand that I'm not a "people person", I'm sarcastic, and I'm a horrible teacher. I also expect a certain amount of technical ability. If I provide a suggestion that you go to "http://source.android.com/source/index.html" and follow the instructions for setting up a build environment on a linux-based machine, I expect you to understand that you need a web browser to go to the URL, that you have to type the URL into the browser (or just click the link), and that you read the webpage and figure out that "Initializing the Build Environment" is probably the proper thing to click. If a person has issues with that, then there really isn't anything I can do to help. It might be because they are beyond all help, or it could be because I'm a grumpy engineer. Either way, I'm sorry - please move on.
I hope to retain the patience to walk people through setting up a github account, branching a stock kernel, pulling that kernel source to your own machine, insert the files for a ramdisk, compiling that kernel, manually pushing that kernel to your own device (for testing), writing an edify script (for installing the kernel), and packaging it all together for installing via a recovery-type program.
Once we have a working kernel, I intend to walk any remaining readers through the process of "cherry picking" (not nose picking) changes from another open source project into this kernel. This is a fairly common practice among open source developers where individual changes from one project can be copied to another. Many of the kernels posted on XDA are made up of nothing but cherry picking.
Then, I hope to add some simple modifications to that same kernel, and explain what those modifications are, why the are done, and HOW they are done. There will be no "new" features being added here... just reinventing the wheel (at least as far as I'm planning.) Actually, these "modifications" will probably be copies of the work of another for the sake of simplicity (and the fact that most of the simple things worth doing have likely already been done.)
There are many steps in between all these, of course. Kernel logs, kernel configs, troubleshooting, recovery, etc.
You will need to have a linux machine suitable for building linux kernels. It doesn't matter if you have a dedicated machine for it, a "dual boot" system, or even if you run it in a virtual machine. (It just so happens that I typically run it in a VM.) We aren't building the entire linux system, just the kernel, so you won't need 50GB of free drive space. No, I don't know exactly how much space you'll need. Plan on 15GB and you should be fine. You'll have to know what adb is, have it installed, AND have it working. That's something I won't walk anyone through - there are plenty of tutorials on the web for it and if you can't get that basic part done without help, you should consider just using your android phone and not developing with it. Really, "adb" is a fundamental android tool and is worth understanding even if you don't want to tinker with kernels.
You will need to have some tools (a toolchain) installed.. However, we'll get to that a bit later. There are MANY prebuilt toolchains floating around, and there's no reason not to take advantage of them. A text editor on that linux machine will be useful. I won't make any particular suggestions, but if you have no idea what to use, "gedit" is a servicable editor, is included in most linux distros, and is good enough.
Keep in mind that I have a real life outside of XDA. A full time job, a wife, two kids (7 & 9) and my own hobbies. There might be significant delays between "chapters" of this HOWTO. There's really nothing I can do about that. It's life.
What do I expect to get out of this? I expect that people LEARN methods, ideas and concepts... and that we DISCUSS and SHARE development. I'm freely giving my knowledge and the lessons I've learned, so I expect that people who learn from it will also give that knowledge freely to others. If this thread gets you going with building kernels and you figure out how to solve some problem as a result, I expect that you'll share that solution just as freely. I expect that when you start building kernels for other devices, you don't bribe people with "extra features" if they give you money. I expect that you give back to the open source community what the open source community is giving you - with interest.
Why am I doing it and What am I getting out of it myself? I'm not completely sure, to be honest. Chances are very high that I'll get extremely frustrated from doing this and regret it. On the other hand, I hope I'll learn from the experience and that I'll learn from the people who participate in the thread. I don't care what you currently do or don't know - there's always something I can learn from you.
I also have to admit that I'm doing this because I'm tired of seeing the "development" subforums on XDA looking like app stores with support threads instead of places where people share and discuss development.
The Build Machine
A short note: There might be some confusion about the use of the "cd" command without any parameters found in a few places. These are intentional, and have the effect of "change to the user home directory." I often use these before creating a new directory to ensure I'm creating it in a "known" location.
The build machine, as I mentioned earlier, should be a linux machine. For the sake of simplicity, (and because I'm too lazy to explain each and every possible program you'll need, how to get it from each linux distro, etc) I'm going to suggest that you follow Google's instructions for setting up your environment. Not only will this ensure you have everything required (except for adb and a toolchain), but also help me to know what kind of system I'm helping with. More advanced users might want to ignore this and set up their own build environment. That's fine too - but please don't ask me to help with it later.
In case you're wondering, I'm actually creating a new virtual machine for the sole purpose of this HOWTO so that I can make sure that each step I give actually works. Here are the specs of the VM I'm creating:
Ubuntu 12.04(LTS) 64 bit on a 30GB HDD with 8GB of RAM and access to 2 processor cores.
If you are doing a fresh install of Ubuntu for this, I'd suggest, after installing it, to press the "Dash Home" button near the upper left of the screen and typing "terminal" and clicking the icon for "Terminal." This will start a command line prompt in a window. It will also put an icon for the terminal program on the toolbar along the left side of the screen. RIGHT-click this icon and choose "Lock to Launcher." This will make it easier to start the terminal program in the future. (you'll need it.) Do the same thing for "gedit" (the icon name comes up as "Text Editor") if you don't already have another text editor you prefer.
If you are doing a new Ubuntu install, I'd suggest running the "Update Manager" and installing available updates.
From this point forward, I have to make certain assumptions about the technical ability of the readers. One of those assumptions is that you'll have the technical ability to follow the instructions I give. I simply cannot spend the time and space to break down HOW a user might "create a file as the root user and copy lines to it." The honest and blunt truth is that if you aren't already able to do that, the rest of this HOWTO will be a nightmare for (and me.) This isn't to be mean, rude, nasty or anything else - we all had to learn addition and subtraction before long division.
Time to begin!
Please go to the following URL and follow the instructions for initializing a build environment on a linux (Ubuntu) machine:
http://source.android.com/source/index.html.
There are instructions here for setting up on Mac OS X, but I have to be completely honest in saying I don't know if kernel development will work in OSX - I've never tried it. You can if you want, but I won't be able to help you getting the machine set up. If you're using a linux distro other than one of the suggested Ubuntu versions, you'll need to read the instructions given and figure out how to apply them to your own installation. Not all of the packages it wants installed are actually needed for kernel development, but I'm too lazy to pick through them. They really don't take too much space.
You will probably encounter an error when you get to the step for the java JDK. (If you don't, then you aren't following the directions.) The java JDK is no longer available when following Google's instructions (and they never bothered to update the page.) Don't worry about that, as we don't need java for kernel building.
For the "Configuring USB Access" section, you'll need to specify a different line for the udev rules (so it sees a Samsung phone.) You can either add this line to the end of what Google provides, or use this single line instead of what they provide (I did the latter on my VM):
Code:
SUBSYSTEM=="usb", ATTR{idVender}=="04e8", MODE="0666"
As an alternative, you could also just copy the following line into a terminal window :
Code:
sudo sh -c 'echo SUBSYSTEM==\"usb\", ATTR{idVender}==\"04e8\", MODE=\"0666\" >> /etc/udev/rules.d/51-android.rules'
If your phone is already plugged into your computer's USB port, those changes won't take effect until after you unplug the phone and plug it back in. If you are using Ubuntu in a VM, you'll probably need to perform some action so the VM takes over that USB port and redirects it to the VM.
Some versions of Ubuntu will pop up dialogs asking what to do with the recently connected device (or even show an error or warning saying that it's unable to lock the device.) If this happens, make the dialogs go away: Select "do nothing" for prompts asking what to do, and just "OK" for any error or warning. We don't want to access the phone as a media player or camera or even as a MTP device. We only need "adb" access!
A short word about adb: We actually don't need to use adb for this. However, it greatly simplifies repeatedly copying kernel image files to the phone, running a shell on the phone, rebooting the phone, getting into recovery, etc. Anyway, if you do kernel development and don't have adb, people will laugh at you. It'd be like using a fork to eat noodles in Hong Kong.
As mentioned previously, after you get the basic machine going, you should follow the steps for initializing a build machine at http://source.android.com/source/index.html (and not worry about the error when trying to install the java JDK.) Be careful that you follow the steps for the version of Ubuntu that you have... the required package list for 10.04 is different than the package list for 12.04. (So if you blindly copy/paste the commands from the wrong sections, you'll get errors... I did when I copied the 10.04 package list for my 12.04 machine.)
There's still a few more things needed. For example, we spent the time editing those udev rules so linux would "see" the phone and we can use "adb" but we haven't actually installed "adb" yet. For that, we'll go to this link: http://developer.android.com/sdk/index.html
There's a few different bundles on this page, as well as some other things. We actually only want a very tiny subset of a subset of what's on here. Near the bottom of the page find the section for "Download for other platforms", and the "SDK Tools Only" section below that. The actual download we want is the SDK tools (only) for linux. As of my writing this, the file is named "android-sdk_r21-linux.tgz." Download that to your linux machine.
You'll then need to unarchive the "tarball" somewhere on your machine. If you don't have a great place in mind to put it already, and assuming you're using a pretty default Ubuntu system, you can open a terminal window and copy the following commands:
Code:
cd
tar xzvf Downloads/android-sdk_r21-linux.tgz
This will end up creating a directory called "android-sdk-linux" and sticking everything in it. But Wait! "adb" isn't actually included with that. So, assuming you've been following along with me so far (and if you haven't, you'll have to adjust the following commands) copy the following commands:
Code:
cd android-sdk-linux
tools/android
This will pop up a window letting you pick and choose which SDK elements you want to install. There's a LOT of stuff here, but all we need are the "Android SDK Platform-tools" (under "tools".) Check that box and click "Install 1 package..." Accept the license and click "Install." When it's done, click "Close" and then close the "Android SDK Manager" window.
To access adb, you might type something like "~/android-sdk-linux/platform-tools/adb devices" That's a pain in the rear, so how about we simplify things? Copy the following commands in that terminal window:
Code:
cd
mkdir bin
cd bin
ln -s ../android-sdk-linux/platform-tools/adb adb
cd
echo 'PATH="$HOME/bin:$PATH"' >> .bashrc
Those lines will create a symbolic link (like a windows shortcut) from the actual adb program to a private "bin" Then it tells "bash" (which is the default command interpreter in ubuntu distros) to look in that private bin for commands you type. Some users who aren't setting up brand new ubuntu systems for this might want to check that they are actually using bash and that $home/bin isn't already in their path - and adjust this accordingly. You'll need to close any 'terminal' windows you have open and restart them in order for this to take effect. (Advanced users: I know this isn't the best way to modify the PATH on a default Ubuntu box, but it's the easiest way to get it working. If you're advanced enough to know that, then you're also advanced enough to do it better on your system.)
Now, you SHOULD be able to plug your samsung android phone into that ubuntu machine, wait a couple seconds (and close any silly warning dialogs about locking the device) and adb will see your phone. To test, use the command "adb devices" (in a newly opened terminal window.) You should see something like "List of devices attached" followed by a string of letters/numbers some spaces and the word "device." For example:
Code:
List of devices attached
12a34b567c890def device
There's only one more thing left to do for this post: Install the toolchain.
A "toolchain" is a set of tools that we use to get from a bunch of source files to a kernel that can be installed. We could build a toolchain from GNU (or other) sources, but google has made this much easier for us by providing a prebuilt toolchain, and even made it easy to get. Personally, I'm EXTREMELY happy about that, as this "page" of the HOWTO is getting entirely too long. (I hope XDA doesn't crash when I copy it into a new post!)
Copy the following in a new terminal window:
Code:
cd
git clone https://android.googlesource.com/platform/prebuilt
This might take a while to download, so go ahead and distract yourself while it's going. When it's done, you've finished this post!
I had originally intended to start discussing git and github in this post, but I'm putting that off for the next post when I'll also have you fork the actual kernel source and pull the source to your build machine.
Before I go on, I'm going to take a little side trip into some open source (and GPL) concepts. I'm not a lawyer and nothing I write should be interpreted as legal advice.
The linux kernel, which is the kernel that android uses, is open source software and licensed under the "GPL." Without knowing all the legal mechanisms, this means that the kernel is freely available. It's free to use and the source is also free. Anyone who distributes a product that makes use of this kernel is REQUIRED to make the source available for free. This includes any changes that a company might make to the linux kernel. As those changes are a derived work, it must retain the free licensing of the main body. That includes even large companies like Samsung, HTC, Google, TiVo, Netgear, etc. There are some things not covered such as separately compiled kernel modules (like some ethernet drivers), but non-GPL licensed kernel modules are fairly rare.
This same rule applies to you and I. If you or I distribute a linux kernel alone or as part of a larger product, we are bound by the licensing of that kernel to make our source code available to anyone who asks. If you post a linux kernel on a website or any other form of distribution, you are legally and morally required to also ensure that the source code for that very same kernel is available.
I feel very strongly about this, so if you have a problem with it, please do me and the entire community a large favor and go away.
In addition, here on XDA, you are required to ask permission to use kernel code that another member wrote. This isn't a GPL requirement, but an XDA one. For that reason, I've asked a couple of other dev's on here if I could use some of their modifications in this HOWTO (and in the spirit of open source software, each one welcomed me to do so.) Even if it weren't a requirement here on XDA, it's always polite to ask the author if you can use their works. In almost all cases, they'll agree to it. In addition, they'll often offer tips and suggestions for using certain things. For example, they might warn you that if you want to cherry pick a certain change, you should also get another one or things will break.
Before reading this page, I want to point out that I'm fairly weak on git and github. I might be giving instructions that use the incorrect phrasing or are just simply the wrong way of doing things. If you see me doing this and you know better, PLEASE post a message on the companion discussion thread. If there's a better way, I want to know about it, and I want to update this post to reflect it. Keep in mind, though, that I'm trying to simplify things here for the HOWTO, so if your suggestion is more complex (or would require me to type two pages to explain it) I might not update this post with it.
Most linux-based kernel development these days uses a form of version control (source control, revision control, etc) that will catalog each change made to the source as a change set (or a commit.) Of these, "git" is the most prevalent. Git was written from the ground up specifically for linux development (by the folks who originally created linux) and to meet the very distributed needs of linux kernel development. For more information on git, wikipedia has a good article: http://en.wikipedia.org/wiki/Git_(software)
Along with "git" as the version control system, we'll be using Github as a free hosting service for out git repositories. Github acts like a server for projects using git for revision control. While the site is somewhat geared as a "social" code sharing website, you don't have to be a friendly person to use it. (They even let me use it!) In fact, it's perfectly acceptable to use Github as a server for a project that only you are working on. It's extremely well suited to act as a remote backup for your code projects (which, I admit, I abuse it as) and it serves one other purpose for us: It's an extremely easy way to meet the legal requirement that our linux source is freely available.
Git (and Github) offers a really nice mechanism for a person (such as a reader of this thread) to start a new git project based on the source code of someone else (such as an original kernel project I'll set up for that purpose.) In git's terms, you will "fork" my project into something that belongs to you. Internally, you're only actually creating a link in your project to a very specific version of mine, but it appears and acts like a full copy. (In ways, it's actually BETTER than a normal copy.)
You can, of course, also create a new project from scratch and add new files directly. However, it's considered a better practice to fork from an existing project if possible. This saves a considerable amount of space on the Github server as it's really only keeping a single copy of that original source. It also makes it much easier to pull in changes from the original project (and push changes back to it.) Unfortunately, it's sometimes difficult to find a "clean" branch to fork from. For the purpose of this HOWTO, I've created a clean copy of the SGH-i317 sources in a project in Github and you'll be able to fork from that.
Before forking my project, you'll first need to create a Github account if you don't already have one. Just go to http://github.com and click "Sign up for free." You'll be presented with a page reminding you of the high cost of this service (free) and asking you for a username, email address, and password. Fill in that information and click "Create an account" (which implies acceptance of their terms of service and privacy policy.) That's it. You now have a Github account.
While logged in with your Github account, you want to find a project to fork from. As mentioned, I have a pristine copy of Samsung's actual source release for the i317 set up for just this purpose. Up at the top of the Github webpage, you can search for that project. Just type "linux_kernel_sgh-i317" into that search bar, and you should get a result pointing to "garyd9/linux_kernel_sgh-i317." If you click that link, you'll end up at "https://github.com/garyd9/linux_kernel_sgh-i317"
Finally, you'll go ahead and fork a copy of that for your own use. This is a highly complicated procedure so please pay very careful attention to these instructions: On the upper left of the Github webpage, there's a button labeled "Fork." Go ahead and press it. That's it. (Some people actually mess this up. I have no idea how.) You now have, at least on the github site, your own copy of the kernel source. Leave that browser window open while doing the next step. It contains something we'll need very shortly...
(Parts of this are taken from Github's own "Bootcamp/Set Up Git" webpage at: https://help.github.com/articles/set-up-git#platform-linux I'd encourage you to go to that link and review it as well.)
Back on your build machine, open up a 'terminal' window and type the following command, replacing "Your Name" with your own name. Leave the quotation marks in place.
Code:
git config --global user.name "Your Name"
This is telling your local git software what your name is. This shows up in all your git commits, so you probably don't want to put something stupid here.
Also in the terminal window, you'll type this next command. This time, replace "[email protected]" with the email address you used to register on Github. (This is important!)
Code:
git config --global user.email [email protected]
Finally, there is one more command to type. This last one will tell git to cache (store) your Github account information so you don't have to type a password every single time you push to the server. (By default, it caches for 15 minutes. Read the Github bootcamp article linked above to find out how to extend that.)
Code:
git config --global credential.helper cache
Before we tell git to actually pull the source code from Github, there's one more little organizational thing I like to do. I like to keep all files related to building of a kernel in a single directory on my linux machine. There's more than just a kernel required for building the kernel! So, go ahead and copy the following commands to your terminal window:
Code:
cd
mkdir kernel-howto
cd kernel-howto
Now, everything related to this HOWTO can be put into one place: That newly created "kernel-howto" directory that we also changed into.
Now, it's finally time to get the source code to your build machine. Remember I suggested leaving the browser window open on the github page showing your newly forked project? Well, what we actually want is the page URL. It should look something like "https://github.com/yourUserName/linux_kernel-i317" "yourUserName" should be your Github username. We need that URL and add ".git" to tell git what to fetch. In this case, we'll be using the "clone" command which basically just tells git to make a copy of the remote project. You won't be able to just copy/paste the following command into your terminal window, because you'll have to change at least the yourUserName part of the URL. Go ahead and type this command (making the change needed.)
Code:
git clone https://github.com/yourUserName/linux_kernel_sgh-i317.git
When the command finishes, you'll have a new sub-directory inside of "kernel-howto" that's called "linux_kernel_sgh-i317". Go ahead and change into this directory.
Code:
cd linux_kernel_sgh-i317
There's only one last thing I want to do here before ending this post, and that's to ensure anyone following this HOWTO is able to push changes. To do this, just copy the following commands that will create an empty (and meaningless) file, add it to git, commit it as a change, and push it to the server.
Code:
touch HOWTO
git add -A
git commit -m "test commit"
git push
After the "git push" command is sent, you should be prompted for your github username and password. After you enter that information, you should see something similar to the following (the characters before "master->master" will be different):
Code:
To https://github.com/yourUserName/linux_kernel_sgh-i317.git
e3d49d2..ea9fbe9 master -> master
You can also go back to your Github page for this project to see the commit on Github. If you aren't on that page anymore, log in to Github, and click the linux_kernel_sgh-i317 repository on the right side of the webpage. Then find and click the "Commits" link (which is between "Files" and "Branches" a few lines down from the top of the page.) The top commit will be the "test commit" that you did on your build machine and pushed to Github.
The kernel isn't quite ready to be compiled. We'll get to that in the next post. In the meantime, if you'd like to learn more about using git and github, there's an interesting interactive walk-through style tutorial at http://try.github.com.
gitignore
When we work with the kernel, we will generate a lot of temporary files that we don't want to put into the git system or upload to GitHub. To have "git" exclude those files from consideration, a file (or set of files) called ".gitignore" should be created that tells git what to ignore. A more complete explanation of the .gitignore file can be found here: https://help.github.com/articles/ignoring-files
Instead of having you create and edit several different ".gitignore" files, I've made a patch that you can apply directory to your kernel tree, using git, that will add those .gitignore files. The following commands, which you should copy to a terminal' window, will change into the kernel source directory, fetch the patch file (using wget), apply the patch (using git am) and then delete the patch file (as it's no longer needed.) If you have the kernel source in a different directory, you'll have to adjust one of the "cd" commands accordingly. After deleting the patch file, we push those changes to the github server. I'll explain the various git commands more in depth in a later section of this HOWTO.
Code:
cd
cd kernel-howto/linux_kernel_sgh-i317
wget https://raw.github.com/garyd9/kernel-howto-misc/master/add_gitignores.patch
git am < add_gitignores.patch
rm add_gitignores.patch
git push
Just a few more things...
The next item to mention before compiling the kernel is the kernel config file. Linux, as a kernel, can be used on literally thousands of different configurations. However, each platform might require a different kernel. How this works is that when the kernel is compiled, a kernel configuration is supplied that tells the compiling tools which drivers to include, how to configure them, what kind of processor the target platform has, etc. For now, we'll just use a default configuration file that Samsung provided with the kernel source. That configuration is called "t0att_04_defconfig" and (luckily for us) actually seems to be accurate. When building for another device, don't assume that the kernel configuration provided with the source is accurate... I've had at least one case when the configuration Samsung provided for their own device simply didn't work for that device. Later in this HOWTO, we'll be discussing the kernel configuration much more in depth, and even making changes to it.
Next, we need to talk about that prebuilt toolchain we downloaded earlier in the HOWTO. When compiling the kernel, we need to tell the system to use that very specific toolchain for doing the work. What we're actually doing is called "cross compiling", which means we're compiling for a different platform than the compiler itself is running on. In this case, we're running on an x86 (or x86_64) platform, but compiling for the "arm" platform. To tell the system which toolchain to use for cross compiling, we will use a command line parameter that defines "CROSS_COMPILE" to point to the proper toolchain files. Assuming you've been following this HOWTO (and if you haven't, you'll have to adjust this accordingly every place it occurs) that will look like this: "CROSS_COMPILE=$HOME/prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi-"
Finally, there's just one more thing before we can compile. The actual tool we use to kick off compiles is called "make." Make doesn't really know much about compiling software, but it's a great tool for figuring out when things need to get done and for doing them. By default, "make" reads a file called "Makefile." A Makefile describes how files might be dependent on each other, how to compile files and put things together. Because of the magic of a Makefile, you won't have to recompile the entire kernel each time you make a minor change. Instead, "make" will only recompile what's needed to include updates to the source.
Are you ready for actually compiling? Okay, go ahead and copy these commands. There are two commands here. The first one just prepares the kernel config file, and the second one compiles the kernel. As always, if you've made changes to paths (locations) of things, you will have to edit these commands.
Code:
cd
cd kernel-howto/linux_kernel_sgh-i317
make arch=arm t0att_04_defconfig CROSS_COMPILE=$HOME/prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi-
make CROSS_COMPILE=$HOME/prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi-
Depending on the speed of your machine, this can take a while to complete. You'll be witness to many different files getting compiled and linked together as it runs. When it's done, you'll see something like this in your terminal window:
Code:
OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready
Building modules, stage 2.
MODPOST 7 modules
CC arch/arm/mvp/commkm/commkm.mod.o
LD [M] arch/arm/mvp/commkm/commkm.ko
CC arch/arm/mvp/mvpkm/mvpkm.mod.o
LD [M] arch/arm/mvp/mvpkm/mvpkm.ko
CC arch/arm/mvp/pvtcpkm/pvtcpkm.mod.o
LD [M] arch/arm/mvp/pvtcpkm/pvtcpkm.ko
CC drivers/interceptor/vpnclient.mod.o
LD [M] drivers/interceptor/vpnclient.ko
CC drivers/net/wireless/bcmdhd/dhd.mod.o
LD [M] drivers/net/wireless/bcmdhd/dhd.ko
CC drivers/net/wireless/btlock/btlock.mod.o
LD [M] drivers/net/wireless/btlock/btlock.ko
CC drivers/scsi/scsi_wait_scan.mod.o
LD [M] drivers/scsi/scsi_wait_scan.ko
Congrats - you've just compiled a linux kernel for Android.
There's only one problem: It's completely useless in it's current form. If you were to install that kernel on your phone, it simply wouldn't boot. What it really needs now is a ramdisk.
The next sections (plural) in this HOWTO will discuss what ramdisks are, how to extract one from an existing kernel, and how to package a ramdisk image with a kernel to make a flashable boot image. We also need to talk about partitions, block devices, making backups, using the 'dd' command, and a few other things.
You didn't think this was easy, did you?
It would be a great idea to get familiar with the "adb" tool; particularly the "shell", "pull" and "push" commands. (Use Google to search for "how to use adb.") Over the next several sections, we make use of these adb commands to copy files to and from the Android device and to interact with the device's operating system. This document also assumes that the device has already been rooted.
To get a "rooted adb shell" on a Samsung TouchWiz device, you'll have to have your device attached to and seen by your linux build machine, and use the command "adb shell" in a terminal window. Now, turn on the display for your android device and unlock it (if required.) Finally, back in the terminal window on the linux build machine, type the command "su". You Android device might pop up a prompt asking to give permission for the root access. If it does, grant the access. After the su command, the prompt should have a "#" symbol (instead of a "$" symbol.)As you might already have realized, just having a linux kernel is interesting, but not extremely useful. The kernel can boot, but once it does, it needs a filesystem to load programs from, to store data, etc. In the UN*X world (which linux is derived from), there's a single root to all filesystems, and other filesystems mount (attach) to that single root. Android (and many other embedded systems) use a root filesystem that isn't real - instead it's a virtual filesystem called a RAM Disk. Briefly, a RAM Disk is a portion of the devices RAM that's put aside and make to look like a persistent storage device. Unlike a real storage device, the RAM Disk (usually typed as "ramdisk") is completely erased each time the device is rebooted.
Linux allows taking that concept to a higher level by allowing a kernel to specify an initial ramdisk image. The image is a snapshot of a portion of an actual filesystem which is used to create the ramdisk. That, in turn, is specified in the kernel configuration to be the "initramfs" - which means that it becomes the root filesystem for the entire device. ("initramfs" is short for "Initial RAM FileSystem.") The initramfs image is embedded within the kernel image, and each time the kernel boots up, it uses that image as a root filesystem. Being it's a ramdisk, we can even make changes within the root filesystem after the device boots, but those changes will be wiped out the next time the device reboots.
Some other initramfs features supported by linux, but that don't seem to be used in Android, include the ability to switch root filesystems from within the initramfs, and the ability to store the initramfs as a seperate file (instead of embedding it in the kernel image.)As well as being an anchor for all the persistent (and non-persistent) storage devices in the phone, the Android initramfs also kick-starts the Android OS itself. (Remember that "linux" is a kernel, not the OS!) It usually contains some lower level system utilities (such as the files needed for adb functionality), a few kernel drivers that might not already be compiled into the kernel, a few config files used at the early stage of booting the device, and scripts used to get the rest of the system going. On some Android devices, the initramfs also contains the files needed for booting the phone recovery. (On other phones, including most that have been released since Android 4.0, recovery has it's own private kernel.) Remember, however, that the initramfs uses RAM, so it shouldn't be bloated with things that could be stored in real persistent storage.
Another aspect of using the initramfs that can be useful is that it allows booting at least a minimally functioning linux system in cases when the rest of the Android OS doesn't load properly. That minimal system could even be used to manually flash a different kernel, recovery, or even the entire firmware for the device.
Now that we have an idea what the initramfs is, and why we need one, the question is how to obtain one. In most cases, it's usually better to at least start with a copy of the initramfs already used by our Android device. This way, we know that everything needed in the initramfs already is there. We can (and will) make changes to it later. The best way to get an unmodified copy of the kernel's initramfs is to use a special tool to pull the ramdisk out of the kernel image.
For this HOWTO, we'll start with a "stock" kernel image (instead of a custom image that might have already been modified.) This gives us a known starting place, and ensures that files haven't been modified by a previous kernel developer. There are a few ways to get this kernel image. Often times, the easiest is to find one online in either a "recovery" style zip file, or packaged in a proprietary file such as Samsung's ODIN tar files. If you haven't installed a custom kernel on your device, you can also usually extract the kernel image directly from our phone.
The remainder of this section will be devoted to the information needed to extract a kernel image from the phone. Even if you don't have a stock kernel installed, it would be useful to read the information and go through the steps. The knowledge (and experience) will be useful when we have a kernel to install. However, different types of devices might have different device configurations, so some of this information might not be accurate for every device.
The kernel image (along with the device modem, firmware, data, etc) is stored on a special type of "flash" RAM that doesn't lose it's memory when power is removed. It's very much like any type of removable memory (such as a SDCard, USB thumb drive, etc.) Unlike those examples, however, this special storage is built into the phone or tablet. This guide refers to that flash memory as "persistent storage" which indicates storage that retains data even when power is off or the device resets.
The persistent storage on our devices is treated in linux much as a hard drive is treated on a PC. As with a hard drive, the persistent storage is split into logical "partitions" and each partition can be (and usually is) treated as a separate logical storage device. Our devices might contain 16GB of persistent storage space, but that 16GB is partitioned into a dozen or more smaller chunks providing separate locations for storing the kernel, the bootloaders, the files needed for recovery, the system files, the modem firmware, user data, etc.
In linux, physical (and virtual) devices have drivers in the kernel, and are represented by a special virtual filesystem usually mounted (attached) to a directory called "dev" in the root filesystem. Modern versions of the linux kernel usually separate the devices into sub-directories based on category. Persistent storage devices are almost always "block" devices. Finally, individual block devices are given names based on their driver and other ordering information as determined by the driver. Actual devices being represented are called "nodes" but are often also just called "device files" to keep the description simple. (This is an intentional simplification of how the device nodes are created.)
Based on the above paragraph, we know that the device nodes for the persistent storage can be found in a directory called "/dev/block" on our device. To see this, we'll need a rooted adb shell on your device (or rooted shell via a terminal emulator installed on the phone) and to copy the following command:
Code:
ls /dev/block
On most modern Android devices, we'll see a device node for "mmcblk0" which represents the entire built-in persistent storage. There will also be devices for each partition, such as "mmcblk0p1", "mmcblk0p2", and so on. One of those device nodes represents the partition used to store the kernel image (often called the boot image) - and we need to figure out which one. This is an area that can get annoying, as there doesn't seem to be a simple generic way to determine which partition the kernel is stored on.
(NOTE: If someone can give me a good way to determine the boot/kernel partition regardless of platform, please let me know and I'll document it here.)
If we poke around the directory tree in /dev/block/platform, we might be able to find a sub-directory buried in there called "by-name" and, at least on some platforms, we can determine the boot/kernel partition from there. We need to be somewhat careful, as the boot LOADER is most certainly not something we want to mess with - and many platforms can make it confusing when they refer to the kernel partition as the "boot" partition. On the Galaxy Note II, that "by-name" sub-directory is in /dev/block/platform/dw_mmc, and you can see the full list of named partitions by copying the following command to a rooted adb shell running on your device:
Code:
ls -l /dev/block/platform/dw_mmc/by-name
The result of running that command will include the following (or something similar)
Code:
lrwxrwxrwx 1 root root 20 Nov 18 12:39 BOOT -> /dev/block/mmcblk0p8
lrwxrwxrwx 1 root root 20 Nov 18 12:39 BOTA0 -> /dev/block/mmcblk0p1
lrwxrwxrwx 1 root root 20 Nov 18 12:39 BOTA1 -> /dev/block/mmcblk0p2
lrwxrwxrwx 1 root root 21 Nov 18 12:39 CACHE -> /dev/block/mmcblk0p12
lrwxrwxrwx 1 root root 20 Nov 18 12:39 EFS -> /dev/block/mmcblk0p3
lrwxrwxrwx 1 root root 21 Nov 18 12:39 HIDDEN -> /dev/block/mmcblk0p14
lrwxrwxrwx 1 root root 21 Nov 18 12:39 OTA -> /dev/block/mmcblk0p15
lrwxrwxrwx 1 root root 20 Nov 18 12:39 PARAM -> /dev/block/mmcblk0p7
lrwxrwxrwx 1 root root 21 Nov 18 12:39 RADIO -> /dev/block/mmcblk0p10
lrwxrwxrwx 1 root root 20 Nov 18 12:39 RECOVERY -> /dev/block/mmcblk0p9
lrwxrwxrwx 1 root root 21 Nov 18 12:39 SYSTEM -> /dev/block/mmcblk0p13
lrwxrwxrwx 1 root root 21 Nov 18 12:39 TOMBSTONES -> /dev/block/mmcblk0p11
lrwxrwxrwx 1 root root 21 Nov 18 12:39 USERDATA -> /dev/block/mmcblk0p16
This gives us the information we need to determine the kernel partition. It's called "BOOT" and the actual device node is /dev/block/mmcblk0p8. Similar information can be found for other device types as well. For example, on a Galaxy Nexus running CM10, the sub-directory containing this information is /dev/block/platform/omap/omap_hsmmc.0/by-name and on a Nexus7 running pure AOSP 4.2, a similar (but more crytpic) set of information is at /dev/block/platform/sdhci-tegra.3/by-name. Sometimes, we just need to search to find the proper partition.
Once we have the name of the device node for the partition containing the boot/kernel partition, we can use that to actually convert and copy the entire partition to a normal file. The command for doing this is "dd" and the parameters we use need to be typed carefully. A simple typo here can result in overwriting a valid kernel unintentionally. Be paranoid! The following command, which should be copied to a rooted adb shell, assumes that the "boot" (kernel) partition is mmcblk0p8, and stores a copy of that entire image to a file called orig_boot.img on the internal user storage (/sdcard):
Code:
dd if=/dev/block/mmcblk0p8 of=/sdcard/orig_boot.img bs=4096
"if" designates the "input file", "of" designates the "output file" and "bs" is the block size. The block size parameter isn't really needed here, but it tends to speed up the copy by telling "dd" to copy 4096 bytes at a time (instead of 1 byte at a time.)
We now have a boot image (kernel image) that we can use to extract an initramfs from. One last step is needed to get that file transferred to our linux build machine. Open a new terminal window and copy the following commands into it:
Code:
cd
cd kernel-howto
adb pull /sdcard/orig_boot.img ./orig_boot.img
This will change into the kernel-howto directory on your build machine, and pull the recently made image to your build machine. (If using the 'Terminal' program in Ubuntu, you can open a new terminal window by right-clicking on an existing terminal window and selecting "Open Terminal." You can also right-click on the 'Terminal' icon in the Launcher bar and select "New Terminal.")
If you are using a different starting image, such as from an ODIN compatible tar file or CWM Recovery compatible zip file, please find the kernel image within that archive, extract it, and place it in the proper directory, renaming it to match. If you can't find a stock image (or don't feel like searching for one) just execute the following commands to get a kernel image that was extracted from a stock LJ2 SGH-i317 (AT&T variant):
Code:
cd
cd kernel-howto
wget https://github.com/garyd9/kernel-howto-misc/raw/master/boot_orig.img.gz
gunzip boot_orig.img.gz
When Things Go Wrong...
Before going any further, there's something extremely important we must do: We must plan for things to go wrong. No matter how careful we might be, and no matter how smart or talented a person might be, something can (and often will) go wrong. Perhaps a kernel will get corrupted when building, or maybe it will accidentally be copied into the wrong partition on the phone. Bad Things happen all the time, and it's best to be as prepared as possible.
If the bootloader partitions get overwritten or corrupted, there isn't much we can do. Without the bootloader, nothing else can work. If that happens, the only resort might be an actual repair service. That repair will likely cost money, as no warranty covers damage like this caused by the customer - not even the "accidental" things sold by companies like SquareTrade. It's one of the risks that people take when working at this level. (There are people working on modifications that would allow booting the phone even if the bootloader is overwritten, but I'm not aware of any of them being available for the SGH-i317.)
Most of the other partitions can be backed up and restored with a recovery tool. At the present time, I'd recommend using TWRP over CWM Recovery for two reasons: First, it supports backing up the EFS partition that CWM Recovery doesn't. Second, it allows restoring individual partitions instead of restoring an entire backup. If you corrupt the kernel, you can use TWRP to only restore the corrupted boot partition without restoring everything else. It is my understanding that the TWRP people are working on adding support for backing up the RADIO firmware partition as well.
It's possible to accidentally overwrite the recovery partition when performing certain actions. With no recovery, any backup made with TWRP or CWM Recovery would be hard to restore. To deal with that, we'll go through some steps later in this section to make images of certain partitions (including the recovery partition) in order to be able to restore them later if they are overwritten. As well, for most Samsung devices, there are ODIN compatible images containing only the recovery partition. Nexus devices can use a tool called "fastboot" to boot into a recovery image that isn't even installed on the phone.
Finally, at least with a Samsung device, a last resort is to fall back to ODIN to restore the phone to a factory state. That's only useful if a full ODIN firmware is available (or at least an ODIN image containing the partition you need to restore.) At the time of writing this guide, there's no full ODIN image available for the SGH-i317. When you see one, be sure to download it and keep it around. Also learn to use the ODIN tool. Having the image doesn't do much good with you can't use the tool for flashing it! ODIN relies on the bootloader of the device not being corrupted, so even ODIN is useless if you corrupt the bootloader.
In order to manually create images of some of the device partitions, first determine which partitions you want backups of. Keep in mind that the size of the images can get quite large, so don't use this method for the USERDATA (/data) partition. I'd suggest backing up the boot/kernel, recovery, radio, param, and efs partitions using this method. Use the information in the previous section to determine the device nodes for each of those partitions, and then use a command nearly identical to how we copied the stock kernel image to a file. If you have an external sdcard in your device, this is the perfect place to store those images. If we are backing up partitions from a SGH-i317 to an external sdcard, here are the commands we'd copy into a rooted adb shell to get those images:
Code:
cd /mnt/extSdCard
mkdir backup_images
cd backup_images
dd if=/dev/block/mmcblk0p3 of=mmcblk0p3.img bs=4096
dd if=/dev/block/mmcblk0p7 of=mmcblk0p7.img bs=4096
dd if=/dev/block/mmcblk0p8 of=mmcblk0p8.img bs=4096
dd if=/dev/block/mmcblk0p9 of=mmcblk0p9.img bs=4096
dd if=/dev/block/mmcblk0p10 of=mmcblk0p10.img bs=4096
Notice that we're naming the images the same as the device nodes they came from (the of= parameter.) A person could also name them efs.img, param.img, boot.img and recovery.img if they want. For the SGH-i317 device, the 4 images will use a combined 132MB of storage space.
To restore one of these images, we'd use the same 'dd' command, but reverse the "if" and "of" parameters. For example, if we corrupted our recovery partition (perhaps by copying a normal boot kernel to it), we could boot the phone normally, get a rooted adb shell, and type the following commands:
Code:
cd /mnt/extSdCard/backup_images
dd if=mmcblk0p9.img of=/dev/block/mmcblk0p9 bs=4096
Again, be paranoid. I can't possibly stress enough the need for backing up, and doing so frequently.
Now that we have a copy of a factory boot image, we'll unpack that image into it's component parts. There are three tools used for manipulating boot images that we should download and place somewhere on our path. The following commands, copied into a terminal window on our build machine, will take care of that:
Code:
cd ~/bin
wget https://github.com/garyd9/kernel-howto-misc/raw/master/mkbootfs
wget https://github.com/garyd9/kernel-howto-misc/raw/master/mkbootimg
wget https://github.com/garyd9/kernel-howto-misc/raw/master/unpackbootimg
chmod a+x mkbootfs
chmod a+x mkbootimg
chmod a+x unpackbootimg
Before going on, it's important to give credit where due: The three boot image utilities were compiled from sources in the cyanogenmod 10 repository. mkbookimg and mkbootfs appear to originate from AOSP, and unpackbootimg appears to originate from Koush (of ClockworkMod fame.) The sources for cyanogenmod can be found at https://github.com/CyanogenMod.
The three tools are:
unpackbootimg - used to unpack a full boot/kernel image into the ramdisk, kernel, and other files containing certain parameters.
mkbootfs - used to combine or archive a directory structure into a ramdisk image. (The same thing can be accomplished with the cpio utility)
mkbootimg - used to combine a kernel and initramfs image into a boot image that can be installed on a device.
To unpack the parts of the stock boot image, open a terminal window on your build machine and copy the following commands. As always, if paths or filenames have been changed from this guide, the commands should be altered to correspond to those changes.
Code:
cd ~/kernel-howto
mkdir stockbootimg
unpackbootimg -i boot_orig.img -o stockbootimg
ls -l stockbootimg
Here, the -i parameter is used to specify "boot_orig.img" as the image to unpack, and the -o parameter specifies that the results of unpacking the image should be placed in a sub-directory called "stockbootimg".
In the list of commands above, the last one just will list the files unpacked in the "stockbootimg" sub-directory. At this point, the only one of those files that is significant to us is the one named "boot_orig.img-ramdisk.gz." That file is the actual compressed ramdisk image used as the initramfs in the stock kernel. The file ending with "zImage" is the actual stock kernel, and the remaining files contain boot image values that are beyond the scope of this guide. (If someone is planning on manipulating things like the page size or base address, they likely have a skill/knowledge level far exceeding the audience of this document.)
There's really nothing special about the compressed ramdisk image. It's simply a directory structure archived with the "cpio" utility and compressed with gzip. In fact, if you copy the following command to your terminal window, you'll be able to see what's inside:
Code:
gunzip -c ~/kernel-howto/stockbootimg/boot_orig.img-ramdisk.gz | cpio -it
Later, we'll extract the files from the stock initramfs and make some changes. For now, however, we'll just use the compressed ramdisk as it is and combine it with the kernel we built earlier. (Normally when rebuilding the kernel, the kernel modules in the initramfs are replaced with ones from the new kernel compile, but we'll skip that step for now.)
Assuming you've already built the kernel (as described a few sections ago), the finished kernel should exist as ~/kernel-howto/linux_kernel_sgh-i317/arch/arm/boot/zImage. The following command will combine that kernel we compiled with the stock initramfs image we just unpacked:
Code:
cd ~/kernel-howto
mkdir first_bootimg
mkbootimg --kernel ~/kernel-howto/linux_kernel_sgh-i317/arch/arm/boot/zImage --ramdisk ~/kernel-howto/stockbootimg/boot_orig.img-ramdisk.gz -o first_bootimg/boot.img
The "--kernel" and "--ramdisk" parameters should be pretty obvious, and the "-o" parameter specifies where to put the final boot image. There are several other parameters for mkbootimg, but none of them are needed here as the defaults work fine. In most cases, the values for the parameters can be found by examining the files unpacked after running unpackbootimg.
We now have a fully functional kernel that we can install on our device and test with. Connect your phone to the build machine, and run the following commands to copy the new boot image to your phone (but not yet installed it.) This uses adb to first create a new sub-directory to place the image, and then to push the new image into that location
Code:
adb shell mkdir /sdcard/first_bootimg
adb push ~/kernel-howto/first_bootimg/boot.img /sdcard/first_bootimg/boot.img
WAIT! DON'T INSTALL THIS KERNEL IF YOU DON'T HAVE A BACKUP. THIS KERNEL WON'T WORK PROPERLY AND YOU WILL NEED TO RESTORE A BACKUP KERNEL FOR FULL FUNCTIONALITY
The first step to installing a new kernel should always be to ensure that a backup has been made of the device. Once a good backup has been made, there are actually multiple ways to install the kernel. Each, however, just copies the boot image to the devices boot/kernel patition device node. This was identified in an earlier section and for the SGH-i317 is /dev/block/mmcblk0p8. To do this manually, and assuming the device is already connected to the build machine, get a rooted adb shell and copy the following command making any required change to the device node. Make sure to be in a rooted adb shell first!
Code:
dd if=/sdcard/first_bootimg/boot.img of=/dev/block/mmcblk0p8 bs=4096
That's it - it's installed. We can unplug our device and reboot the new kernel into existance.
If a person is using a Samsung device that's supported by Mobile ODIN (developed by Chainfire), they can also use that program to install the boot image directory by simply selecting it as the kernel to flash. This actually tends to be a bit safer than using the 'dd' command, as there's no risk of typing the wrong device node filename. However, the boot image MUST be named "boot.img" for this to work.
When it is done rebooting, you'll be using your new kernel. As we haven't made any changes to the kernel at this point, the only way to tell the difference will be to go into Settings (on the device), and tap "About device" The line for the "Kernel Version" should now include the username you use on your linux build machine. In this particular case, there's another way to determine if we're using the new kernel: Try using the "soft" keys (the 'menu' and 'back' keys on either side of the home button.) They don't work.
Welcome to the Wonderful World of Kernel Development. There's a reason why those soft keys don't work, and we'll discuss that reason (and a resolution) in the next section. For now, we should just restore our kernel backup and wait for the next section to be posted.
After a fully functional kernel is restored, copy the following commands into a terminal window with the device connected to clean up the mess:
Code:
adb shell rm -r /sdcard/first_bootimg
cd ~/kernel-howto
rm -rf first_bootimg
Not sure if I have anything worth while, but ASUS left a "Tools" folder on my desktop...attached a screenshot of what it included.
phxtravis said:
Not sure if I have anything worth while, but ASUS left a "Tools" folder on my desktop...attached a screenshot of what it included.
Click to expand...
Click to collapse
Can you please zip that and upload them?
Yeah please zip it up and upload them so we can see what it's doing.
Yes, ZIP them please - the auditmode files are for sysprep audit mode, securebootdebug - are probably from microsoft HCK and add debug key to key storage (so you can run testsigned apps), FWVar - probably allows editing UEFI firmware variables (I've already made the same tool myself), everything is interesting of cause.
Here you go
Thank you for the tools.
SetAuditMode/ClearAuditMode - clears the sysprep audit mode (useless)
setup.cmd, SecureBootDebug* - installs "secure boot debug" policy. I.e. allows running of testsigned (or unsigned) apps. More info: http://msdn.microsoft.com/en-us/library/windows/hardware/hh998740.aspx
Securebootdebug needs the signed policy file. It is probably left on your device too, as "tools" directory is not erased. To obtain it - run CMD or powershell as administrator, then type there: "mountvol S: /s" without quotes. This would assign S: to your BCD partition. There should be SecureBootDebugPolicy.p7b file. Please share it too
To dismount disk S: after copying that file - type "mountvol s: /d"
reset.cmd - this file would delete the tools directory and all other files reverting things back.
hsc.vbs, pdq.vbs - tiny support scripts for reset.cmd
FWVar.exe - writes firmware variables. Not UEFI vars that are documented, but it plays with some other asus-specific vars like sensors calibration and platform IDs. Would be interesting to decompile it
mamaich said:
Thank you for the tools.
SetAuditMode/ClearAuditMode - clears the sysprep audit mode (useless)
setup.cmd, SecureBootDebug* - installs "secure boot debug" policy. I.e. allows running of testsigned (or unsigned) apps. More info: http://msdn.microsoft.com/en-us/library/windows/hardware/hh998740.aspx
Securebootdebug needs the signed policy file. It is probably left on your device too, as "tools" directory is not erased. To obtain it - run CMD or powershell as administrator, then type there: "mountvol S: /s" without quotes. This would assign S: to your BCD partition. There should be SecureBootDebugPolicy.p7b file. Please share it too
To dismount disk S: after copying that file - type "mountvol s: /d"
reset.cmd - this file would delete the tools directory and all other files reverting things back.
hsc.vbs, pdq.vbs - tiny support scripts for reset.cmd
FWVar.exe - writes firmware variables. Not UEFI vars that are documented, but it plays with some other asus-specific vars like sensors calibration and platform IDs. Would be interesting to decompile it
Click to expand...
Click to collapse
I was unsuccessful at finding the file, I wiped the tablet yesterday as it was running like crap(freezing, and wouldn't download updates), not sure if that wiped what you are looking for.
OK, I see what that setup.cmd file is doing. It generates an unsigned SecureBootDebug.p7b file authorizing full debug unlocking for the particular serial number of your device - the -u switch to createsecurebootpolicy.exe locks it to your device. It then sends that file to a server aptly named "\\secureboot" on ASUS's internal network and waits for a reply. Sometime later, a signed version of that file appears on that server's share, and the Vivo RT copies it to the EFI system partition to use with SecureBootDebug.efi.
After the service center does what it needs to do, they run reset.cmd, which deletes the Secure Boot policy from EFI NVRAM at next reboot and deletes SecureBootDebugPolicy.p7b from the EFI system partition.
phxtravis said:
I was unsuccessful at finding the file, I wiped the tablet yesterday as it was running like crap(freezing, and wouldn't download updates), not sure if that wiped what you are looking for.
Click to expand...
Click to collapse
If you had not done that, we could have made an image of your EFI system partition and recovered the deleted SecureBootDebugPolicy.p7b file, which would have permanently jailbroken your Vivo RT forever. Sadly, it was locked to your device. What method did you use to wipe it? Did you ask it to repartition your system?
Myriachan said:
OK, I see what that setup.cmd file is doing. It generates an unsigned SecureBootDebug.p7b file authorizing full debug unlocking for the particular serial number of your device - the -u switch to createsecurebootpolicy.exe locks it to your device. It then sends that file to a server aptly named "\\secureboot" on ASUS's internal network and waits for a reply. Sometime later, a signed version of that file appears on that server's share, and the Vivo RT copies it to the EFI system partition to use with SecureBootDebug.efi.
After the service center does what it needs to do, they run reset.cmd, which deletes the Secure Boot policy from EFI NVRAM at next reboot and deletes SecureBootDebugPolicy.p7b from the EFI system partition.
If you had not done that, we could have made an image of your EFI system partition and recovered the deleted SecureBootDebugPolicy.p7b file, which would have permanently jailbroken your Vivo RT forever. Sadly, it was locked to your device. What method did you use to wipe it? Did you ask it to repartition your system?
Click to expand...
Click to collapse
I did the "remove everything and reinstall windows" option in settings.
phxtravis said:
I did the "remove everything and reinstall windows" option in settings.
Click to expand...
Click to collapse
It *might* still be there then if we take an image of your EFI System Partition and search it manually with a hex editor. There are definitely no guarantees, though. Also, this would likely reveal your device's serial number to whoever you give the image to. It probably would *not* have any other information, though, like personal documents or your Windows RT CD key--those're on the main partition.
If this file were found, I think it would permanently jailbreak your device - Windows RT would let you run whatever unsigned code.
The SecureBootDebug.efi tool needed to use this .p7b file is also part of the publicly-available final 8.1 Windows Driver Kit. The one in your .zip file looks like the 8.0 version.
Myriachan said:
It *might* still be there then if we take an image of your EFI System Partition and search it manually with a hex editor. There are definitely no guarantees, though. Also, this would likely reveal your device's serial number to whoever you give the image to. It probably would *not* have any other information, though, like personal documents or your Windows RT CD key--those're on the main partition.
If this file were found, I think it would permanently jailbreak your device - Windows RT would let you run whatever unsigned code.
The SecureBootDebug.efi tool needed to use this .p7b file is also part of the publicly-available final 8.1 Windows Driver Kit. The one in your .zip file looks like the 8.0 version.
Click to expand...
Click to collapse
Not sure if I am sold on jailbreaking, what's the advantages other than being able to run "hacked" exe's? Aslo, what would you need me to do?
The EXEs are not "hacked" in any proper sense of the word, just recompiled. Sometimes some changes are needed, but they're usually basic. It lets you run (normal) Windows programs. .NET programs run as-is, native ones need to be ported (usually a pretty simple recompile, if they built under Visual Studio in the first place, but we need the source code), and it unlocks full Powershell scripting power. Programs written in other languages, like Python and Java, can be run using ported runtimes. In fact, it's even possible to run some x86 programs (unmodified Win32 native EXEs) via a dynamic recompilation layer written by @mamaich here on XDA; I can play some old games and run some nice old programs that I like that way.
phxtravis said:
Not sure if I am sold on jailbreaking, what's the advantages other than being able to run "hacked" exe's? Aslo, what would you need me to do?
Click to expand...
Click to collapse
Hacked EXEs?
No hacking. We legally take the source code for a program and compile it for win32 in THUMB_2 instead of win32 on x86. You can then run these desktop applications on your lovely ARM tablet as you could on a normal windows PC. That is an absolutely huge advantage which should have been a default option.
Quake alone makes it worth it.
I didn't mean to use "hacked" in a negative context, merely meant it as a general term of modifying original EXEs to run on ARM, I haven't been following the RT jailbreaking at all.
Can't modify an existing EXE. Totally incompatible (unless its a .NET application in which case no mods needed). Need to rebuild the EXE and any supporting libraries from source.
phxtravis said:
Here you go:
https://docs.google.com/file/d/0BzebTu1H3-aIbXlTV09BMjZsLVk/edit?usp=sharing
Click to expand...
Click to collapse
This requires approval. Does anyone still have a copy?
jordanmills said:
This requires approval. Does anyone still have a copy?
Click to expand...
Click to collapse
It doesn't require approval
Curiously, there is a Secure Boot debug policy creator ,signed by Microsoft for ARM but sadly not generating signed policies...
It may be used for jailbreak as the other tools are unsigned(most of them) if there is a bug allowing to load a unsigned policy somewhere(there should be one)
Will try downloading the HCK to see if there is something useful there
black_blob said:
It doesn't require approval
Click to expand...
Click to collapse
Hmm, it doesn't now. But there doesn't seem to be any way to download the whole thing. It only shows individual files.
jordanmills said:
Hmm, it doesn't now. But there doesn't seem to be any way to download the whole thing. It only shows individual files.
Click to expand...
Click to collapse
There is the download button at the top
I've seen a lot of people asking about this in various threads, so I decided to release a version modded to run under Android. First things first:
1. This is not a support thread. I'm not even providing compiled binaries. I am providing code that will compile under CyanogenMod and should work once compiled.
2. I will answer legitimate questions about compiling or use, but not generic "will this work on xyz phone". I expect posts to state what attempts you have already made and where you got stuck. All other requests will be mocked mercilessly.
MODS TAKE NOTE: I already have two suspensions for mocking n00bs.
What is CVE-2016-0728? It is an overflow bug in the Sysv IPC interface. You can read the technical write-up here http://perception-point.io/2016/01/14/analysis-and-exploitation-of-a-linux-kernel-vulnerability-cve-2016-0728/
What does it do for you? It allows a userspace program to grab a root token, and execute an arbitrary command. In the example code provided, it opens a root shell. This will provide temporary root on your phone.
Why a special version for Android? Two reasons: 1st, Android's Bionic lacks Keyutils needed to compile this successfully. You can cross-compile it using regular tools, but the provided code will compile under Android (using the NDK or CyanogenMod). 2nd, my makefile uses static linking to avoid problems with missing symbols. On my test device, a dynamically linked compiled version failed to find some libc symbols.
How do I patch my device to avoid the same problem: Here is the upstream fix: https://github.com/torvalds/linux/commit/23567fd052a9abb6d67fe8e7a9ccdd9800a540f2
How do I use it?
1. This exploit only works under kernel version 3.8 or higher on kernels with Sysv enabled. The execution may also be blocked by selinux. Google recently removed Sysv from the recommended Android kernel configs, and Linux has a patch for this problem, so expect devices to begin patching against this. Note this excludes the vast majority of older Android devices on kernel versions 3.4 and earlier.
2. I will assume you already know how to set up your build environment and sync CyanogenMod. Breakfast a device with a similar or the same processor and sync it. For example, my test device is arm64, so I breakfasted h815, LG G4, and then modified this line in the make file to say "generic" instead of cortex-a53. The resulting binary should work on any arm64 device.
3. Grab the exploit code (download zip) and place in your device folder (G4-common in our example).
4. "mka rootz"
5. The resulting binary is quite large (300 kb) due to static linking. You can pare it down by using dynamic linking if you think your device has compatible libc.
6. The exploit takes some time to run (6 hours plus on a phone). Due to it's nature it is reported to only have a 1 in 3 chance of success (you have to intentionally overflow the keyring while other processes are also using the keyring, so there is another chance a different process could cause it to overflow before you do).
7. Most modern Android devices do not let you execute arbitrary code from user accessible locations anymore. If you are attempting to root a device with this, you have to place the binary in a location that will let you execute it. This location or availability of it may vary by device. "/data/local/tmp" may be available. There are various programs that install command line tools in a location a non-root user can access them as well. You can use Better Terminal Emulator Pro for instance. It lets you open up a shell on your device, and it has a bin directory at "/data/data/com.magicandroidapps.bettertermpro/bin" you can copy the executable to. Make sure you chmod 777 the executable after copying it. Better Term Pro also comes with busybox, so you can potentially remount your system r/w after getting temporary root for more permanent root
8. Usage is something like "./rootz PP1" where PP1 is the name of the key you want to use. That's name should work for you.
9. Wait 6 hours and enjoy
Phones known to not be affected (I am only listing phones that meet the basic critieria, ie kernel version 3.8 and up. Also only stock firmwares. So your crappy Galaxy S III is not going on this list. And neither is your already rooted phone):
1. BlackBerry Priv (/proc/keys node missing)
2. LG G4 (/proc/keys node missing)
Phones known to be affected
1. NONE. (It appears very few devices ship with CONFIG_KEYS on Android OEM builds, although CyanogenMod has it turned on.)
If the /proc/keys is missing this exploit will fail, yes?
[email protected]:/data/local/tmp $ ./rootz PP1
uid=2000, euid=2000
msgget: Function not implemented
I do have /proc/keys kernel version 3.18 6.01 marshmellow
Guess I lack SysV.... I have /proc/keys in the filesystem, but the actual file is empty when I cat it.
I have /proc/sysvipc present on this device.
I don't think this exploit can actually work for android. It requires the adresses of symbols (like commit_creds). These adresses can only be taken from /proc/kallsyms but if you are not a root user the output will be 0's.
Also many devices luck SysV IPC. Though we can still replace msgget with sendmmsg or something equivalent and then bypass ASLR. I saw you have written a fucntion get_symbols but it actually return something like: 000000000.
rm
rm
symbols
First sorry for my bad writing!
I also see only 0x000... in /proc/kallsyms
Could we extract the adresses of symbols directly from a kernel image an then use that list instead /proc/kallsyms?
Cynob
Hello dears.
I have been working for 5g volte vowifi since i bought the phone.
Also You know we have carrier settings files in /product/etc/carriersettings I edited one of them and now I need to replace with original one to try.
but this /product fs cannot be touched cannot be mounted R/W. But We have a option from Magisk. it is overlayfs for all R/O filesystem acting like real one.
Question is How we do this /product fs as overlayfs?
Here one guide we have but I did not excaly get it because I didnot work about building magisk models before.
https://topjohnwu.github.io/Magisk/guides.html
Also I find a module but When I try this I am getting Unzip error. (I already tried to zip again without upper folder )
GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
Emulate read-write partition for read-only system partitions - GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
github.com
I worked with for ex vodafone_tr.pb file I get parameters from att5g_us.pb( It has lots of parameters for 5g volte vowifi) And I add these information to my carrier file. But I have to push this file to product.
Thank you
You need to create a Magisk module. Best is to download an existing module, unzip it, then modify it, the re-zip it.
Under the directory of the Magisk module, create a subdir called 'system', create a subdir 'product' underneath and 'etc/carriersettings´ subdirs as well. Put your file in there. Re-zip the module. Use adb to push to phone and install the module with the Magisk app. Reboot phone.
Magisk will mount your (modified) file in /product/etc/carriersettings/...
Check if the file is there:
Code:
> adb shell
$ su
# cd /product/etc/carriersettings/...
# ls
Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post
tauio111 said:
Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post
Click to expand...
Click to collapse
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.
furkanosman said:
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Click to expand...
Click to collapse
I just copy and paste the xml values every time, its persistent until you enter a different sim or update android.
I think it would be more viable to make a magisk script that adds the part you want to the front of the xml instead of replacing the bp files.
furkanosman said:
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.
Click to expand...
Click to collapse
Hi
Did u manage to work on the pb files in /product
I tried the cfg method but my ims registration isn't changing