Related
WARNING:
Use this on your own risk! I will not be held responsible for any failure that might be caused by using this.
"How should I know if it works? That's what beta testers are for. I only coded it."
- Linus Torvalds
What:
Yet another attempt at breaking free from RFS on the SGS phone (ie, lagfix), for Eclair, Froyo, and hopefully Gingerbread...
Which:
Supported devices are I9000 ATM. Theoretically it should work on all SGS variants, but since I only have I9000 to test on, I can't confirm this. I'd appreciate if other model owners will give it a try and report back.
Why:
KISS. I don't want to compile my own kernel or use a specific kernel (see voodoo) for my lagfix. I'd like to have a lagfix for ANY kernel out there.
z4mod is the 1st attempt in achieving that.
Another issue is standards. We don't need multiple lagfixes, all doing the same thing. One method, easy to customize and implement - thats the way to go.
How:
1. Use the online z4mod patcher to patch your kernel.
2. Download and apply one of the following update.zip files to convert your /data partition to the filesystem of your choice.
z4mod update.zip variants:
z4mod.ext2.update.zip - convert to ext2 - works on ALL kernels.
z4mod.ext3.update.zip - convert to ext3 - works only on modified kernel which supports ext3.
z4mod.ext4.update.zip - convert to ext4 - works only on modified kernel which supports ext4.
z4mod.auto.update.zip - automatically find which fs are supported by the kernel and convert to best suited - works on ALL kernels.
z4mod.rfs.update.zip - convert back to RFS - works on ALL kernels..
These update.zip files can be applied only to kernels which has been patched with z4build - a shell script which patches zImage to allow the init scripts additional non-RFS mount to each RFS partition, or with our online z4mod patcher. If the ROM you wish to install wasn't originally patched with z4build already, you can run z4build yourself on the zImage from the ROM package and put the patched zImage back to the ROM package for flashing, or use the online z4mod patcher to do it for you. It is that easy.
[NOTE:] Kernels (zImage) patched by z4build supports both mounts - RFS and non-RFS. So it doesn't change anything for users who doesn't apply z4mod.
You can also patch your z4mod scripts to enhance it's functionality, add more filesystems support, etc.
Although z4mod allows converting all the partitions from RFS to ext2/3/4, our update.zip files only convert /data - since this is ment for the 'masses'. If you feel brave enough, feel free to experiment with converting more partitions. All the sources are well documented.
Who:
z4ziggy, aka Elia Yehuda.
___
Original post [for reference]:
Although this is a half-baked project and far from complete, im introducing it to the xda community so i might get more ideas, and fix/implement more functionality, as the project advances.
z4mod is the 1st step in breaking free from RFS death grip - or at least tries to achieve this. The project adds ext3 support to a given zImage's init scripts, and creates update.zip for converting all RFS partitions and flash the patched zImage.
...thats the idea anyway.
ext4 support is planned for future official release. for testings purposes ext3 is sufficient.
sources and information can be found on my blog here: http://z4ziggy.wordpress.com
And of course, I'm not the only one interested in your measurements and conclusions
This looks really awesome, I'm loving it!
As far as the kernel patch goes, though:
Does the same kernel patch work for both 2.1 and 2.2 kernels? They seem pretty different.
Instead of requiring a linux box to patch the kernel, do you see any problems with doing the actual patching on the stock device itself? I didn't see anything in the script that should make this too difficult.
At any rate, this looks like The Solution.
love the work all 3 of you are doing with the file system on the i9000!
awesome job.
the patch is for the initscripts only - doesn't mind which kernel version. im doing my testings on Froyo 2.2 but patching should work the same on 2.1 scripts.
I hope to have some time later tonight to update the updater-script to follow all RFS partitions, and test it. ATM it works with my /cache partition only since its easy for testings (or "crashes" should i say? ).
i will update soon.
[edit]
regarding the linux part - no can do. We patch the initramfs scripts (which resides in the zImage), and i don't know of an easy way todo so from recovery mode. The easiest way is to use linux (even in a WM - you don't even need a full blown desktop, just CLI and compile tools) and patch the zImage, but i saw some Windows tools who might be able to do the same - I don't really know since i hardly use windows.
Thanks for your efforts! Keeping a close watch on this thread
z4ziggy said:
the patch is for the initscripts only - doesn't mind which kernel version. im doing my testings on Froyo 2.2 but patching should work the same on 2.1 scripts.
I hope to have some time later tonight to update the updater-script to follow all RFS partitions, and test it. ATM it works with my /cache partition only since its easy for testings (or "crashes" should i say? ).
i will update soon.
[edit]
regarding the linux part - no can do. We patch the initramfs scripts (which resides in the zImage), and i don't know of an easy way todo so from recovery mode. The easiest way is to use linux (even in a WM - you don't even need a full blown desktop, just CLI and compile tools) and patch the zImage, but i saw some Windows tools who might be able to do the same - I don't really know since i hardly use windows.
Click to expand...
Click to collapse
That's what rooting your phone is for! Just root the device, copy over all of the required binaries (arm statically linked if required, but busybox should handle 99% of what you're using in the script) and the kernel itself can be easily exacted using dd, of course.
Whole thing can be wrapped in a UI using java without too much hassle as well.
Most users aren't gonna be up to installing linux in a VM to run the script!
just replacing those scripts on your system won't do. you need to patch the initramfs and flash zImage back.
repackaging zImage with a new initramfs is not that simple. I'm using kernel_repacker todo so.
im sure there must be a better way than kernel_repacker, but until then, this is what we use
and about the Linux usage - the user would probably never use z4build script. only the update.zip. Modders, ROM creators, etc, should use z4build (or similar method to add the appropriate "mount ext3..." command to the "mount rfs", as i showed in my blog) to allow their kernel to support ext3. Simply apply z4build on any rom's zImage, and it's users can now apply the update.zip to have this rom' work with ext3.
I hope this explanation is suffice.
z4ziggy said:
just replacing those scripts on your system won't do. you need to patch the initramfs and flash zImage back.
repackaging zImage with a new initramfs is not that simple. I'm using kernel_repacker todo so.
im sure there must be a better way than kernel_repacker, but until then, this is what we use
and about the Linux usage - the user would probably never use z4build script. only the update.zip. Modders, ROM creators, etc, should use z4build (or similar method to add the appropriate "mount ext3..." command to the "mount rfs", as i showed in my blog) to allow their kernel to support ext3. Simply apply z4build on any rom's zImage, and it's users can now apply the update.zip to have this rom' work with ext3.
I hope this explanation is suffice.
Click to expand...
Click to collapse
Ah... kernel_repacker uses GCC. That would be pretty hard to port over to the SGS's stock kernel I guess.
Guess you're right then - would have to be run externally to the device. Bit of a shame since it adds to the complexity of the whole thing, still.
This does make it nice and quick to patch up any kernel at all though, which should make it much easier to keep the kernel patching up with any leaked kernels that come out!
looks like a very promising project! i liked voodoo but sadly we can not use it for 2.2 until we get the sources and it only converts the /data partition...
thanks for your work and i will be sure to check back here & on your blog soon
z4ziggy said:
Although this is a half-baked project and far from complete, im introducing it to the xda community so i might get more ideas, and fix/implement more functionality, as the project advances.
z4mod is the 1st step in breaking free from RFS death grip - or at least tries to achieve this. The project adds ext3 support to a given zImage's init scripts, and creates update.zip for converting all RFS partitions and flash the patched zImage.
...thats the idea anyway.
ext4 support is planned for future official release. for testings purposes ext3 is sufficient.
sources and information can be found on my blog here: http://z4ziggy.wordpress.com
Click to expand...
Click to collapse
Did you manage to compile ext3 or ext4 support to Froyo kernels? I only found ext2 support in them. (At least in JPK)
Another thing I see that you mount the rfs file on the device, but for me it only got mounted as vfat, so the symlinks in /bin in factoryfs.rfs were gone. (I solved this issue by adding commands to the initramfs that restore these symlinks)
ext3 is built into 2.2 kernel, i assumed the same for 2.1.
ext4 module will have to be rebuilt for each kernel, unless we build a standard kernel with ext4 support built-in, but all this will have to wait until samsung releases the sources.
[EDIT:]
Don't use the method which mounts the rfs files on the device. It uses ext2 and its obsolete already. I've only used the rfs mount on the device since it is the only way to have RFS readable as RFS and not as FAT. if your device mounted the RFS images as FAT, either:
1. you didnt use busybox mount
2. you might need to supply "-t rfs" to your mount command
z4ziggy said:
ext3 is built into 2.2 kernel, i assumed the same for 2.1.
ext4 module will have to be rebuilt for each kernel, unless we build a standard kernel with ext4 support built-in, but all this will have to wait until samsung releases the sources.
[EDIT:]
Don't use the method which mounts the rfs files on the device. It uses ext2 and its obsolete already. I've only used the rfs mount on the device since it is the only way to have RFS readable as RFS and not as FAT. if your device mounted the RFS images as FAT, either:
1. you didnt use busybox mount
2. you might need to supply "-t rfs" to your mount command
Click to expand...
Click to collapse
I know ext4 is missing, but I couldn't even get ext3 to work on JPK. (ext2 works)
for the mounts:
1. I did use busybox mount (using toolbox mount I couldn't even mount it in vfat mode)
2. I did supply -t rfs, but it told me invalid parameter
Some additions:
When I try to mount factoryfs.rfs in rfs mode it won't work (invalid argument). in vfat mode it works, but no symlinks and other rfs stuff. I can also mount it in j4fs mode, where I get some files not available in vfat mode (like param.blk or charging.jpg), the strange thing is these files are actually the same as in /mnt/.lfs
RyanZA said:
Most users aren't gonna be up to installing linux in a VM to run the script!
Click to expand...
Click to collapse
There's an easier workaround for that.
Provide instructions to:
- Copy required files to USB stick
- Download Ubuntu Live CD
- Boot from CD
- Run scripts
If you wanted to get fancy it could be prepackaged into an existing bootable CD distro with the scripts included (checking for updates when they're launched).
From what I've read, Desire owners used bootable CDs to root their phones, so it's obviously a viable option for the standard user.
Alright, that's the extent of my expertise. I'll leave you guys to sort out the real issues now!
I'm also very interested in this so subing myself to this thread.
I quite like the fact you Windows users have to use a Linux install to do it, I get a bit fed up having to find a Windows box somewhere to use Odin and Kies all the time.
RyanZA said:
Most users aren't gonna be up to installing linux in a VM to run the script!
Click to expand...
Click to collapse
Why? Surely the people wanting to do this kind of thing will be the more technical users anyway and us Linux users have to do the more difficult task of installing Windows in a VM to use the normal tools.
my ct-ng seems to be broken, my binaries doesn't execute on my SGS for some reason, so this will have to wait for tomorrow so I can investigate this further.
I need to compile the mkfs.ext3 staticly (which seems to compile fine using several different toolchains, but none works) so i can execute it from z4mod/updater-script.
dirk1978 said:
Why? Surely the people wanting to do this kind of thing will be the more technical users anyway and us Linux users have to do the more difficult task of installing Windows in a VM to use the normal tools.
Click to expand...
Click to collapse
Not true at all!
z4ziggy said:
my ct-ng seems to be broken, my binaries doesn't execute on my SGS for some reason, so this will have to wait for tomorrow so I can investigate this further.
I need to compile the mkfs.ext3 staticly (which seems to compile fine using several different toolchains, but none works) so i can execute it from z4mod/updater-script.
Click to expand...
Click to collapse
I've had so much trouble with statically compiling stuff...
We really need some kind of complete guide on all the ins and outs of statically compiling these system utilities. For a lot of them it doesnt seem to be all that straight forward...
i've a question
i've tried to mount the partitions as ext2/3 before by replacing the rfs in init.rc and recomplied the kernel and pack the zimage, but the system wont boot. it seems it wont accept any other fs other than rfs, rootfs, etc. my init.rc looks like this:
mount ext2 /dev/block/stl9 /system
so how did u simply mount them as ext3 by just changing the init.rc and recompile the kernel?
EDIT: sorry, i just tried it again and it's ok now, but not with the "||" (or) in the middle
Development continues here: http://forum.xda-developers.com/showthread.php?t=881837
Universal lagfix kernel project.
This project is aimed at multiple audiences.
If you think you're an end user read on.
For lagfix and kernel developers check the second post.
Download, changelog, bugs and other links can be found at the third post.
Credits
ChainFire for the original cf-root initramfs
hardcore for the speedup hacks
nikademus for the compiled jfs utils
neldar for BackLightNotification
newmail for some kernel modules
RyanZA for OCLF and for being helpful
supercurio for Voodoo
ykk_five for being helpful (and for the memory tweaks)
z4ziggy for z4mod
and for everyone else I might have missed. (like myself - I'm an egoist you know... and for all the people that helped the guys I credited above. And for the people who helped the guys that helped the guys credited above. And for XDA of course.)
If you only care about points
OhmygodIcanget2kpointsatquadranttoobadI'vealreadysoldSGSbecauseitonlyhas300MBraminsteadof512asadvertisedandboughtadesirebecauseilikehowitcrapsatmultitouch.
For the rest of the people
What is this about?
This project has multiple aims:
A JPM based kernel with ext4, jfs, tun and BLN support
An init script which supports multiple lagfix schemes
Init.d support (without early script support)
A modified ClockWorkMod that
Has better support for some features of the SGS-I9000
Includes support for rooting the device
Includes support for switching the lagfix schemes
Has some tweaking options
Various additions that might help the other lagfix/kernel developers
How does this work?
Simply flash the kernel to any JPM/JP6 based, unlagfixed ROM. By default it won't apply any lagfix scheme, so you might use it as a simple ROM, with built in ClockworkMod + rooting support. If you have a complete ROM package (with dbdata.rfs and an older csc file (it won't work with some of the newer csc files) ) you can also flash it with the other parts, just replace the zImage in the PDA.tar with the one supplied. The ROM was tested with the original JP6 and JPM ROMs, and with my Multinational 3.0 ROM.
How can I use the extra features?
Reboot your phone to recovery mode. Under "Advanced ULK", there are some new options:
Reboot to download: switches to download mode
Run 2e recovery: Switches to 2e recovery
Install superuser: installs busybox and superuser to the phone
Lagfix options: switch between various lagfix schemes
Tweak options: switch between various startup tweaks
BLN options: Turns BackLightNotification on or off
Reset permissions: Resets permissions to the same state as init does
What are the lagfix schemes?
They are ways on how to format and bind the various filesystems. You can chose how you want to format the partitions (data,dbdata and cache), whether you want to use an extra loop device or not, and whether you want to bind some directories from /data to /dbdata.
There are also 8 included pre-defined schemes:
OCLF: /data stays at rfs, but has an ext2 loop device on top of it
Voodoo: /data is ext4, all others are kept at rfs
JFS Voodoo: /data is jfs, all others are still rfs
No-RFS Standard: /data is rfs+ext2, all other partitions are ext4
No-RFS Advanced: all partitions are ext4
No-RFS Advanced JFS: all partitions are jfs
No-RFS Overkill: turns on all options: everything is ext4+ext2 and /data/data is bound to /dbdata
No-RFS Overkill JFS: turns on all options: everything is jfs+ext2 and /data/data is bound to /dbdata
(If using the overkill scheme using the bind option is dangerous if using too much apps, so you should disable it in the advanced menu.)
Although I used the names OCLF and Voodoo, the ROM is not compatible with any of them, it's just a naming convention I used. Hope RyanZA and supercurio won't mind
Which one should I use?
You decide. OCLF or Voodoo might seem simple but they are actually working great (but for those you can actually use the originals, they are probably much more stable), while NO-RFS Overkill might seem cool, it is clearly an overkill.
What happens after I chose a lagfix scheme?
If the chosen one is different from the active one the kernel will reformat your device the next start. Before reformatting it will create a nandroid backup, reformats the device then restores the data from your backup.
Is this ROM safe
alpha 0.2 means no. Switching between lagfix schemes might break, and data loss (mostly database corruption) can happen if you're using a lot of loop devices. And this kernel isn't thorougly tested anyway.
How to switch to another kernel/ROM
After disabling the lagfix everything should be fine (except for the possible data corruption). You can always flash another complete ROM package over this using Odin, you'll lose all your data however.
I've found a bug
And you'll find more of them. I'll try to collect them and answers to them at post #3
Internal workings.
In this post I'll try to describe how this works, and how it might be useful for other people.
The Kernel uses both the pre-init trick and the bootlogos trick to load up, and do it's stuff. In the pre-init phase it does the following:
initializes the devices
checks the actual state of the partitions, and decides which lagfix scheme is active
checks the config (resides in /system/etc/lagfix.conf), and if they differ runs the converter application
if they don't differ then it starts to mount the devices, and let's init run.
if during the mounting a fatal error occurs, it prints the log, waits for a couple of seconds and reboots the device.
It does the filesystem checks by mounting and checking whether mounting succeded or not. To defend against samsung's init's reformatting, it simply removes the fat.format command.
In the post-init phase it removes some files (mainly the symlinks to busybox, so they won't interfere with the busybox installed on the system). It also reinstalls fat.format, because it might be needed later.
The lagfixer is built inside clockworkmod, which has the following advantages:
Can use nandroid backup
Can use the framebuffer to show the progress (this makes it much easier to debug problems)
Besides the lagfixer there is also a "graphsh" application, which will run a specified command and prints its output to the framebuffer (the reasons for this are the same: it's much easier to debug this way than to check a log file copied to the sdcard). If a fatal error occurs this command is run so the end-user can see some logs. (all of this is still pre-init). The only problem with using the framebuffer is that core android won't boot up properly if the framebuffer was initialized beforehand (it switches to QVGA with overlapped screens... it's actually quite fun to try using android that way). That's why I always reboot after using either 'graphsh' or 'lagfixer', except for recovery modes, because CWM is using the same framebuffer anyway.
The initramfs is "ro.debuggable=1" so if anything happens one can connect to the phone in adb root mode, to check for errors. Adb is enabled by default in both CWM and base android mode. I don't know the drawbacks of using debuggable=1.
CWM was modified in order to be able to handle the various lagfix schemes. If it tries to mount, unmount or format /data,/dbdata or /cache, it asks for the lagfixer to do the actual mount, unmount or format. Therefore if one saves or restores a backup it will always see and use the actual files, and not the loop device file.
There were some other minor modifications, like binding the home button too to the enter key, using /mnt/sdcard instead of /sdcard, and the ability to back up the efs partition too. (/efs won't be restored however).
There is also a small modification in how the updates work. If there is an update, the init will always run samsung's 2e recovery, if not it will use CWM (you can use CWM's menu to run an update using CWM if desired). This is to maximize compatibility with updates.
About the compiled files:
I had a hard time getting the binaries to work, even when compiling in static mode. All of the binaries are either compiled by me using bionic (AOSP), or used from other sources (but they seem to be using bionic too). The main advantage is that the recovery binary is actually only around 700k, but has a working busybox implementation, a working recovery and some other small stuff. The main problem is with the other binaries (mainly with e2fsprogs). If I could cut from the size the whole initramfs might fit into an official kernel, so it could be used by anyone.
Links
Da kernel 0.3 + BLN 2.2: http://forum.xda-developers.com/showpost.php?p=9157498&postcount=738
Da kernel 0.3: http://android.sztupy.hu/dl/Universal_lagfix_kernel_JPM_0.3.tar
Da kernel 0.2: http://android.sztupy.hu/dl/Universal_lagfix_kernel_JPM_0.2.tar
Da kernel 0.1: http://android.sztupy.hu/dl/Universal_lagfix_kernel_JPM_0.1.tar
Source of the modified CWM: http://github.com/sztupy/android_bootable_recovery
Source of vendor/samsung/galaxys: http://github.com/sztupy/android_vendor_samsung_galaxys
Kernel sources I'm using: http://github.com/sztupy/universal_lagfix_kernel
Bionic port of jfsutils: http://github.com/sztupy/android_external_jfsutils
The contents of the initramfs: http://github.com/sztupy/universal_lagfix_kernel_initramfs
Changelog
Version 0.3
- before the lagfix gets applied there are now multiple options to chose from:
-- backup+restore (the original way)
-- backup, but don't restore (this eventualy results in a factory reset)
-- no backup (this also results in a factory reset)
-- go to recovery mode (to debug)
-- erase config file (to get back to the old options)
- the screen glitch in the first screen is now gone (because the kernel got smaller)
- jfsutils are ported to bionic (takes less space)
- there is a new option that will do the same restore permissions run, as the init script does
- I also added a few fixes to the CWM fix_permissions script, although I don't tested them yet (so it might brick the phone)
- Added two more rooting option:
-- the old one simply copyes busybox, superuser and su to the phone and adds symlinks to busybox
-- the new ones will also remove some old toolbox commands, so if you use the shell a lot, they won't interfere. The first mode only removes the base utils, like ls,mkdir,rm or ln
-- the second mode removes every toolbox command that has an equivalent busybox command (like mount, lsmod, rmmod, etc.)
- Adeed an option to get back to 2e recovery from CWM
Remarks
It seems that formatting dbdata as rfs only works, if you use the 512 PIT file (more exacty it only works if you have used the 512 PIT file the last time you've ticked re-partition in odin). This is probably due to the fact thet the 512 and 803 pit files reserve different space for the dbdata and for the system partition (the 803 pit file reserves 40 units less for the dbdata than the 512 pit file)
Version 0.2
- efs is not backed up automatically during lagfixing (but it's still backed up during ordinary backups)
- added jfs support and some more lagfix schemes using jfs
- added backlightnotification support, with the option to turn it on and off
- added startup script tweaks support, with options to turn them on and off
- removed ext2 support for the base mounts (might be put back into later)
- bind now only binds /data/data (should use less space)
- some minor bugfixes relating
Remarks
- If Updating from 0.1 to 0.2: If binds were turned on you have to turn them off before updating. After the update has finished you can turn them back on if desired.
Known bugs
After the lagfix conversion is done the phone reboots and the lagfix conversion starts over again
This means that the conversion took place, but the result is not the same as was desired. You should reboot into recovery mode and remove the config file.
After rebooting I have a lot of FC's - After switching lagfix schemes I have a lot of FC's
This is probably because of a database corruption. Try loading an earllier backup.
Can't download anything from market
Some configurations might have this problem, I'm still trying to find out why this happens.
Can't download some apps, like angry birds from the market
Retry, it should work after a while. If it still fails remove the external sd.
Shutdown fails on some configurations
Until I find a way to hook onto the shutdown script this might happen on more complex configurations (like the NO-RFS overkill)
Can't get back to rfs on /dbdata
Reverting the lagfix on /dbdata only works if you used the 512 pit file the last time you checked re-partition in odin.
I for one am excited
Dan
Can't wait to see this.
I am waiting too ++++
Lol 100% teasing ^^
gingerbread?
shahadat said:
gingerbread?
Click to expand...
Click to collapse
would be a bit early
Fantastic
Testing now on a very stripped JP6...
very interresting i think i will give it a try hehehe
any feedback?
Would it be unsavory to say that I got a little motion in my pants after reading that first post?
Nah, I think we all did
Sounds great! And I'll wait for a few more version points on that alpha before I try it... but I'm tempted!
wonderful,i'll try it
Sztupy is this kernel and method usable with your multilingual rom v3.0?
Thanks
kurudisease said:
Sztupy is this kernel and method usable with your multilingual rom v3.0?
Thanks
Click to expand...
Click to collapse
check first post. Tested with JPM, JP6 and Multinational 3.0
Test drive
I just used it, I had no issue. Everything worked fine and there is no lag at all, still too soon to say it's faster than voodoo though(which I was very happy with). All data is still there and no problem! Im using JP6 with NO-RFS Extended. Quadrant score is ~1900 for anyone that want to know .
looks great! gonna try this
ist it possible to add in that also??
Startup script speed tweaks
http://forum.xda-developers.com/showthread.php?t=813309
I applied the no-rfs advanced fix and everything worked fine. No issues so far.
Thanks!
**** EXT4 for DATA (SD) install on second post *****
I've been slowly working on our Kaiser Kernel hoping to get extra life from it. Here are the changes I made so far.
1. Updated to the latest initrd and made a few changed in the init file
2. Changed the default I/O scheduler to noop.
3. Added block data integrity support and v4 support
4. Updated the latest yaffs2 filesupport up to changes dated 1/14/2011
5. Added PM module for Qualcomm 7500 chip
Will try to add BFQ I/O scheduler and see how the performance is compared to noop.
1/29/11 ** UPDATED KERNEL for better YAFFS2 best suited for Kaiser Samsung Memory chip **
Here is the nbh. This is type2 screen and keyboard1.
Will release my latest changes and update to git (need help on this one) when stable.
Please reinstall ROM and wipe data after flashing the kernel. The changes in yaffs2 code might cause issues. You can restore your data from BACKUP after installing the ROM.
************ TO FORCE CHECKPOINT IN DATA PARTITION PLEASE DO THIS IN TERMINAL *******************
su
echo 6 > /sys/module/yaffs/parameters/yaffs_auto_checkpoint
sync
This will change auto checkpointing to 2 and create a checkpoint.
This is just temporary, the checkpoint is deleted in reboot so the commands have to be done before every reboot.
One solution is to use Gscript.apk and type the commands in Gscripts. Execute via Gscript before reboot
**** To verify if you have checkpoint in Data Partition Please do in Terminal ******
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
scroll down and look for "blocks_in_checkpt" should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)
Update 2/7/2011
******************EXT4 FILESYSTEM FOR SD INSTALL***************
Because of bad blocks on my Kaiser, I have to give all NAND partition to install on system while installing data on SD with EXT4 filesystem. Here are the requirements:
1. An SD card with at least class 4
2. A way to partition and format your SD with FAT/ext2/Ext4 (ext2 should be as small as possible), I use Ubuntu for this.
3. Kernel with ext4 support (this will format your whole NAND partition to use system and use EXT4 on SD for Data)
Instructions:
1. FORMAT SD (I use Ubuntu DiskUtil) with FAT/ext2/EXT4, on my 8GB SD I use 7gb for FAT/8mb for ext2/1gb for EXT4 but it's your call. The most important part is that you only give the second partition (ext2) the least amount of size.
2. Download the KAISIMG.zip, extract and you will get a file KAISIMG.NBH
3. use Atools to edit accordingly with your Kaiser's screen type and keyboard (NBH is using screen 2 and keyboard 1)
4. Flash the KAISIMG.NBH
HERE IS WHERE IT GETS TRICKY!!! PLEASE FOLLOW THESE INSTRUCTIONS CAREFULLY!!!!
a. PRESS CENTER D PAD to enter ANDROID Installer
b. Choose (*) Sys on NAND
Choose (*) Data on SDCard Partition
c. Scroll Down to Install System and choose it
d. This will wipe your Existing system! Are you sure CHOOSE (YES)
e. Would you like to clear data too (Suggested) CHOOSE (NO) ***DO NOT CLEAR DATA OR THIS WILL CHANGE YOUR THIRD PARTITION TO EXT2 and will not work.
That's it. After installation reboot and you have an optimized ext4 on your SD for data. The mount options for ext4 in this kernel are:
data=writeback, barrier=0, nobh, noauto_da_alloc
Will use barrier=1 for better data security on next release, just enjoying the speed of barrier=0 for now
clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
Yes, Clemsyn If it's not a big problem please update 2.6.25 too. Thank you.
updates kernel 2.6.25
clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
This would be nice. Thank you
mt1976 said:
This would be nice. Thank you
Click to expand...
Click to collapse
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.
Finally got a checkpoint in my userdata partition...instructions posted on first post, no update on yaffs needed
clemsyn said:
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.
Click to expand...
Click to collapse
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol
aceoyame said:
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol
Click to expand...
Click to collapse
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).
clemsyn said:
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).
Click to expand...
Click to collapse
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.
aceoyame said:
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.
Click to expand...
Click to collapse
Agreed, that's what my wife thinks too I'll PM you my location
BTW, please make sure that the location of yaffs_auto_checkpoint is in /sys/module/yaffs/parameters (some ROM might have it on modules).
UPDATE:
Checkpointing is working great for me so far. Command on first post has to be executed in every reboot to create a checkpoint so typing the commands in first post before rebooting will create a checkpoint and save data partition (me thinks).
You can install Gscript and run the script before reboot
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
blocks_in_checkpt should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)
Can anyone help adding this to the shutdown script?
I've done 5 restarts, 2 battery pulls and 2 resets, made sure data partition was checkpointed before doing so....So far data partition has been OK with no FC's or errors, no lost apps with 18mb's left in data partition The thing that surprises me is that Phone Storage size hardly changes in size in every reboots so it's good news I have a feeling we finally nailed down this data corruption issue, but only time can tell since I will shift to prolong use before doing any restarts...
I have sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion
clemsyn said:
sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion
Click to expand...
Click to collapse
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.
scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
Thanks scooter, I'll try it out. I got stuck on a fake vsync and can't get out of it that I was unable to create a checkpoint and upon reboot without a checkpoint, everything was messed up. This is perfect, getting a checkpoint on startup and another in shutdown Thanks a lot.
Krazy-Killa said:
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.
Click to expand...
Click to collapse
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?
scooter1556 said:
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?
Click to expand...
Click to collapse
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.
Krazy-Killa said:
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.
Click to expand...
Click to collapse
lol, rookie error
OK, it works on boot up since I get a checkpoint 1 every boot not sure about shutdown but if scooter says it works, I bet you it works in shutdown too
Scoot, I'm going to try to do a git diff and post it here so all the changes I've made in the config will be there.
Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^
Krazy-Killa said:
Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^
Click to expand...
Click to collapse
That's great news and if my memory serves me right, you easily have data corruption in shutdowns and reboot. IMO, scooter's script is the best option we have for this issue and you don't have to remember to use gscript (especially from an old person like me with slight dementia Anyways, I'll continue on with my testing with this script. If it is still working after a few days, I'll try to add Namespace support, CIFFS, PPOE in our kernel.
BTW, here is my git diff and .config file. Please put it up in GIT. Thanks.
I've tested this and it works. I only have a class 2 SDcard, so my testing shows it's really slow, a faster card would probably help.
With just this class 2 SD card, the boot-from-sd process is really only useful to see if the ROM being tested will straight-up brick your device or not.
Edit!
A class 6 card works fabulous! I'm posting from a tweaked version of mmarz's port of ath3nos' port of cm7 running from my brand-new class 6 card right now.
This process is hacked together from multiple other devices' howtos here on xda, sorry I don't know who to credit for the bits and parts. All of it is pretty generic, actually, and might well be applicable to other devices when tried as a whole.
For the moment, I'm only posting a brief skeleton how-to without specific walkthroughs for the steps, and I don't really want to upload many files until more of the bugs are swatted.
The following info should be enough for you, the savvy dev, to put the ROM of your choice on your SD card and boot it (or watch it fail to boot) without risk of bricking your device!
noobs, don't you dare, bricking is always a risk if you don't know your way around fastboot and adb!
Of course, savvy dev or not, a fresh nandroid backup is MANDATORY before hacking at your phone like this.
Standard disclaimer:
There is always a possibility of bricking your phone when messing with adb and fastboot.
If you're not willing to take that risk, don't try this at home (or at work, or school....)
Here goes:
The basic plan I followed was:
1) make and format 3 extra ext2 partitions after the default fat32 on the SD card, in this order: data (at least 180Mb) , system (170Mb to match stock), and cache (102Mb stock).
2) split up the boot.img from your chosen ROM so you can mod the ramdisk.
edit: turns out this next step in the quote is not required, there is an easier way.
A nifty command called devwait for init.rc
3) compile a modified init, adding a "pause (5);" after the ANDROID text. It goes on line 569 in the gingerbread init.c from a recent repo of google source.
use the newly compiled init in place of the init from the ramdisk.
since getting the android source uses so much time and bandwidth, I'm being nice and attaching a modified gingerbread init. If you test it with a froyo ROM and it doesn't work, don't blame me. If it does work with froyo ROMS, let me know!
Click to expand...
Click to collapse
3) change your init.rc from the ramdisk as follows:
replace the "mount yaffs2 [email protected] /system" , cache, and data lines with
Code:
devwait /dev/block/mmcblk0p3
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime ro
devwait /dev/block/mmcblk0p2
mount ext2 /dev/block/mmcblk0p2 /data nodev nosuid noatime nodiratime
devwait /dev/block/mmcblk0p4
mount ext2 /dev/block/mmcblk0p4 /cache nodev nosuid noatime nodiratime
be sure to comment out any other mounts which go to /system anywhere, like the "mount squashfs [email protected]/system/blahblah/blah.sqf /blah/blah" lines in the aospCmod init.rc
4) repack your boot.img, with the correct cmdline ("mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" works,) etc.
5) prepare the ext2 /system partition with your target ROM /system files.
How to load the ROM:
steps a) to f) done on your PC. step g) is done on the phone.
a) Unzip the ROM on your PC, to get at the files to copy to the new /system directory in adb, and allow you to modify the updater-script.
b) Mod that ./META-INF/com/google/android/updater-script as follows:
As an example, I removed the following from the aospCmodOV-5-16-11 updater-script:
everything except the "ui_print" "symlink" and "set_perm" lines, including the "unmount" line at the end of the script.
I thinned it down because the update-binary wants to write to MTD partition for /system, and I didn't want or need that to happen.
I did need it to install the symlinks and permissions, though.
Without those, the keyboards kept FCing, and the phone couldn't connect to the network.
c) rezip the ROM with the modified updater-script.
d) boot phone into recovery and connect to PC with cable.
e) adb push or otherwise copy the modified ROM.zip to your sdcard.
f) next, copy /system from the unzipped ROM to the new partition with adb
Code:
adb shell
mount -t ext2 /dev/block/mmcblk0p3 /system
exit
adb push /path/to/unzipped/ROM/system /system
g) on phone, in recovery: install .zip from sdcard
select the new ROM.zip you reassembled and pushed to the SD card
select yes, you really want to install the .zip
this should write the /system symlinks and permissions to the new /system partition.
Click to expand...
Click to collapse
6) back on your PC, still connected to phone with cable in recovery;
Code:
adb reboot bootloader
fastboot boot /path/to/boot.img
Step 6 is important!
this will boot from your modified boot.img without actually flashing it into the phone's NAND, so a reboot (or restart after battery pull in case of issues) just goes to your regular installed ROM in the phone.
That's about it for now. It's pretty much a hack as yet, but it's cool in a geeky way to be able to do this.
<reserved for future use>
Thanks! This should be very helpful to devs and those of us who are adventurous with our phones!
Very interesting concept, I'd love to see how many uses this could have!
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
bigsupersquid said:
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
Click to expand...
Click to collapse
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
I personally still have problems recompiling a ramdisk.
mmarz said:
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
Click to expand...
Click to collapse
I can post files one rom at a time, because the ramdisk and kernel will be a bit different for each rom, and so will the permission/symlink updater-script.
first, I'll have to get rom-dev permission, because they get modded, and also cause it's nice to have permission.
I personally still have problems recompiling a ramdisk.
Click to expand...
Click to collapse
fastboot provides a fantastic shortcut for that, if you cpio-gzip the ramdisk together after editing it to your specs
from http://android-dls.com/wiki/index.php
in a command line linux shell, from the directory containing your unpacked ramdisk,
Code:
find . | cpio -o -H newc | gzip > ../newramdisk.cpio.gz
will put newramdisk.cpio.gz in the directory above the one containing your ramdisk
Click to expand...
Click to collapse
fastboot can make the boot.img for you and either dump it into the phone (assuming your shell current directory contains your kernel (zImage) and ramdisk (newramdisk.cpio.gz))
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" flash:raw boot zImage newramdisk.cpio.gz
or boot your phone from the assembled image without actually flashing it in.
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" boot zImage newramdisk.cpio.gz
those fastboot lines assume an optimus v (or s.)
looking at all the great work you've posted on the forums, I'm surprised you have trouble handling anything to do with these phones!
I'm slowly working up to a modded recovery to install-to-sd directly and avoid all this hacking.
Nice, thanks for that. I had no idea fastboot could do that.
mmarz said:
Nice, thanks for that. I had no idea fastboot could do that.
Click to expand...
Click to collapse
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
bigsupersquid said:
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
Click to expand...
Click to collapse
Very cool i loved this idea when it was executed in a way over at Ubuntdroid for the intercept when i was doing alpha testing. Very convenient for many of the scarier tests when i wanted to keep my existing system intact. The big difference was that on the intercept the bootable partitions were written into the kernel. Downside you couldn't use it to test kernels but could switch kernels from the phone and flip-flop between completely isolated systems on the go without a computer. Which was great when i wanted to test a new ROM on the way to work but i needed my phone when i got there and had to have a painless method of switching to my usual running system. So i'd flash a "non SD" kernel and i was back up and running in just a minute or so.
I'm excited to play with this when i have some free time.
Thanks dude!
you could easily use an update.zip for each of the sd-boot and regular phone boot.img files and swap between working systems with those in recovery.
before getting the fast sd card, I was looking at this more as for testing foreign roms than two-system operation, but now I see that would work too.
for a second-system option I was looking at a market-enabled rom on sd since I don't use g-apps on my daily driver.
note: the GB rom I'm running on won't 'see' a card with more than four partitions properly, and won't mount the vfat partition with vold, although you can manually mount and access the first four partitions including the vfat. android just won't admit it's there. and I couldn't get at a fifth partition through the phone os, even though my pc could see it fine (to get 5 partitions I had to use an extended partition to contain most of the virtual partitions)
android could read virtual partitions in an extended partition just fine as long as the total number on the card didn't exceed four.
I learned a neat trick today.
going through the init.rc from the stock rom, I noticed a section at the beginning which was labeled 'on emmc' which would imply running from a card if the device hardware lacked nand flash memory.
I'd read on xda while researching this concept of boot-from-sd about an unspecified wait-for-device command in init.rc, but most of the info I've gotten on init scripting language has come from picking apart init scripts, as google has not documented the available commands very well.
long story short, I tried using devwait with the regular init and it worked just fine to mount the ext partitions. yay!
first post edited to reflect the improved method.
I've also been testing a recovery mod. if recovery.fstab is modified in the recovery ramdisk (or a fakeflash recovery) then the recovery will copy a tweaked rom.zip to the ext /system partition with some minimal modifications to the updater-script and a few commands in the adb shell.
the recovery with the modded recovery.fstab can also back up from and restore to the sdcard ext /data and /system partitions.
I will update the first post with more specific instructions as I hammer them into something consistent.
soon I'll be ready to release a modded fakeflash recovery to make experimentation easier, as well.
Let me preface this by saying this is my first ever real post in a forum, so if im doing something wrong, please tell me. I would post this in the linux on android forum, but I am apparently not allowed to do that. Hopefully this helps some people out.
I have been experimenting with the ubuntu.sh script and various linux images. After getting an image to work I attempted to expand the image file by using dd. This would create a 12 gb (or whatever size) image, but on copying the file system from kali.img to the new .img, the new image failed to execute and ended up wiping my kali.img, my modified ubuntu.sh script and my new .img (my fault for rebooting with the images mounted). This and subsequent attempts all failed to create a 12 gb image that would run kali.
Before I go further, I should state that I'm mounting my images to /storage/extSdCard/mnt
So all this got me thinking on other ways to expand the file and I thought why not move ./usr to a folder on extSdCard and use mount to bind it to the kali.img file. That way you'd get about 2gb extra on the image and installed programs could be installed to space on the extSdCard. If this works it could be applied to the entire linux filesystem, and just use a small .img to mount everything to when you wanted to start it.
That being said I'm having difficulty implementing this. Kali will boot and I can execute commands from it after running:
busybox mount -o bind ../kali/usr/usr ./usr
And
busybox mount -o remount,exec ./usr
It would appear that things work, but I can't connect through vnc as vncserver isn't installed and I can't install things because of an error with dpkg: error setting ownership of ... Operation not permitted.
That is where I'm at right now. I figured I would make a post and hopefully get some more input on this, or maybe help someone out with the whole image file size thing.
Sent from my SM-N900V using xda app-developers app
I have made some progress, but this idea as a whole would appear to be impractical at this moment. The permission errors occur because of the exfat file system used on android devices. This implementation does work on ext(4) formatted sdcards, as tested by my linux box running backtrack. However, the android system, at least for the note3 does not easily support ext4 formatted anything, despite /system maybe being ext4. I haven't looked too far into that as it is read only and would also be a hassle to configure. This may become an effective means of running linux on android systems once ext4 is widely supported, but not likely until then.
If I were more skilled I would try to set up a raid system with ext4 image files, since exfat has a 4gb file size limit. Unfortunately though, I do not have the skill to do this at the moment nor the time to take on such a task.
I wish you all the best of luck with your projects.
Sent from my SM-N900V using xda app-developers app