Develop Bugs

relocate_apps.sh - When size matters [final] [2010-08-17]

I've made a script that relocates all your apps into an EXT2 filesystem in /sdcard/andboot/apps.img.
Advantages
- Get a LOT of space for apps even if everything else is running in NAND
- Never need to reinstall apps when upgrading (as long as the release has this script)
- Dual boot, for example, Eclair in NAND, Froyo on SD-card, but share the same apps.
- Wipe data (all settings) but keep all the apps stays (but not their settings)
Disadvantages
- Propably a bit slower than keeping everything in NAND
- If you intend to keep apps when when upgradring/switching releases then odex:ing your Market apps is not a good idea when using this.
Attached is an update that should work on any Android build (for our devices). The script in the update is only used once to create /sdcard/andboot/apps.img. Once it exist, the init script in the new initrd.gz (included in the attached NBH for NAND users) will automaticly mount /sdcard/andboot/apps.img to /data/app on boot no matter what Android build you are using.
Instructions
1a. Flash the attached kernel (but configure it first if needed with Loserskaters NBH editor) (NAND Users)
1b. Replace andboot\initrd.gz (HaRET users)
2. Apply attached update (adds relocate_apps.sh to /system/bin)
3. Do this in a terminal:
$su
#relocate_apps.sh 256 (for 256Mb storage, the choice is yours)
<wait until it finishes>
That's it. Upon the next boot your old apps in /data/app will be deleted to free up space.
To uninstall, delete the apps.img file on the SD-card, then wipe your data.
I've done some testing and it seems to work fine. I'm currently dual booting VaniljEclair (NAND) and Fresh Froyo (SD-card instal) and they share the same apps. Works fine for me but I cannot guarantee that it is without bugs.
Known limitations
- Won't work with apps that installs to /data/app-private (like some banking apps) so you have to reinstall them if you switch build or wipe your data.
- You can't remove the SD-card without powering off the phone since your apps are now stored on it.

kallt_kaffe said:
I've made a script that relocates all your apps into EXT2 filesystem in /sdcard/andboot/apps.img.
Advantages
- Get a LOT of space for apps even if everything else is running in NAND
- Never need to reinstall apps when upgrading (as long as the release has this script)
- Dual boot, for example, Eclair in NAND, Froyo on SD-card, but share the same apps.
- Wipe data (all settings) but keep all the apps stays (but not their settings)
Disadvantages
- Propably a bit slower than keeping everything in NAND
- If you intend to keep apps when when upgradring/switching releases then odex:ing your Market apps is not a good idea when using this.
Attached is an update for VaniljEclair RLS10b but it could tweaked into any release if you know what you are doing ("/system/bin/relocate_apps.sh boot" needs to be executed early, could be done either in a userinit.sh if the build has one or set up as a run-once service in sysinit.rc)
Instructions
$su
#relocate_apps.sh 256 (for 256Mb storage, the choice is yours)
<wait until it finishes>
That's it. The final version will delete your old apps on the next reboot to free up space in /data but this experimental version does not do that.
THIS IS AN EXPERIMENTAL SCRIPT. USE IT AT YOUR OWN RISK.
Click to expand...
Click to collapse
Interesting....so to get it work on fresh froyo I need to add this line /system/bin/relocate_apps.sh boot into userinit.sh?
Also when could we expect the version that deletes apps from system after moving them to sd?

-Arturo- said:
Interesting....so to get it work on fresh froyo I need to add this line /system/bin/relocate_apps.sh boot into userinit.sh?
Click to expand...
Click to collapse
Not anymore, the new version has that part in the initrd.gz so it will work in ANY build without modifications once you've relocated your apps.
Also when could we expect the version that deletes apps from system after moving them to sd?
Click to expand...
Click to collapse
Right now.
If this works as well as it seems to do I will push my changes to the bootenv and tinboot repo's so that all new kernels will have support for this. I will also include the relocate_apps.sh script in next VaniljEclair release and push it to Fresh Froyo.
While I have never needed more space on /data I still find this very useful as I now only need to reinstall one single app (my banking app) when I reinstall my phone (which I do quite often).
btw, I'm using a 512Mb image for /data/apps right now I don't think I will ever come close to fill it up.

kaffe, can we put apps (only apps, data still on NAND) on a second ext2 partition instead ext2 filesystem? I think it will be faster.

kallt_kaffe said:
Not anymore, the new version has that part in the initrd.gz so it will work in ANY build without modifications once you've relocated your apps.
Click to expand...
Click to collapse
Just a quick question, i know i could just try it and find out for myself but with the new kernel, does it mount /data/app normally if the apps.img does not exist??
EDIT::
Just realised that it must do this for it to work the first time but i'm guessing at the minute there is no way to revert back to apps on the /data partition without wiping and starting a fresh.

Hi..thanks for the app.
I follow direction unitl the app.img was mounted and saw the msg saying /data will be deleted after reboot, but after reboot the internal memory is still the same. Any idea?
ps. I am running your 2.1 built (on polaris) so I didn't follow your step1, so maybe that is why it didn't work?

Cookiekaikai said:
Hi..thanks for the app.
I follow direction unitl the app.img was mounted and saw the msg saying /data will be deleted after reboot, but after reboot the internal memory is still the same. Any idea?
ps. I am running your 2.1 built (on polaris) so I didn't follow your step1, so maybe that is why it didn't work?
Click to expand...
Click to collapse
Are you saying you didn't flash the kernel provided?? If so the apps.img on the sdcard will not be mounted. You need to modify the kernel NBH with the NBH editor to work with Polaris, flash it and try again. Sorry if i misunderstood and you have already done this.

Can the kernel file you attached be used for vogue at 320x240 res?
edit: actually, if it's not too much trouble...could you explain how to get this working on a vogue froyo nand 320x240? Thanks!

kallt_kaffe said:
Not anymore, the new version has that part in the initrd.gz so it will work in ANY build without modifications once you've relocated your apps.
Right now.
If this works as well as it seems to do I will push my changes to the bootenv and tinboot repo's so that all new kernels will have support for this. I will also include the relocate_apps.sh script in next VaniljEclair release and push it to Fresh Froyo.
While I have never needed more space on /data I still find this very useful as I now only need to reinstall one single app (my banking app) when I reinstall my phone (which I do quite often).
btw, I'm using a 512Mb image for /data/apps right now I don't think I will ever come close to fill it up.
Click to expand...
Click to collapse
Ok seems to be working fine for me with 512 image. Will report back in a few days.

scooter1556 said:
Are you saying you didn't flash the kernel provided?? If so the apps.img on the sdcard will not be mounted. You need to modify the kernel NBH with the NBH editor to work with Polaris, flash it and try again. Sorry if i misunderstood and you have already done this.
Click to expand...
Click to collapse
I didn't flash the kernel because the file name says FROYO but I am running eclair. However I am pretty sure the file was created and mounted as well.
I do know how to use NBH editor but just want to be sure that the kernel is safe even if I am running eclair.
Thanks for the reply
UPDATE: never mind I forgot you can select eclair as type. I will report back once I am done with the rest of the steps. THX!

Cookiekaikai said:
I didn't flash the kernel because the file name says FROYO but I am running eclair. However I am pretty sure the file was created and mounted as well.
I do know how to use NBH editor but just want to be sure that the kernel is safe even if I am running eclair.
Thanks for the reply
Click to expand...
Click to collapse
Yes it's safe.i think Kalt has made all the required changes to make kernel work on both froyo and eclair.
Kalt I'm wondering about something.I've already created the image for apps but I'm not sure what's going to happen when I instal new applications.will they automatically be moved to the image or do I have to run the script after ever installation off apk file?

tuannam said:
kaffe, can we put apps (only apps, data still on NAND) on a second ext2 partition instead ext2 filesystem? I think it will be faster.
Click to expand...
Click to collapse
It would propably be a little bit faster but it will be a lot more complicated to make and perhaps even more complicated to use. We allready have a de facto standard with this partitioning:
part 1 - FAT32
part 2 - system
part 3 - data
part 4 - swap
Now, not many uses swap on SD-card so we could propably say partition 4 should be used for this, right? But then we also would need to now that partition 4 really is ext2 and is having apps on it before we mount it. And it is also quite complicated for most people to partition their SD-cards in the first place.
So basicly, what I'm saying is that it propably could make small difference in performance but I'm not convinced it would be worth it. The way it works now is pretty safe as it won't do anything unless /sdcard/andboot/apps.img exist and setting it up is a simple one-liner in the terminal.
On the other hand an advanced user could set up their own apps on ext2-partition manually and add some lines to userinit.sh that mounts it.
So I don't know, at least I don't have any plans to add it at the moment. Perhaps someone could add it later if the advantages turns out to be big enough to make it worth it.
EDIT: Another aspect is that /data/app is only written to when installing, all other accesses should be only reads. All writing to /data is still in NAND so I really think the performance penalty is minimal. I haven't been able to tell the difference myself.

scooter1556 said:
Just a quick question, i know i could just try it and find out for myself but with the new kernel, does it mount /data/app normally if the apps.img does not exist??
EDIT::
Just realised that it must do this for it to work the first time but i'm guessing at the minute there is no way to revert back to apps on the /data partition without wiping and starting a fresh.
Click to expand...
Click to collapse
Your logic is correct.

Cookiekaikai said:
UPDATE: never mind I forgot you can select eclair as type. I will report back once I am done with the rest of the steps. THX!
Click to expand...
Click to collapse
If you are using the latest VaniljEclair then use a Froyo kernel. With the latest version of the libcamera.so the Froyo kernel can and should be used to get the extra 11Mb of memory.

-Arturo- said:
Yes it's safe.i think Kalt has made all the required changes to make kernel work on both froyo and eclair.
Kalt I'm wondering about something.I've already created the image for apps but I'm not sure what's going to happen when I instal new applications.will they automatically be moved to the image or do I have to run the script after ever installation off apk file?
Click to expand...
Click to collapse
As long as you have the new kernel and have run the script once then everything you install will end up i apps.img.

I must be doing something wrong. I flashed the new kernel, performed the update then ran the script(saw the /data/app copied screen and was told after reboot it would take effect, but internal memory is still going down after reboot).
UPDATE: after doing some PD I found the app.img disappeared from andboot. Not sure what is going on but i am recreating another one now.

Played around with this, and I think it's working on my vogue... ~60mb free on phone. Performance isn't too bad. Thanks!

kallt_kaffe said:
As long as you have the new kernel and have run the script once then everything you install will end up i apps.img.
Click to expand...
Click to collapse
Kalt I have a feeling that the script messes up compcache(it could also be the new cell location patch) My sd card is not recognized after restart of the system with compcache enabled. I remember that we had already suffered with this problem and you fixed it by adding 120 s delay to the script. I've notice that with apk image the system takes a bit more time to boot and load widgets in comparison to apks on nand. Could that be the problem? I would like to increase the delay in compcache script to maybe 180 seconds or so to see if it fixes the issue.
What should i change in this script to change the delay time?
Does this 2>&1 stands for 120 seconds?
Code:
#!/system/bin/sh
#
# Compcache manager
#
#
# Change these variables to tweak the compcache
#
opts= # Use default (25% of the avaliable RAM (~23-24Mb))
#opts='disksize_kb=16384' # 16Mb compcache
#opts='disksize_kb=32768' # 32Mb compcache
#opts='backing_swap=/dev/block/mmcblk0p4' # compcache (default size 15% of available RAM) + backing swap (on SD card)
#opts='backing_swap=/dev/block/mmcblk0p4 memlimit_kb=32768' # 32Mb compcache + backing swap (on SD card)
# Swappiness
swappy=30
case "$1" in
start)
sleep 120
insmod /system/lib/modules/lzo_compress.ko >/dev/null 2>&1
insmod /system/lib/modules/lzo_decompress.ko >/dev/null 2>&1
insmod /system/lib/modules/xvmalloc.ko >/dev/null 2>&1
insmod /system/lib/modules/ramzswap.ko $opts >/dev/null 2>&1
swapon /dev/block/ramzswap0
sysctl -w vm.swappiness=$swappy
;;
stop)
swapoff /dev/block/ramzswap0
;;
enable)
setprop persist.service.compcache 1
echo "Compcache will be enabled in 120 seconds."
;;
disable)
setprop persist.service.compcache 0
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
Thanks

-Arturo- said:
Kalt I have a feeling that the script messes up compcache(it could also be the new cell location patch) My sd card is not recognized after restart of the system with compcache enabled. I remember that we had already suffered with this problem and you fixed it by adding 120 s delay to the script. I've notice that with apk image the system takes a bit more time to boot and load widgets in comparison to apks on nand. Could that be the problem? I would like to increase the delay in compcache script to maybe 180 seconds or so to see if it fixes the issue.
What should i change in this script to change the delay time?
Does this 2>&1 stands for 120 seconds?
Code:
#!/system/bin/sh
#
# Compcache manager
#
#
# Change these variables to tweak the compcache
#
opts= # Use default (25% of the avaliable RAM (~23-24Mb))
#opts='disksize_kb=16384' # 16Mb compcache
#opts='disksize_kb=32768' # 32Mb compcache
#opts='backing_swap=/dev/block/mmcblk0p4' # compcache (default size 15% of available RAM) + backing swap (on SD card)
#opts='backing_swap=/dev/block/mmcblk0p4 memlimit_kb=32768' # 32Mb compcache + backing swap (on SD card)
# Swappiness
swappy=30
case "$1" in
start)
sleep 120
insmod /system/lib/modules/lzo_compress.ko >/dev/null 2>&1
insmod /system/lib/modules/lzo_decompress.ko >/dev/null 2>&1
insmod /system/lib/modules/xvmalloc.ko >/dev/null 2>&1
insmod /system/lib/modules/ramzswap.ko $opts >/dev/null 2>&1
swapon /dev/block/ramzswap0
sysctl -w vm.swappiness=$swappy
;;
stop)
swapoff /dev/block/ramzswap0
;;
enable)
setprop persist.service.compcache 1
echo "Compcache will be enabled in 120 seconds."
;;
disable)
setprop persist.service.compcache 0
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
Thanks
Click to expand...
Click to collapse
How about the line "sleep 120".

I've experienced some strange things with this so far. I installed a new app this morning. Some hours later I got a freeze and pin-reset my phone. The new app was gone... The .apk is no longer in apps.img, not even in lost+found. The system however believes it is still installed and I cannot reinstall it.
I have now added the "sync" flag when mounting the sdcard in /init and I'll give it another try. With a little luck it might prevent it from happening again.
I attached the new NBH and initrd.gz here if anyone else experienced the same problem and want to give it another try.

[DOC][DEV] More free space on /data for the Hero

Hey Devs,
As you may have noticed, I recently released a ROM with 276 MB free space on /data.
I accomplished this by using the MTD hack discovered by firerat and lbcoder.
It changes the kernel cmdline and includes addresses where the partitions are located.
This way, we are able to change the sizes.
That's exactly what I did.
The bad thing is that you need a recovery with a cmdline, which matches the ROM's boot.img's one!
So, a user needs to flash a recovery first and can then flash the ROM after a reboot into the new recovery.
Now, if the user wants to change his ROM to something else, he will need to revert the recovery.
It would be a lot easier if some other developers here would be interested in the MTD hack, so there is no need to revert.
I hope that some other developers will jump on the bandwagon now.
Here is the command I use for creating boot.img's, with modified partition sizes:
Code:
mkbootimg --kernel kernel --ramdisk ramdisk --cmdline 'mtdparts=msm_nand:[email protected](misc),[email protected](recovery),[email protected](boot),[email protected](system),[email protected](cache),[email protected](userdata)' -o boot.img --base 0x19200000
This reduces the /cache partition to 4 MB and makes /data as big as possible (the space which is left).
Now, the Market would normally fail to download huge APKs, because /cache is so small.
The problem can be easily solved by symlinking /cache to /data/cache and mounting the native cache partition on /dev/cache.
This can be done even without modifying the ramdisk, just do it in a script which runs before the Android frameworks start, like a2sd scripts.
Here are the necessary commands:
Code:
mount -o rw,remount /
umount /cache
rm -rf /cache
mkdir /data/cache
ln -s /data/cache /cache
mkdir /dev/cache
mount /dev/block/mtdblock4 /dev/cache
chmod 777 /data/cache
mount -o ro,remount /
Regarding the recovery:
Klothius from the Sapphire section created an update.zip which automatically patches the current recovery to use the provided cmdline.
You can find it here and include it in your instructions:
http://www.megaupload.com/?d=FY6CBAEE
I hope this little documentation will help you and will make you interested in doing this for your ROM.

Nice work. We'll look at implementing this into the next FroydVillain release. My only reservation would be keeping a wee bit more space on /cache than 4MB since our OTA app uses it. We could always hack it to use a dir on /data I suppose.

Hacre said:
Nice work. We'll look at implementing this into the next FroydVillain release. My only reservation would be keeping a wee bit more space on /cache than 4MB since our OTA app uses it. We could always hack it to use a dir on /data I suppose.
Click to expand...
Click to collapse
Yea, that would be better.. The whole purpose of this thread is to make things more compatible, means same cmdline Also you wll need to write the instructions for the recovery to /dev/cache then, keep that in mind
I will also add the RecoveryMod and an explanation in the documentation now.

I think the limitation of needing to switch recovery when changing ROMs is quite bothersome. Is there anyway we can get a recovery which supports both modes?

erasmux said:
I think the limitation of needing to switch recovery when changing ROMs is quite bothersome. Is there anyway we can get a recovery which supports both modes?
Click to expand...
Click to collapse
Sure, it can be done, but that would take a looot of work!

maxisma said:
Sure, it can be done, but that would take a looot of work!
Click to expand...
Click to collapse
More than I think it would be worth, to be honest. It'd make more sense for all the ROM developers to pull together on this and just make it a fact that we're giving more space to data, since it also eliminates the need to create fancy hacks moving dalvik to cache or to sdcard or splitting it between 2-3 file systems to cater for people who like to have the entire Android Market on their phone.
This has my/Villain's full support Maxisma, it makes perfect sense. Might I suggest you PM the other ROM developers here, or at least the popular ones like Fusion etc, and refer them to this thread so we can all work together on it.
Now that it's safe to assume that HTC have pretty much ditched the Hero, it makes sense for us to fix/improve -everything- including partition layout. There is a lot of wasted space in /cache that only gets used if a ROM developer starts fudging about with where dalvik lives, but /cache isn't quite big enough to completely migrate dalvik cache to for people who like to have a lot of apps.

Is there any more work on this? :d

shinyfong said:
Is there any more work on this? :d
Click to expand...
Click to collapse
I believe maxisma stopped work on it due to it breaking something or causing something not to work. I think he mentioned it in his Rom thread...

pulser_g2 said:
I believe maxisma stopped work on it due to it breaking something or causing something not to work. I think he mentioned it in his Rom thread...
Click to expand...
Click to collapse
It was me who broke it ;-) I accdidentally booted with a different mtd layout and thus killed all my data.
This method is perfectly fine though.
But I switched to a Nexus and I don't have a Hero anymore.

[HOWTO] Optimal ext4 mount options

Hi guys,
Now that several lagfixes are using the ext4 filesystem, perhaps we should look at optimizing ext4 for performance and lifespan of the flash memory.
One of the main things we can change is the journaling method to data=writeback. This should reduce writes and improve performance at a slight expense of reliability. Quote:
"In data=writeback mode, ext4 does not journal data at all. This mode provides a similar level of journaling as that of XFS, JFS, and ReiserFS in its default mode - metadata journaling. A crash+recovery can cause incorrect data to appear in files which were written shortly before the crash. This mode will typically provide the best ext4 performance."
One explanation here:
http://blog.smartlogicsolutions.com...ions-to-improve-ext4-file-system-performance/
Reference here:
http://git.kernel.org/?p=linux/kern...02ac5fa36d7f4c07856fe9cf89391e08986f7;hb=HEAD
HOW TO
1. Change the default mount option of the partition using tune2fs. You need to use a version of tune2fs that supports ext4, the one in busybox 1.17 does NOT. A working version is attached to this post.
- Push the tune2fs file to the phone's SD CARD:
adb push tune2fs /sdcard/
- Copy the tune2fs file to /data/
adb shell
su
cp /sdcard/tune2fs /data/
Now change the options:
/data/tune2fs -o journal_data_writeback /dev/block/mmcblk0p2
To verify:
/data/tune2fs -l /dev/block/mmcblk0p2
Look for the line that says:
Default mount options: journal_data_writeback
* Repeat the above for all other ext4 partitions.
It should take effect after a reboot. The next time it will be mounted automatically with data=writeback. You might have to do the tune2fs settings and reboot twice to get it to stick - I'm not sure why, but I had to.
You can verify this using the mount command:
# busybox mount | grep ext4
/dev/block/mmcblk0p2 on /data type ext4 (rw,noatime,barrier=0,nobh,data=writeback,noauto_da_alloc)
/dev/block/stl10 on /dbdata type ext4 (rw,noatime,barrier=0,nobh,data=writeback,noauto_da_alloc)
2. In addition to that, you can also edit the mount options to include the nobh option, which is a further minor optimization for data=writeback mode.
I personally use the options noatime,barrier=0,nobh,data=writeback. Voodoo already uses some of them like noatime and barrier=0.
I do it using a startup script (you need to know how to use/modify a startup script), with the following commands:
for k in $(busybox mount | grep ext4 | cut -d " " -f3)
do
sync
busybox mount -o remount,barrier=0,nobh $k
done
EDIT: Attached tune2fs that supports ext4. Works on Froyo kernels with ext4 support.

I posted the same request in another thread and i though it will be better if i do it here.
I found e2fsprogs-ext4.zip, but not sure if this is the correct zip.
it will be great if you can provide it here.
For others willing to try it, please change the /dev/block/mmcblk0pX to whatever you are using now.
my system is using /dev/block/mmcblk0p4

s88 said:
I posted the same request in another thread and i though it will be better if i do it here.
I found e2fsprogs-ext4.zip, but not sure if this is the correct zip.
it will be great if you can provide it here.
Click to expand...
Click to collapse
I've attached it here, it worked for me with the Universal Lagfix Froyo kernel that supports ext4.

hardcore said:
I've attached it here, it worked for me with the Universal Lagfix Froyo kernel that supports ext4.
Click to expand...
Click to collapse
And did you notice any improvements with new mount options?

busybox 1.17.1 says invalid option -o on tune2fs
==
i need the version attached to this thread, busybox version only supports ext2 and ext3

danzel said:
busybox 1.17.1 says invalid option -o on tune2fs
Click to expand...
Click to collapse
copy tune2fs into /system/xbin and chmod 755

Write back will cause data loses on crashes and it will increase read time on read miss since you need to write the block back to the main mem before reading a new block.

how & where do i edit mount options?

mdalacu said:
And did you notice any improvements with new mount options?
Click to expand...
Click to collapse
I didn't do any benchmarks, but in theory it should be better - reduced periodic journal writes (better battery life, flash lifespan and performance).

I don't know about performance but my SGS feels like a little bit snappier.
Thanks for this info.

chanw4 said:
Write back will cause data loses on crashes and it will increase read time on read miss since you need to write the block back to the main mem before reading a new block.
Click to expand...
Click to collapse
There is always a risk of slight data loss during crash, even with data=ordered. And we are already using some even more 'risky' options like barrier=0 and noauto_da_alloc anyway.
This setting increases the risk a bit more.

s88 said:
copy tune2fs into /system/xbin and chmod 755
Click to expand...
Click to collapse
He's referring to the busybox 1.17.1 version. It doesnt support the ext4 options. For the version attached here, you have to copy it to somewhere else like /data/ before u can execute it. You don't really need to copy it to /system/xbin.

hardcore said:
There is always a risk of slight data loss during crash, even with data=ordered. And we are already using some even more 'risky' options like barrier=0 and noauto_da_alloc anyway.
This setting increases the risk a bit more.
Click to expand...
Click to collapse
Yeah but I think at some point you're just throwing away the safety of EXT4, and might as well just use EXT2 since it writes faster even with these options on. (Can anybody confirm this? I did a check and EXT2 is still faster even with these mount options... while EXT4 reads faster. But writes are more important than reads on MoviNAND from my usability experience, since the reads are generally fast enough anyway.)

RyanZA said:
Yeah but I think at some point you're just throwing away the safety of EXT4, and might as well just use EXT2 since it writes faster even with these options on. (Can anybody confirm this? I did a check and EXT2 is still faster even with these mount options... while EXT4 reads faster. But writes are more important than reads on MoviNAND from my usability experience, since the reads are generally fast enough anyway.)
Click to expand...
Click to collapse
Hi Ryan, it's different. ext4 with data=writeback still uses journaling but only for metadata, which supposedly "provides a similar level of journaling as that of XFS, JFS, and ReiserFS".
So it's still safer than ext2 but without the speed (and more importantly, battery or flash write lifespan) impact full data+metadata journaling.
More detailed information in the sources here:
http://git.kernel.org/?p=linux/kern...02ac5fa36d7f4c07856fe9cf89391e08986f7;hb=HEAD

hardcore,
how do you change the mount options? like buffer from 1 to 0 and nobh, etc

s88 said:
hardcore,
how do you change the mount options? like buffer from 1 to 0 and nobh, etc
Click to expand...
Click to collapse
I've made a clearer procedure in the first post. To change some options you might need a startup script though.

hardcore said:
I didn't do any benchmarks, but in theory it should be better - reduced periodic journal writes (better battery life, flash lifespan and performance).
Click to expand...
Click to collapse
That sounds very promising
Followed the procedure - had to do it twice as well to make it stick...dont see any obvious changes in the last 5 minutes. Will update after using it for a while.
PS: how persistent is that setting? what circumstance does it get reset? When a new kernel is installed?

Does it work with 2.1 too?

bobbel said:
Does it work with 2.1 too?
Click to expand...
Click to collapse
yes, it does. I am using JM9, and it feel faster.
take note i only change the data to write_back, haven't figure out a way to do the rest.
for write back, it can be done on the terminal emulator.

s88 said:
yes, it does. I am using JM9, and it feel faster.
take note i only change the data to write_back, haven't figure out a way to do the rest.
for write back, it can be done on the terminal emulator.
Click to expand...
Click to collapse
Cool, I give it a try.
One Question:
When I enter the mount cmd I have:
/dev/block/mmcblk0p4 /data ext4 rw,noatime,...
So I have to use
/data/tune2fs -o journal_data_writeback /dev/block/mmcblk0p4 (instead mmcblk0p2)
Right?

Clemsyn's Kaiser Kernel Corner for 2.6.32 with EXT4 for DATA (SD) install

**** EXT4 for DATA (SD) install on second post *****
I've been slowly working on our Kaiser Kernel hoping to get extra life from it. Here are the changes I made so far.
1. Updated to the latest initrd and made a few changed in the init file
2. Changed the default I/O scheduler to noop.
3. Added block data integrity support and v4 support
4. Updated the latest yaffs2 filesupport up to changes dated 1/14/2011
5. Added PM module for Qualcomm 7500 chip
Will try to add BFQ I/O scheduler and see how the performance is compared to noop.
1/29/11 ** UPDATED KERNEL for better YAFFS2 best suited for Kaiser Samsung Memory chip **
Here is the nbh. This is type2 screen and keyboard1.
Will release my latest changes and update to git (need help on this one) when stable.
Please reinstall ROM and wipe data after flashing the kernel. The changes in yaffs2 code might cause issues. You can restore your data from BACKUP after installing the ROM.
************ TO FORCE CHECKPOINT IN DATA PARTITION PLEASE DO THIS IN TERMINAL *******************
su
echo 6 > /sys/module/yaffs/parameters/yaffs_auto_checkpoint
sync
This will change auto checkpointing to 2 and create a checkpoint.
This is just temporary, the checkpoint is deleted in reboot so the commands have to be done before every reboot.
One solution is to use Gscript.apk and type the commands in Gscripts. Execute via Gscript before reboot
**** To verify if you have checkpoint in Data Partition Please do in Terminal ******
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
scroll down and look for "blocks_in_checkpt" should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)

Update 2/7/2011
******************EXT4 FILESYSTEM FOR SD INSTALL***************
Because of bad blocks on my Kaiser, I have to give all NAND partition to install on system while installing data on SD with EXT4 filesystem. Here are the requirements:
1. An SD card with at least class 4
2. A way to partition and format your SD with FAT/ext2/Ext4 (ext2 should be as small as possible), I use Ubuntu for this.
3. Kernel with ext4 support (this will format your whole NAND partition to use system and use EXT4 on SD for Data)
Instructions:
1. FORMAT SD (I use Ubuntu DiskUtil) with FAT/ext2/EXT4, on my 8GB SD I use 7gb for FAT/8mb for ext2/1gb for EXT4 but it's your call. The most important part is that you only give the second partition (ext2) the least amount of size.
2. Download the KAISIMG.zip, extract and you will get a file KAISIMG.NBH
3. use Atools to edit accordingly with your Kaiser's screen type and keyboard (NBH is using screen 2 and keyboard 1)
4. Flash the KAISIMG.NBH
HERE IS WHERE IT GETS TRICKY!!! PLEASE FOLLOW THESE INSTRUCTIONS CAREFULLY!!!!
a. PRESS CENTER D PAD to enter ANDROID Installer
b. Choose (*) Sys on NAND
Choose (*) Data on SDCard Partition
c. Scroll Down to Install System and choose it
d. This will wipe your Existing system! Are you sure CHOOSE (YES)
e. Would you like to clear data too (Suggested) CHOOSE (NO) ***DO NOT CLEAR DATA OR THIS WILL CHANGE YOUR THIRD PARTITION TO EXT2 and will not work.
That's it. After installation reboot and you have an optimized ext4 on your SD for data. The mount options for ext4 in this kernel are:
data=writeback, barrier=0, nobh, noauto_da_alloc
Will use barrier=1 for better data security on next release, just enjoying the speed of barrier=0 for now

clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
Yes, Clemsyn If it's not a big problem please update 2.6.25 too. Thank you.

updates kernel 2.6.25
clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
This would be nice. Thank you

mt1976 said:
This would be nice. Thank you
Click to expand...
Click to collapse
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.

Finally got a checkpoint in my userdata partition...instructions posted on first post, no update on yaffs needed

clemsyn said:
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.
Click to expand...
Click to collapse
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol

aceoyame said:
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol
Click to expand...
Click to collapse
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).

clemsyn said:
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).
Click to expand...
Click to collapse
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.

aceoyame said:
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.
Click to expand...
Click to collapse
Agreed, that's what my wife thinks too I'll PM you my location
BTW, please make sure that the location of yaffs_auto_checkpoint is in /sys/module/yaffs/parameters (some ROM might have it on modules).
UPDATE:
Checkpointing is working great for me so far. Command on first post has to be executed in every reboot to create a checkpoint so typing the commands in first post before rebooting will create a checkpoint and save data partition (me thinks).
You can install Gscript and run the script before reboot
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
blocks_in_checkpt should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)
Can anyone help adding this to the shutdown script?

I've done 5 restarts, 2 battery pulls and 2 resets, made sure data partition was checkpointed before doing so....So far data partition has been OK with no FC's or errors, no lost apps with 18mb's left in data partition The thing that surprises me is that Phone Storage size hardly changes in size in every reboots so it's good news I have a feeling we finally nailed down this data corruption issue, but only time can tell since I will shift to prolong use before doing any restarts...
I have sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion

clemsyn said:
sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion
Click to expand...
Click to collapse
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!

scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.

scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
Thanks scooter, I'll try it out. I got stuck on a fake vsync and can't get out of it that I was unable to create a checkpoint and upon reboot without a checkpoint, everything was messed up. This is perfect, getting a checkpoint on startup and another in shutdown Thanks a lot.

Krazy-Killa said:
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.
Click to expand...
Click to collapse
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?

scooter1556 said:
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?
Click to expand...
Click to collapse
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.

Krazy-Killa said:
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.
Click to expand...
Click to collapse
lol, rookie error

OK, it works on boot up since I get a checkpoint 1 every boot not sure about shutdown but if scooter says it works, I bet you it works in shutdown too
Scoot, I'm going to try to do a git diff and post it here so all the changes I've made in the config will be there.

Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^

Krazy-Killa said:
Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^
Click to expand...
Click to collapse
That's great news and if my memory serves me right, you easily have data corruption in shutdowns and reboot. IMO, scooter's script is the best option we have for this issue and you don't have to remember to use gscript (especially from an old person like me with slight dementia Anyways, I'll continue on with my testing with this script. If it is still working after a few days, I'll try to add Namespace support, CIFFS, PPOE in our kernel.
BTW, here is my git diff and .config file. Please put it up in GIT. Thanks.

[Guide]Functional ext4 for external microSD with just a few bumps left

Y.G. said:
I formated my sd card to Ext4 and when insert it in to my phone, it says that's it's blank and has unsupported files. Any reasons for that?
Sent from my SPH-L710 using xda premium
Click to expand...
Click to collapse
The SPH-L710 Samsung Stock LJ7 TW 4.1.1 Android doesn't understand/support the change to ext4 external SD card (microSD) without a few things being done.
I'm working this out right now. So far I have manually been able to mount the newly created ext4 partition on the microSD card through adb, and after some chown/chmod I was able to go back to "Settings and Storage" and the "Mount SD Card" picked it up, and I was up and running ext4. But this didn't persist after a restart. So I'm looking into: /etc/vold.fstab MODS to keep it after restart right Now !!
If Some one else already has this perfected please chime in. I'm wanting to do most of the devices in the house this way when I get time because better performance, having a file system with a journal, and getting rid of thins like 4 Gig per file limitations is pretty Sweet in my humble opinion *Grin*
0) Assuming you already have your microSD card formatted ext4. I also happened to label mine extSdCard for the volume label within gparted
1) Can mount with:
mount -w -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard/
2) To get the correct owner and permissions run:
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
3) Should make the extSdCard owner/permissons match the regular internal sdcard you can verify this like so:
cd /storage/ && ls -l
drwxrwxr-x root sdcard_rw 2013-01-12 18:16 extSdCard
drwxrwxr-x root sdcard_rw 2013-01-12 17:05 sdcard0
4) After that you can go to the "Settings and Storage" to run "Mount SD" and you will have ext4 extSdCard Show up and it bring up the File System Status !! --> Until you reboot and it goes to crap because I don't have the vold.fstab edit/MOD complete _yet_ ... So, for now a boot script has been put in place to bring our external SD card back online during restart, so the system will acknowledges it, making the world a better place.
Example of how things look file system wise: mount | grep extSdCard
/dev/block/mmcblk1p1 /storage/extSdCard ext4 rw,relatime,user_xattr,barrier=1,data=ordered 0 0
5) Have not been able to resolve the vold.fstab to make this ext4 extSdCard matter fully Legit (in my opinion), but I did manage to make it remount the card on boot, so its online when the system comes up instead of having to manually mount it. Did this by -->
Added the following lines to the very bottom of: /etc/init.qcom.post_fs.sh
## sponix MOD to match with ktoonz kernel for better power management
stop mpdecision
## sponix MOD to mount extSdCard prior to GUI work around to make ext4 function
## read and write extSdCard mount
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
mount -w -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
## if you want read only extSdCard mount
## mount -r -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard
Still attempting to automate the process so the Stock+root LJ7 can pick up the extSdCard _normally_ without having to do the mount command manually, but so far its kicking my butt. Also this is more a "General, or Question and Answer type Topic" the Kernel(s) obviously support ext4 the system fs uses/requires it *Grin*.. So we might get Our Friendly Neighborhood Moderator to Migrate it to the proper place to help others. Just hoping to get the last few bumps smoothed out, or find someone that already documented the process that I've overlooked *Grin*..
Current Known Issues: If you unmount the card through the "Settings | Storage | Umount SD" or by hand with umount, you will either need to reboot for it to reattach through the /etc/init.qcom.post_fs.sh boot script script addition, or will have to mount it manually if you want to keep the system up and running. Guess you could also probably just run the /etc/init.qcom.post_fs.sh as root from a terminal emulator (or adb).
Still searching for vold.fstab bits of wisdom but that will have to continue next weekend -->
Sexy and You Know it,
Keep on Flashing,
sponix2ipfw (sponix
:fingers-crossed:

Ha! Sorry. Deleted: Didn't understand that you had it running on boot (can't read properly )

Great idea
Am I really the only one who also thinks this idea is the nuts?
Am I the only one who longs to transform the mess that passes for a filing system on the internal sd using symbolic links into a beautifully organized, encrypted and cloud synced system on my external sd?
Is it just me and a few others that want to be able to achieve the above so that we can move from one ROM to another or recover from a lost phone with the minimum of fuss?
Are we freaks? :cyclops:
Say it isn't so XDA!! :crying:
I'm gonna try this on my international S3 running Null_ Rom 25 JB 4.1.2
PS do you have any idea how the entire ExtSD or just a folder can be enrypted using Cryptonite and automatically mounted at boot time?

emp111 said:
PS do you have any idea how the entire ExtSD or just a folder can be enrypted using Cryptonite and automatically mounted at boot time?
Click to expand...
Click to collapse
Is this along the lines of what you're looking for?
http://forum.xda-developers.com/showthread.php?t=1141467
Also your idea is pretty insane, but also genius
If you get that to work please do come back and share

Insane ideas are the best lol
CNexus said:
Is this along the lines of what you're looking for?
http://forum.xda-developers.com/showthread.php?t=1141467
Also your idea is pretty insane, but also genius
If you get that to work please do come back and share
Click to expand...
Click to collapse
Thank you for your prompt reply, yes it is asking a lot I know but I think that it can be done.
Now if you really thought that idea was insane......check this out:
Imagine that we asked every Android app developer to submit the various paths used for their config (and config backup) files to a central database and had the ability to add or own custom paths (which could be added to the central database once approved).
We could build an script/app that would retrieve a list of currently installed apps on your phone then automatically build a symbolically linked file system (and/or backup file system) in the location of your choice that you could either encrypt and/or sync using your current tools or even incorporate this functionality into the app itself along with the ability to choose what was encrypted/backed up and how i.e. either synced to the Cloud or (S)FTP or SMB as either a dd copy or even a cwm flashable zip.
Could I dare hope for a Tasker module or the ability to add custom scripts?
I wish I could do this myself but my coding skills are non existent
Anyway the LUKS manager app won't automatically mount a file system, but I really like it anyways, thank you for pointing me to it!
And on the Ext4 front, the mount command (yes the 1st one ) failed, maybe the op could offer a suggestion. :angel:
BTW is there a place for people to suggest ideas for apps here?

Wait really it wont? I couldve sworn I remembering that it did
But dude....
You need to learn yourself some java and start whipping stuff up
Idk about the whole central database thing, but the rest could definitely be done with root access
I think the main problem with that is the proprietary aspects...i mean even here on XDA where binaries released are supposed to be GPL compliant, many of them arent and its sad because it deteriorates the overall quality of work thats released afterward

This whole thing is just hard work!
CNexus said:
Wait really it wont? I couldve sworn I remembering that it did
Click to expand...
Click to collapse
Doesn't seem to unfortunately
But dude....
You need to learn yourself some java and start whipping stuff up :D :D[/QUOTE said:
You make it sound soooo easy lol, and at another point in my life maybe it would have been but right now I'm operating at a reduced level due to some unforeseen circumstances that have left me lacking focus, motivation etc
You know all the things you need to be creative, learn etc lol
Anyway back to the matter at hand, I have got my ext4 SD card to the stage where I have to manually mount it from within the Settings/Storage as I'm using the international S3 and don't have the init.qcom.post_fs.sh, I think the qcom refers to Qualcomm chipset in US S3's.
As for modifying vold.fstab so we can avoid the above workaround it would seem that maybe thats a dead end as according to a German guy on android-hilfe. de, Vold may have been modified by Samsung to only deadl with exFAT on External SD's.
Looks like I'm not gonna be in Android nirvana for a while :crying:
Unless anyone else on XDA fancies getting in on this !!!!
Click to expand...
Click to collapse

Got it working ..... kinda
Got an app called ezymount (by ezynow) that automounts my ext4 64GB microSD at boot time.
I have to wait a few seconds for the boot process to complete but it's automatic, am pretty happy!!
Now gotta get symlinks, encryption and cloud synchronization sorted :/