How to maximize NAND /data size - Tilt, TyTN II, MDA Vario III Android Development

All Kaiser owners complain at one time or another about size limitations to the /data partition.
I had asked around about increasing the /data partition to a maximum, keeping the /system partition to a bare minimum.
Well, I figured out a way to maximize your /data partition and still give the /system partition some "wiggle room".
The size of your "androidinstall.tar" file is the key here.
Depending on how it is constructed, you can easily figure out how much room you'll need for the /system partition in one of two ways.
1. Build your kernel with ATools as usual. Give it "normal" settings for now because you'll be changing and installing the kernel once more. Install your build and let it finish. Once it does, open up "Terminal Emulator", usually found as part of the "Dev Tools" app.
In Terminal Emulator you'll need to type "df -h" without the quotes. This detects the filesystem partitions and gives it a "human" number, hence the argument "-h".
You are looking for "dev/block/mtdblock2" and the numbers following.
If you are using the "system_froyo_us_odex" file from SF, then your mtdblock2 "should" read 90.7MB. Other builds are larger or smaller, this is why we're looking at mtdblock2.
You will also notice mtdblock3... this is your /data partition.
Once you have your numbers in MB, you can go back and build your kernel based on the size of your /system partition as installed on the phone, releasing as much room as possible for /data.
2. Going back to the "androidinstall.tar" file, most recently Devs have been combining the "app" folder inside the system folder. In the past, the data folder was separate from the system folder and gave the install a "bloated" size, appearing larger than what actually ends up in the mtdblock2 partition.
Most of these builds are built this way, with a few exceptions.
"Packed" size of "system_froyo_us_odex" comes out to around 87MB as a .tar file, relatively close, but you can't be sure unless the file is unpacked to a new folder.
When I unpack mine, it comes out to 90.7MB.
See where this is going?
I go to ATools and build a kernel based on the actual size of the /system file, plus 1 or 2%. (wiggle room)
I build the kernel with 92MB /system and 10MB for /cache, giving me 112MB for /data.
As long and drawn out as it may seem, I strongly suggest option 1. This will give you the best judge of size vs. kernel build size and is a more accurate measurement.

I have gotten it smaller than that once I stripped KingShui's rom to the bare essentials, around 52mb.
I'm just trying to share my knowledge is all.

Related

[initrd][DL Data2SD/Ext4+ V1.5] Enable your build with fast ext4/data.img support

This is my development based on Takaaki's initrd, during I develop my build at http://forum.xda-developers.com/showthread.php?t=913597, I spent a lot of time to enhance the data2sd and ext4, after studying almost all the popular methods: App2SD, Data2SD, Data2ext, Data2whatever, I develop this:
Features:
1. Support both data 2 data.img or ext4.
2. Support SquareFS.
3. Auto-format the 2nd partition with block size of 4096.
4. Utilize the Nand data partition to improve the performance.
5. Switch the PPP/RMNET.
6. Config the gps_zone
7. Support the update without re-flash. (you can fully customize your build based on this feature)
8. Backup the ext4 to data.img (implement in future)
9. Restore the data.img to ext4 (implement in future).
To enable external data.img
1. create '.data' folde under sdcard (/sdcard/.data).
2. create 'config.txt' under '.data' directory.
3. copy old data.img to '.data' directory, otherwise a new data.img will be created.
4. boot into Android.
Config.txt for data2SD.
data_img=1
data_ext=0
sd_boost=1
nand_data=0
Click to expand...
Click to collapse
To enable ext4
1. create '.data' folde under sdcard (/sdcard/.data).
2. create 'config.txt' under '.data' directory.
3. Config.txt for data2SD.
data_img=0
data_ext=1
sd_boost=0
nand_data=0
format_ext4=0
Click to expand...
Click to collapse
For first installation, it is always good to set the format_ext4=1 to format your ext4 partition with proper block size, after 1st boot, change it back to 0.
To config PPP/RMNET and GPS_ZONE
ppp_mode=1 will enable the ppp, set to 0 will use rmnet.
gps_zone='your country code', - country code, us, fr, jp, etc
To update/customize your build
1. Create a folder /DLZUPDATE under your sdcard.
2. put your app's apk under AndroidApps in /DLZUPDATE, that apps will be automatically installed into your data partition.
3. put your kernel file or other under root in /DLZUPDATE, all the files under root will be copied to your build, if your want to upgrade your kernel, just put zImage in /DLZUPDATE/root/system, and module files under /DLZUPDATE/root/system/lib/modules. etc.
Download
View attachment 492146
Install
1. extract the zip to a folder on your pc
2. backup your existing initrd.gz using adb
3. In "command prompt" env, run install.
4. your existing initrd.gz will be backup at the same folder named backup.initrd.gz.
5. if your have problem to boot into the system, just try: adb push backup.initrd.gz /system/initrd.gz. (even system can not be booted into the launcher or sense, in some point of time, the adb still can function, so just keep trying.)
Caution: if your build current use app2sd, you have to disable it before using this, rename the init.local.rc under etc to init.local.rc.bak, remove the 00start, 04apps2sd from /system/etc/init.d.
If your build uses boot partition, you should change the installation script to push the initrd.gz to replace the existing one wherever it is.
Thanks to
- ownhere (he is the expert of the ext4 or other file systems, and developed the data2ext.)
- melethron (the author of data2whatever)
- Takaaki (the author of data2sd)
- Cedesmith (He is guru of initrd )
Donation
I do not work for money, but it is always nice to know people really like my work and encourage me carrying on.
Please could you provide also information in case that NAND rom already use EXT4 for data2ext4 (like AF403 DesireHD stock based 1.0a)?
Good stuff.. thank you!
I am trying your NAND ROM now....
Good knowledge ... keep up cool stuff
i already use ur nand rom
If you don't want to look into the code to find out the difference, then, how about just try 1st.
totalcmdext said:
Please could you provide also information in case that NAND rom already use EXT4 for data2ext4 (like AF403 DesireHD stock based 1.0a)?
Click to expand...
Click to collapse
Sent from my DL DesireZ v2.5 using XDA App
Ok, i've managed to understand the concept of data2ext, and when on initial setup everything gets what i expect, i've 1 GB (depends on ext partition size) of free memory space, but i get an horrible performance even if formated with your tool, i guess it's SD performance/quality because i can't run any ext build published here where most of users claim to get good performance. So i try data2sd but instead of the 512MB (img size created with this) of memory i get only the 80MB of the free nand internal ram. What is stored on this img file ? only settings ? On 20 applications i use only 4 are movable to sd, so after my initial setup (and of course with good performance) i get a lot of memory low warnings.
Thanks
Looks like the data2sd was not enforced, what's your configuration in config.txt?
jpl69 said:
Ok, i've managed to understand the concept of data2ext, and when on initial setup everything gets what i expect, i've 1 GB (depends on ext partition size) of free memory space, but i get an horrible performance even if formated with your tool, i guess it's SD performance/quality because i can't run any ext build published here where most of users claim to get good performance. So i try data2sd but instead of the 512MB (img size created with this) of memory i get only the 80MB of the free nand internal ram. What is stored on this img file ? only settings ? On 20 applications i use only 4 are movable to sd, so after my initial setup (and of course with good performance) i get a lot of memory low warnings.
Thanks
Click to expand...
Click to collapse
Sent from my DL DesireZ v2.5 using XDA App
This is good and detail steps http://forum.xda-developers.com/showpost.php?p=10666187&postcount=341 on how to config, better than mine
Sent from my DL DesireZ v2.5 using XDA App
If i understand this, this is only change existing inird with modified initrd?
I'm sorry but with all this flash/unflash install/uninstall download configure i've lost my config, but i'm sure data_img=1 and data_ext=0
(just a note gps_zone=pt should configure gps to be used here in Portugal right ?)
If you say that it should work i'm going further with my tests and post here results because i don't want to flame this thread.
Update:
I've a stock 8GB class 4 card swaped with 4GB class 2, results on nand/ext4:
Initial setup (with format ext4) 1/2 time, second boot 1 minute and everything runing muuuch smother and faster so or i've got a malfunction card (?) (and i've read someone complaining about a 16GB class 10) or lowest class gets more speed (?)
Update2:
The diference is really amazing, i've got a completly functional android now with this card.
You should mention on your threads that if you get bad results on initial setup forget about that nand, return to SD builds or get other card.
Can you please post your card type please ?
how long does it take to format the ext4?
Hi,
I am trying to setup this new initrd, i put the format ext4 parameter in the config file and start my HD2.
wondering how long will it take to format my 1G ext4 partition?
thanks!
data.img Question
3. copy old data.img to '.data' directory, otherwise a new data.img will be created. ... Can I somehow set how big will the data.img be ?
Default will be 512M, you can config it by data_size.
BoneMo said:
3. copy old data.img to '.data' directory, otherwise a new data.img will be created. ... Can I somehow set how big will the data.img be ?
Click to expand...
Click to collapse
Sent from my DL DesireZ v2.5 using XDA App
Looks promising
This new initrd and file management seems well thought through and very promising.
In other builds I experienced that there can be a huge performance difference between the different file management choices (full NAND > adsd > ads2+ > data2ext > data2sd). It's good to see that you got into this subject and tried to find an optimal solution which can be configured by other chefs.
I assume that you implemented all this in your own DL DesireZ v2.5 build? If I find time I'm certainly gonna give it a try!
Keep up the good work!
works with AF403 DesireHD stock based 1.0a
there are a lot of discussions on my nand build about mixing using all the parameters, some got the performance improved a lot, some got things worse, your can refer to that to config your build for the best performance.
Dear, i got this error message...
D:\a>install
back your initrd...
remote object '/system/initrd' does not exist
replace your initrd...
1442 KB/s (0 bytes in 1523605.001s)
failed to copy 'mkfs.ext4' to '/system/xbin/mkfs.ext4': No space left on device
1263 KB/s (0 bytes in 303212.000s)
1410 KB/s (0 bytes in 428664.000s)
Unable to chmod /system/xbin/mkfs.ext4: No such file or directory
Unable to chmod /system/xbin/e2fsck: No such file or directory
Press any key to continue . . .
please help... I am using Gauner1986 build
Looks like the system in Gauner1986 build is read only or have the boot partition, or really occupied all the space? anyway, your existing build should be unchanged.
you can try adb shell ls /system, or adb shell ls /boot, whether there is an initrd.gz under it, then, manually push all the necessary files.
rasumia said:
Dear, i got this error message...
D:\a>install
back your initrd...
remote object '/system/initrd' does not exist
replace your initrd...
1442 KB/s (0 bytes in 1523605.001s)
failed to copy 'mkfs.ext4' to '/system/xbin/mkfs.ext4': No space left on device
1263 KB/s (0 bytes in 303212.000s)
1410 KB/s (0 bytes in 428664.000s)
Unable to chmod /system/xbin/mkfs.ext4: No such file or directory
Unable to chmod /system/xbin/e2fsck: No such file or directory
Press any key to continue . . .
please help... I am using Gauner1986 build
Click to expand...
Click to collapse
dandiest said:
Default will be 512M, you can config it by data_size.
Sent from my DL DesireZ v2.5 using XDA App
Click to expand...
Click to collapse
So you mean data_size= ... (512,1024,2048)
This is in cofig.txt?
jan-willem3 said:
So you mean data_size= ... (512,1024,2048)
This is in cofig.txt?
Click to expand...
Click to collapse
Yes, but please don't use 2048.

[DEV][Proof-of-Concept] Boot from SD on Optimus V

I've tested this and it works. I only have a class 2 SDcard, so my testing shows it's really slow, a faster card would probably help.
With just this class 2 SD card, the boot-from-sd process is really only useful to see if the ROM being tested will straight-up brick your device or not.
Edit!
A class 6 card works fabulous! I'm posting from a tweaked version of mmarz's port of ath3nos' port of cm7 running from my brand-new class 6 card right now.
This process is hacked together from multiple other devices' howtos here on xda, sorry I don't know who to credit for the bits and parts. All of it is pretty generic, actually, and might well be applicable to other devices when tried as a whole.
For the moment, I'm only posting a brief skeleton how-to without specific walkthroughs for the steps, and I don't really want to upload many files until more of the bugs are swatted.
The following info should be enough for you, the savvy dev, to put the ROM of your choice on your SD card and boot it (or watch it fail to boot) without risk of bricking your device!
noobs, don't you dare, bricking is always a risk if you don't know your way around fastboot and adb!
Of course, savvy dev or not, a fresh nandroid backup is MANDATORY before hacking at your phone like this.
Standard disclaimer:
There is always a possibility of bricking your phone when messing with adb and fastboot.
If you're not willing to take that risk, don't try this at home (or at work, or school....)
Here goes:
The basic plan I followed was:
1) make and format 3 extra ext2 partitions after the default fat32 on the SD card, in this order: data (at least 180Mb) , system (170Mb to match stock), and cache (102Mb stock).
2) split up the boot.img from your chosen ROM so you can mod the ramdisk.
edit: turns out this next step in the quote is not required, there is an easier way.
A nifty command called devwait for init.rc
3) compile a modified init, adding a "pause (5);" after the ANDROID text. It goes on line 569 in the gingerbread init.c from a recent repo of google source.
use the newly compiled init in place of the init from the ramdisk.
since getting the android source uses so much time and bandwidth, I'm being nice and attaching a modified gingerbread init. If you test it with a froyo ROM and it doesn't work, don't blame me. If it does work with froyo ROMS, let me know!
Click to expand...
Click to collapse
3) change your init.rc from the ramdisk as follows:
replace the "mount yaffs2 [email protected] /system" , cache, and data lines with
Code:
devwait /dev/block/mmcblk0p3
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime ro
devwait /dev/block/mmcblk0p2
mount ext2 /dev/block/mmcblk0p2 /data nodev nosuid noatime nodiratime
devwait /dev/block/mmcblk0p4
mount ext2 /dev/block/mmcblk0p4 /cache nodev nosuid noatime nodiratime
be sure to comment out any other mounts which go to /system anywhere, like the "mount squashfs [email protected]/system/blahblah/blah.sqf /blah/blah" lines in the aospCmod init.rc
4) repack your boot.img, with the correct cmdline ("mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" works,) etc.
5) prepare the ext2 /system partition with your target ROM /system files.
How to load the ROM:
steps a) to f) done on your PC. step g) is done on the phone.
a) Unzip the ROM on your PC, to get at the files to copy to the new /system directory in adb, and allow you to modify the updater-script.
b) Mod that ./META-INF/com/google/android/updater-script as follows:
As an example, I removed the following from the aospCmodOV-5-16-11 updater-script:
everything except the "ui_print" "symlink" and "set_perm" lines, including the "unmount" line at the end of the script.
I thinned it down because the update-binary wants to write to MTD partition for /system, and I didn't want or need that to happen.
I did need it to install the symlinks and permissions, though.
Without those, the keyboards kept FCing, and the phone couldn't connect to the network.
c) rezip the ROM with the modified updater-script.
d) boot phone into recovery and connect to PC with cable.
e) adb push or otherwise copy the modified ROM.zip to your sdcard.
f) next, copy /system from the unzipped ROM to the new partition with adb
Code:
adb shell
mount -t ext2 /dev/block/mmcblk0p3 /system
exit
adb push /path/to/unzipped/ROM/system /system
g) on phone, in recovery: install .zip from sdcard
select the new ROM.zip you reassembled and pushed to the SD card
select yes, you really want to install the .zip
this should write the /system symlinks and permissions to the new /system partition.
Click to expand...
Click to collapse
6) back on your PC, still connected to phone with cable in recovery;
Code:
adb reboot bootloader
fastboot boot /path/to/boot.img
Step 6 is important!
this will boot from your modified boot.img without actually flashing it into the phone's NAND, so a reboot (or restart after battery pull in case of issues) just goes to your regular installed ROM in the phone.
That's about it for now. It's pretty much a hack as yet, but it's cool in a geeky way to be able to do this.
<reserved for future use>
Thanks! This should be very helpful to devs and those of us who are adventurous with our phones!
Very interesting concept, I'd love to see how many uses this could have!
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
bigsupersquid said:
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
Click to expand...
Click to collapse
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
I personally still have problems recompiling a ramdisk.
mmarz said:
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
Click to expand...
Click to collapse
I can post files one rom at a time, because the ramdisk and kernel will be a bit different for each rom, and so will the permission/symlink updater-script.
first, I'll have to get rom-dev permission, because they get modded, and also cause it's nice to have permission.
I personally still have problems recompiling a ramdisk.
Click to expand...
Click to collapse
fastboot provides a fantastic shortcut for that, if you cpio-gzip the ramdisk together after editing it to your specs
from http://android-dls.com/wiki/index.php
in a command line linux shell, from the directory containing your unpacked ramdisk,
Code:
find . | cpio -o -H newc | gzip > ../newramdisk.cpio.gz
will put newramdisk.cpio.gz in the directory above the one containing your ramdisk
Click to expand...
Click to collapse
fastboot can make the boot.img for you and either dump it into the phone (assuming your shell current directory contains your kernel (zImage) and ramdisk (newramdisk.cpio.gz))
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" flash:raw boot zImage newramdisk.cpio.gz
or boot your phone from the assembled image without actually flashing it in.
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" boot zImage newramdisk.cpio.gz
those fastboot lines assume an optimus v (or s.)
looking at all the great work you've posted on the forums, I'm surprised you have trouble handling anything to do with these phones!
I'm slowly working up to a modded recovery to install-to-sd directly and avoid all this hacking.
Nice, thanks for that. I had no idea fastboot could do that.
mmarz said:
Nice, thanks for that. I had no idea fastboot could do that.
Click to expand...
Click to collapse
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
bigsupersquid said:
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
Click to expand...
Click to collapse
Very cool i loved this idea when it was executed in a way over at Ubuntdroid for the intercept when i was doing alpha testing. Very convenient for many of the scarier tests when i wanted to keep my existing system intact. The big difference was that on the intercept the bootable partitions were written into the kernel. Downside you couldn't use it to test kernels but could switch kernels from the phone and flip-flop between completely isolated systems on the go without a computer. Which was great when i wanted to test a new ROM on the way to work but i needed my phone when i got there and had to have a painless method of switching to my usual running system. So i'd flash a "non SD" kernel and i was back up and running in just a minute or so.
I'm excited to play with this when i have some free time.
Thanks dude!
you could easily use an update.zip for each of the sd-boot and regular phone boot.img files and swap between working systems with those in recovery.
before getting the fast sd card, I was looking at this more as for testing foreign roms than two-system operation, but now I see that would work too.
for a second-system option I was looking at a market-enabled rom on sd since I don't use g-apps on my daily driver.
note: the GB rom I'm running on won't 'see' a card with more than four partitions properly, and won't mount the vfat partition with vold, although you can manually mount and access the first four partitions including the vfat. android just won't admit it's there. and I couldn't get at a fifth partition through the phone os, even though my pc could see it fine (to get 5 partitions I had to use an extended partition to contain most of the virtual partitions)
android could read virtual partitions in an extended partition just fine as long as the total number on the card didn't exceed four.
I learned a neat trick today.
going through the init.rc from the stock rom, I noticed a section at the beginning which was labeled 'on emmc' which would imply running from a card if the device hardware lacked nand flash memory.
I'd read on xda while researching this concept of boot-from-sd about an unspecified wait-for-device command in init.rc, but most of the info I've gotten on init scripting language has come from picking apart init scripts, as google has not documented the available commands very well.
long story short, I tried using devwait with the regular init and it worked just fine to mount the ext partitions. yay!
first post edited to reflect the improved method.
I've also been testing a recovery mod. if recovery.fstab is modified in the recovery ramdisk (or a fakeflash recovery) then the recovery will copy a tweaked rom.zip to the ext /system partition with some minimal modifications to the updater-script and a few commands in the adb shell.
the recovery with the modded recovery.fstab can also back up from and restore to the sdcard ext /data and /system partitions.
I will update the first post with more specific instructions as I hammer them into something consistent.
soon I'll be ready to release a modded fakeflash recovery to make experimentation easier, as well.

Nandroid defrag

Hi
Before someone say it, yes, i know that in a nand doesnt matter if the files are or not fragmented. But i had read in some other places in the internet that it actually comes to a decrease in performance when its fragmented... If its a fact or not i dont know...
But i have a lot of apps that havent been updated in a while, and im concerned about my phone become slow...
So my question is:
if a make a nandroid backup and flash and restore (in case it become slow), this restore puts all the files defragmented? Or it just do an exact image of all the things including their location (some parts here other there...)?
Thanks
According to http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_and_facts/:
Myth: Flash cards unlike hard drives do not have movable parts so defragmentation is useless.
On theory, since a flash drive does not have moving parts, its access time is independent on where the data is stored, which is why supposedly flash drives don't need defragmenting.
The fact is that flash memory is physically organized in blocks (or pages) of data, usually 128K or 256K large. Things get even worse from the fact that in order to change even one single byte, the entire page has to be first erased and then re-written with its contents again. In our example the time needed to change one byte of information is calculated the following way:
T = R + E + W
T is the total time, R is the time needed to read the entire flash page containing the byte we wish to change, E is the time required to erase the page and W is the time it takes for the data to be written back to the empty page. Not only we had to read 128KB in order to change a single byte but we also had to erase the entire block (which is very slow) and then write 128KB over again.
To complicate matters even further, it must be noted that there are additional layers between the flash card controller and the file system that cache pages being read and written to. The cache serves for improving performance. It is a simple trade-off between read/write performance and some RAM being used to cache the pages. It is most effective to read or write entire flash pages performance wise. When the operating system instructs the controller to read a particular sector on the card the cache normally retrieves the entire block and stores it internally. What this means is that information that is stored in a contiguous matter is more likely to be found in the cache than non-contiguous information.
Suppose we have a file that is 263892 bytes large and is fragmented. On a FAT32 file system using 1K cluster the file will occupy 260 clusters. In the worst case scenario the clusters will be dispersed across 260 different flash pages. If the file is contiguous all 260 clusters will fit inside 3 flash pages. Caching of the fragmented file will be impossible as it won't fit in the cache (260 pages will require 33MB of RAM to cache) while the defragmented file will fit in just 384K.
And finally the FAT file system stores folders the same way files are being stored - in cluster chains. A large folder that is fragmented and is not cached represents a huge performance penalty for standard file operations like listing the files in that folder, renaming or even deleting a file.
Conclusion: Fragmentation has a serious impact on flash card performance especially during write operations and when the file system is heavily fragmented across many different flash pages.
Click to expand...
Click to collapse
Now, about your question. It would seem logical, as at that time, every partition is written in order, thereby it should be defragmented. During normal usage, you change this here, change that there, thereby different blocks are edited, causing fragmentation.
^^
Hey thanks!!
I supose that this is why explore an album or images cause some lag.
Also, i had noted that not even putting all images in order (contiguous), the cache always creates a lot of chunks...
Using this thread...
I just formatted the sd-ext, and it appears in link2sd as ext2, but i had it as ext4.
Could it lead to problems? Do i need to reformat my sd card with an external reader? Or from the cwm?
Ive restored all my apps and they work, but im not sure...
Thanks

[MOD] ext2 file system for /system - Feb. 17 UPDATED

In short, what I will be doing here is mounting the /system partition as file system EXT2. Default is EXT4. The reason for doing this is simple. The /system partition is more of an access database where information is read and never written. The goal here is to remove journaling and not only journaling but the redundancy of the code tree of EXT4 by simply killing the EXT4 journal and leaving it mounted and formatted as EXT4.
EXT4 – The file system uses a function called journaling. Journaling, in short, is a file system log. What this provides to you as the user is a quick method of recovery if there is ever a system failure: unclean shutdown, file corruption, etc. If there is ever a problem, the journal is called and the data is restored with no issues. The cost of this journaling feature, however, is CPU time/usage. CPU cycles are used to write the journal to the disk.
EXT2 – This file system does not use journal. No recovery method, etc. So if something is corrupted, there is no way to recover corrupted files/blocks/data.
What does all of this mean? – Simple. What it means is those of us who are rooted, running a stable ROM/Kernel combo, and using a backup method such as TWRP or CWM, can safely use EXT2 for /system without any worry because all we need to do is make a backup of our /system partition via recovery, tuck it away on our internal SD card and save it for a rainy day - restore it if there is every an unclean shutdown (battery pull, reboot caused by an unstable kernel, etc.).
What benefits will you see/feel?
Honestly, I have no idea at this point. I know for a fact that the device will boot faster and read operations on the /system partition will be absolutely insanely fast. As fast as they can possibly be on the Note 3. As far as “end of the day” battery savings, well, this is kind of the point of me doing this and sharing. I am going to find out, and post my results here so other people can decide if they do or don’t want to do this as well. If all goes well, I’ll edit this post tomorrow with some quick instruction on how to convert the file system. It should be a quick and easy process.
Other benefits of using EXT2 instead of EXT4:
EXT2 is backward compatible with EXT4. What this means is you can have a file system formatted to EXT2, and it can be mounted as EXT4 and it will only utilize the newer useful features found in EXT4 that were not originally present when EXT2 was introduced. When mounted as EXT4, EXT2 will not use journaling, but will use EXT4’s superior block allocation and “tagging” feature. EXT4 has the ability to mark unused blocks on the disk so it knows to not look there for data – this saves precious time in read operations. EXT2 does not have this feature… except when it is mounted as EXT4
The biggest advantage, here, is to get rid of that useless EXT4 journaling feature used on our RO /system and save CPU cycles. We use backups.We are a different breed of users, right? We demand performance and push our devices to the limits and somehow manage to maintain a stable and usable system, correct?
YES! So, we’re gonna go ahead and take advantage of this super awesome backward compatibility of EXT2 on the /system partition and get some positive performance index out of it
INSTRUCTIONS - convert /system to EXT2
1. Unzip the .zip folder of your ROM of choice.
2. Find /META-INF/com/google/android/updater script.
3. Open the updater-script file in a text editor.
Any lines where you see "EXT4" and "system" in the same line, you need to change that 4 to a 2. Only do this for lines with system and EXT4 in the same line.
4. Save the changes, then zip the ROM back up.
5. Place ROM on your internal SD card, then boot into recovery - YOU MUST BE USING TWRP FOR THIS
6. Once you get into recovery, you will see your TWRP options (there are 8 tiles in recovery) - select the one that says "Advanced"
7. You will now see 6 tiles. On the left side, select the one that says "Terminal Command"
8. At the top you should see a / symbol - this is the directory you are going to begin your session in. You want it to be /
9. On the bottom right of the screen, hit "Select"
This next part is very important. Do not mess up.
When you get to the terminal screen, type this command:
Code:
mke2fs /dev/block/mmcblk0p23
DO NOT continue until you have double checked that the number you punched in is 23! I cannot stress this enough. 23.... 23..... 23..... again that last part is "...mmcblk0p23"
After you have verified the correct partition number 23, hit the GO button on the keyboard to execute the command. It will run for a couple seconds and finish up.
After it finishes reformatting, flash your ROM of choice (the one where you edited any lines). When you are done, boot up.
You are now running an EXT2 file system for your /system partition. Welcome to the thunder dome
Cool, can't wait to try, thanks for this finding
Sent from my SM-N900T using Tapatalk
Why not just create and mount an ext4 filesystem with no journal? There is a reason they made the ext4 driver able to mount ext2,3 filesystems. Aside from journaling, which can be disabled, ext4 and it's driver are generally better.
edit - Didn't mean to quote.
muqali said:
Why not just create and mount an ext4 filesystem with no journal? There is a reason they made the ext4 driver able to mount ext2,3 filesystems. Aside from journaling, which can be disabled, ext4 and it's driver are generally better.
edit - Didn't mean to quote.
Click to expand...
Click to collapse
The reason is because the op paths in the file system tree (speaking of EXT4) are optimized for the journal. Without the journal the code path is redundant and the journal-optimized block has empty overhead. EXT2's strength is it's simplicity in structure.
Basically what this translates to is EXT4 formatted disk, mounted as EXT4 and with no journal, is not as fast or efficient as EXT2 formatted disk mounted as EXT4.
I am running this set up right now. This thread will be updated soon with instruction on how to convert it.
Ext4 has quicker read times than Ext2 which too me is going to the biggest perofmance increase.
This sounds interesting, but I think I'll wait to see others' experiences before giving it a go myself. Will be keeping an eye on this thread though.
Sent from my SM-N9005 using XDA Premium 4 mobile app
The only reason EXT4 has faster read times is because of what I just explained - ext4 uses a method of marking unused blocks on a disk, so it doesn't scan them looking for something. EXT2 does not have this feature. However, when an EXT2 formatted disk is mounted as EXT4, it does utilize this feature of EXT4.
Do you have any data, your own or third party to back this up? Do you plan on using an ext2 system with the ext4 driver? Or the old ext2 fs driver? As I understand it, the ext4 driver was given the ability to mount 2/3 filesystems for simplicity sake as well as the newer driver being better but I could be mistaken.
edit - The way you explained it in your last post seems to sound like you'll use ext4 driver on an ext2 fs, I was just asking because I wasn't 100% sure.
muqali said:
Do you have any data, your own or third party to back this up? Do you plan on using an ext2 system with the ext4 driver? Or the old ext2 fs driver? As I understand it, the ext4 driver was given the ability to mount 2/3 filesystems for simplicity sake as well as the newer driver being better but I could be mistaken.
edit - The way you explained it in your last post seems to sound like you'll use ext4 driver on an ext2 fs, I was just asking because I wasn't 100% sure.
Click to expand...
Click to collapse
That was the intention, correct. Using EXT2 formatted /system but mount it as EXT4 - for the reasons I outlined. To effectively remove journaling and not only that, but the redundant overhead of journaling in the fs tree which would still remain if I were to, say, run this command...
Code:
tune2fs -O ^has_journal /dev/block/mmcblk0p23
EDIT** and so far, yes the difference has been noticeable in boot up time - which is a clear indicator of a performance increase. It loads the /system much faster during boot. Battery life has been slightly better as well.
Do you have a video or benchmark showing the differnce?
LancerV said:
Do you have a video or benchmark showing the differnce?
Click to expand...
Click to collapse
A benchmark? People actually use those? lol.
My own device before doing this mod booted in about 21 seconds from the off position. After doing this it was 15 seconds. I full booted it 3 times for each EXT4 and EXT2... it was consistently the same.
You don't need a benchmark to understand there is an obvious performance increase there. Battery life has also been a little better today. My usage on a daily basis is consistently the same with no changing variables. 15 hours off the charger and 75% battery with 1 hour and 30 minutes screen on time is what I typically see. Today I was 15 hours off the charger, 81% battery with 1 hour and 30 minutes screen on time.
CPU cycles have been reduced, obviously.
If you wanna give it a go... You won't regret it. Post your results here so we can compare :highfive:
Nothing wrong with storage benchmarks, hell the best Windows/Linux/Unix storage benchmark is from 2006 and works perfectly. It's pretty much a staple when it comes to setting up Fibre, and 10Gbe SAN's to make sure the throughput you should be getting you are getting and to test various raids in their IO performance along with read/write performance.
So to say lol people still use benchmarks is pretty naive.
LancerV said:
Nothing wrong with storage benchmarks, hell the best Windows/Linux/Unix storage benchmark is from 2006 and works perfectly. It's pretty much a staple when it comes to setting up Fibre, and 10Gbe SAN's to make sure the throughput you should be getting you are getting and to test various raids in their IO performance along with read/write performance.
So to say lol people still use benchmarks is pretty naive.
Click to expand...
Click to collapse
I have not seen one benchmark available on the playstore that is even remotely relevant. What a joke it is to see a tech article running a quadrant test comparing two devices using a benchmark... so, if you were talking about one of those - quadrant, antutu, etc... none of these are going to give me an accurate representation of what kind of performance increase or decrease I am seeing for /system because they don't test those partitions. There are more accurate ways to run a performance test on a disk.... but it's quicker for me to just go ahead and reboot the device since, well, I know what I am doing and happen to be doing it on a portion of the disk where a majority of the work by the CPU is coming from accessing this particular partition of the device.
During boot, there is one task being executed and one task only... booting! Any most everything being loaded is coming from where? /system
Naive? I spent time on Google's team as a software engineer a couple years back... I am far from naive. You are just not looking at this from a practical or logical point of view.
I'll give you an analogy before I go to bed so you can better understand:
If your car runs out of gas, and stops running, and somebody gives you a ride home, then somebody else calls you and says "Hey, I just saw your car on the side of the road, saw your post on G+ about running out of gas. Well, I happened to have a gas can with me and put a gallon of gas in it for you. Go pick it up!"
Now you get a lift back to your car, put the key in, and it starts right up. Do you then turn it off and walk to a gas station, get a gas can, walk back, put more gas in it, and drive away? No, you don't. Because you have already obtained a positive result. There is no need for "testing" at that point. You have clearly established a positive result.
Anyways, benchmarks.... can be useful sometimes. But you are mistaken if you think:
A) They will give me anymore information than I know at this point regarding the test method...
B) I am going to be allocating more time and resources to run a "relevant" test on my /system partition
C) I have not already obtained a clear result
Not trying to sound arrogant... but I've been doing this a while. I'll pass on the benchmarking this time around.
cun7 said:
I have not seen one benchmark available on the playstore that is even remotely relevant. What a joke it is to see a tech article running a quadrant test comparing two devices using a benchmark... so, if you were talking about one of those - quadrant, antutu, etc... none of these are going to give me an accurate representation of what kind of performance increase or decrease I am seeing for /system because they don't test those partitions. There are more accurate ways to run a performance test on a disk.... but it's quicker for me to just go ahead and reboot the device since, well, I know what I am doing and happen to be doing it on a portion of the disk where a majority of the work by the CPU is coming from accessing this particular partition of the device.
During boot, there is one task being executed and one task only... booting! Any most everything being loaded is coming from where? /system
Naive? I spent time on Google's team as a software engineer a couple years back... I am far from naive. You are just not looking at this from a practical or logical point of view.
I'll give you an analogy before I go to bed so you can better understand:
If your car runs out of gas, and stops running, and somebody gives you a ride home, then somebody else calls you and says "Hey, I just saw your car on the side of the road, saw your post on G+ about running out of gas. Well, I happened to have a gas can with me and put a gallon of gas in it for you. Go pick it up!"
Now you get a lift back to your car, put the key in, and it starts right up. Do you then turn it off and walk to a gas station, get a gas can, walk back, put more gas in it, and drive away? No, you don't. Because you have already obtained a positive result. There is no need for "testing" at that point. You have clearly established a positive result.
Anyways, benchmarks.... can be useful sometimes. But you are mistaken if you think:
A) They will give me anymore information than I know at this point regarding the test method...
B) Allocating the time and resources to run a relevant test on my /system partition
C) I have not already obtained a clear result
Not trying to sound arrogant... but I've been doing this a while. I'll pass on the benchmarking this time around.
Click to expand...
Click to collapse
Right on. Great work OP. While EXT2 isnt for everyone. Experienced users can gsin value from using it. Nice Work!!!
Sent from my Samsung Galaxy Note 3 using JellyBombed Tapatalk.
My phone takes about 15secs to boot up not sure Im going to really notice a 1-2sec diffrence, a better analogy would be taking a car that does 0-60 in 6secs and a pos looking car and one that does 0-60 in 6.2 secs and be looking like a sports car. The differnce would be so minuscial that if you asked people which one accelerated faster they would naturally pick the one that looks like a sports car because thats how peoples minds are trained to think sportty looking car = quicker.
Its kinda a mental placebo effect, hence why I asking if you had a boot up video seems like that would be proof enough.
Also ran a top command and suprise system is using 1% and its the top command so I dont see where you are coming up with the extreme cpu cycles you say journaling is causing. Not to mention the fact /system is pretty much only called upon boot. Most other times its going to be pulling r/w cycles from /data or /sdcard.
LancerV said:
My phone takes about 15secs to boot up not sure Im going to really notice a 1-2sec diffrence, a better analogy would be taking a car that does 0-60 in 6secs and a pos looking car and one that does 0-60 in 6.2 secs and be looking like a sports car. The differnce would be so minuscial that if you asked people which one accelerated faster they would naturally pick the one that looks like a sports car because thats how peoples minds are trained to think sportty looking car = quicker.
Its kinda a mental placebo effect, hence why I asking if you had a boot up video seems like that would be proof enough.
Also ran a top command and suprise system is using 1% and its the top command so I dont see where you are coming up with the extreme cpu cycles you say journaling is causing. Not to mention the fact /system is pretty much only called upon boot. Most other times its going to be pulling r/w cycles from /data or /sdcard.
Click to expand...
Click to collapse
The placebo effect you are speaking of are typically people that don't know what they are doing, and are simply just excited about something.
An example of this would be the 9 out of every 10 posts I see in a ROM thread over the years talking about how much smoother their device is after flashing "xxxxx ROM"... a stop watch is a pretty reliable method for checking boot times, which is what I used for this particular scenario because I knew it would be relevant. Also the fact that my usage is pretty much exactly the same every day is a pretty solid test attribute. Any change in battery life at 15 hours... even if it is 1%... I notice it.
Also, my own personal ROM that I use is entirely odexed. So, nothing is being drawn from /data/dalvik-cache that is a /system/framework or /system/app function. It is noticeably more efficient because the 280+ .odex files are being read from the same location... which is now ext2, and not ext4.
Is this kind of starting to make sense now or am I losing you more?
I am not aware of all the file systems on linux. What happens to the partition if you restore a nandroid backup of the system partition that was converted to ext2 but the backup was ext4? And/or if backup was ext2? Just curious.
Sent from my SM-N900T using Tapatalk
Compusmurf said:
I am not aware of all the file systems on linux. What happens to the partition if you restore a nandroid backup of the system partition that was converted to ext2 but the backup was ext4? And/or if backup was ext2? Just curious.
Sent from my SM-N900T using Tapatalk
Click to expand...
Click to collapse
It will write the data to the disk as it should. You can do this without having any issues.
I've never tried restoring an EXT2 backup to an EXT4 formatted disk... So I have no answer for you there. It all kind of depends on exactly how TWRP's backups are created... I think they use .tars ... I'm not sure
Would this by any chance work on an AOSP/CM/AOKP ROM? I've tried using it with CM and LiquidSmooth but it stops every time after "erasing system" with "script aborted (no error message)".
Nevermind, I found out the issue. I went back into the updater-script and removed this line: "format("ext4", "EMMC", "/dev/block/platform/msm_sdcc.1/by-name/system", "0", "/system");". It flashed perfectly after that (Liquid Smooth AOSP ROM).
Edit: Had an issue with the ROM not booting at first, it hung on the splash screen. What I did was reboot into TWRP, wiped cache and dalvik cache, went into terminal (in advanced) and typed this command: "setenforce 0". After that, type "getenforce" and make sure SELinux is set to Permissive. I'm not sure if the SELinux command helped at all but I did it just in case.
Hypercore said:
Would this by any chance work on an AOSP/CM/AOKP ROM? I've tried using it with CM and LiquidSmooth but it stops every time after "erasing system" with "script aborted (no error message)".
Nevermind, I found out the issue. I went back into the updater-script and removed this line: "format("ext4", "EMMC", "/dev/block/platform/msm_sdcc.1/by-name/system", "0", "/system");". It flashed perfectly after that (Liquid Smooth AOSP ROM).
Edit: Had an issue with the ROM not booting at first, it hung on the splash screen. What I did was reboot into TWRP, wiped cache and dalvik cache, went into terminal (in advanced) and typed this command: "setenforce 0". After that, type "getenforce" and make sure SELinux is set to Permissive. I'm not sure if the SELinux command helped at all but I did it just in case.
Click to expand...
Click to collapse
Code:
"format("ext4", "EMMC", "/dev/block/platform/msm_sdcc.1/by-name/system", "0", "/system");"
Change that 4 to a 2...
Code:
"format("[COLOR="Red"]ext2[/COLOR]", "EMMC", "/dev/block/platform/msm_sdcc.1/by-name/system", "0", "/system");"

Project: Galaxy Nexus Partition 2.0. GNP2!

Edit:
There has been no new information provided in this thread about how big /cache NEEDS to be. To me it seems like /cache is simply unused. It's there for Google to use with stock ROMs, which we do not use anymore. So redistributing storage from /cache to /system makes good sense to me. Exactly how much should be removed from /cache is up for debate, but if we take most of it away, that improves /system significantly, and still leaves ~100mb in /cache for contingency.
@osm0sis has contributed work which will allow an enthusiast to shrink their /cache and add that space to /system. Because of the way @Ziyan's ROMs are flashed, flashing a ROM will revert any change you make to /system back to the default size, until you "use 'resize FS' option in TWRP" according to @defy_owner. http://forum.xda-developers.com/gal...n-2-0-gnp2-t3147253/post64482777#post64482777
@osm0sis' flashable script UPDATE-GN.PIT.Editor.zip is an excellent bit of work contributed to the community of users and developers -- users can modify their devices themselves, and ROM developers now know better than ever how to make the switch to a bigger /system when necessary. Thank you ALL for your contributions to this thread!!!
Original post:
Per: http://forum.xda-developers.com/gal...eta-builds-t3060319/post60805347#post60805347 and http://forum.xda-developers.com/gal...eta-builds-t3060319/post61089590#post61089590 -- I'm an interested newbie and not a developer, but I will try to contribute where I can... firstly by starting this thread.
The /system and /cache partition sizes on Nexus devices have changed multiple times since Galaxy Nexus launched. By the time the 64bit Nexus 9 launched, /system had more than doubled in size! Luckily we don't need to worry about accommodating 64bit binaries. We also don't necessarily need to accommodate the giant Google Apps packages that Google includes, because Google provides most of those apps on the Play Store.
But we DO need more space if we want more FLEXIBILITY in the future. Flexibility to add features... flexibility to migrate to larger versions of Android... If we want Galaxy Nexus to survive to 2020, we need flexibility!
If the community developers for Galaxy Nexus can come to an agreement, we can all standardize on one partition layout to move forward with. That way, flashing back and forth between new OS distribuitions will be easy. So because I'm not a developer, I'll do my part by helping this discussion get started. I'll invite some developers to this thread who I've seen active. Anyone can of course participate. Then, when the smart people have said what they think, we can decide where to go from there. I'll begin by inviting: @Ziyan @bsmitty83 @MWisBest @Carlos_Manuel @zzpianoman @osm0sis @Fenix46 @theBlackEnd @sgspluss @aosp
The plan here is for you guys to debate with each other about this topic. To start the debate, I will start with an opening statement which is not necessarily my opinion. Then I will provide a list of helpful proposed tasks which volunteers can complete. I will then list a discussion agenda which you may or may-not want to talk about.
Opening statement: Galaxy Nexus partitions are out of date, but don't need much change (if any at all). But because F2FS on /data is faster and also requires a phone wipe, we should make both of these changes at once. The incremental hassle of doing one change on top of the other is minimal. The exact partition sizes don't matter. Because the Nexus 5 partition layout seems sufficient for future flexibility, we should just copy it. It was not designed for Android L or M like Nexus 6 was, but the Nexus 5 layout is close enough to "good enough".
Tasks v0.1:
0) What are the partition sizes of every Nexus device?
1) What are the partition sizes of popular 16GB NON-Nexus devices? Does Google have suggestive documentation for hardware developers?
Discussion Agenda v0.2:
-1)If F2FS is good for users, and a new partition layout is good for users, and BOTH require destructive migration of users' data, should we make both changes at the same time?
0) What are the obvious pros/cons of changing the partition layout? How do we balance the needs of the people who want LOTS of new space against those who need NO new space?
1) How long do we expect this size change to be sufficient for the community? 1 year or 10? Until Google stops supporting Nexus 5 or 6, or long after?
2) Decide how big /system should be. How much margin for future growth? Is there a rationale for any given size, or is choosing a round number ("1GB") as good as any?
3) Decide how big /cache should be. Is shrinking it a good idea? Would that saved space be large enough to matter? Is there risk of compatibility problems?
4) Reality check: Is it okay that we're thinking of shrinking the /data partition this much? What are the negative repercussions?
5) How do we maintain our development work and migrate all developers into the future? Do we maintain two different builds of the OS in parallel for some time? What timeline can we operate this way? 1 month or 1 year?
6) Does anyone absolutely OPPOSE attempting to standardize on a new partition layout for community development?
I invite any interested developer to speak about any topic. You are the people who really matter, because you volunteer to do the real work the rest of us need. Let's begin. Good luck to us all!
Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83
bamtan2 said:
Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83
Click to expand...
Click to collapse
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .
I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, and even that's probably just playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.
bsmitty83 said:
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .
Click to expand...
Click to collapse
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
osm0sis said:
I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, even playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.
Click to expand...
Click to collapse
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...
bamtan2 said:
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...
Click to expand...
Click to collapse
Galaxy Nexus: 654mb system, 432mb cache
Nexus 4: 840mb system, 560mb cache
Nexus 5: 1074mb system, 734mb cache
Nexus 6: 1946mb system, 248mb cache
Moto X (2013): 1611mb system, 1067mb cache
Moto G (2013): 1023mb system, 694mb cache
Until the N6, Nexus phone device /cache has been ~66.6% of /system. I think the N6's stance on it should be standard going forward however, especially for us, since as I mentioned in my previous post, /cache is only really used by OTAs in >ICS Android and the GN is certainly not receiving any more of those, making it mostly up for grabs. If it's still percentage-based, the N6 would make it only ~12.5%.
"Stock" PIT for our device: http://forum.xda-developers.com/galaxy-nexus/general/guide-restore-to-stock-unbrick-galaxy-t2065470
We can extract it to use as our starting point. I'm unsure whether we need a different PIT file for each variant (maguro vs. toro vs. toroplus), but I'll figure out the correct offsets for the command to extract the PIT from our emmc (a la this) and we can find out definitively.
@mn.code's 1gb system PIT: http://forum.xda-developers.com/showpost.php?p=60798125&postcount=663
A good example of a working modified PIT for the GN. I believe he reclaimed all of the added space from /data however, and he has still yet to provide his documentation on modifying and flashing the PIT file. Maybe he can provide it now in this thread to save us some time. :good:
Official ODIN download site: http://odindownloader.com/
PIT Magic: http://forum.xda-developers.com/showthread.php?t=1916936
Seems to be the defacto app for creating and modifying PIT files.
@zzpianoman's ZMoD 0518 tar: http://forum.xda-developers.com/showpost.php?p=60824237&postcount=704
Since mn reported ROM flashing reverted the partition size changes. Though this can maybe be worked around by flashing the new nightly, backing it up, formatting and then restoring. Better would be @Ziyan switching from block system.dat patching back to the old separate file-based zip format; Google hasn't even switched the newer (Nexus 4, 5 and 7) devices' OTA zips to the new block format, so there's no reason the GN needs to be on it, especially when it reverts our partition tinkering due to a specific partition size flag with which the system.dat has to be built. It would also allow addon.d to work again, which would be very handy with our new expanded /system partition.
This also solves the question of different builds, provided they merely remain small enough to fit on the base-size partition. With system.dat OTA-style out of the picture not even formatting should affect the size. At worst, GApps flashing would have to be sacrificed by those not willing to move forward if the standard ROM size kept increasing. As for the F2FS query, it never needs to be mandatory so I wouldn't worry about it. Some people seem to have some issues with it as well, but that could be an issue of devs using either too aggressive flags or not aggressive enough flags. F2FS is still pretty new for Android and I'm not sure anyone's really mastered the tweaks.
I expect increasing to a reasonable size like 1.2gb would allow for the proper expansion Lollipop has caused and M will likely also, as well as all the GApps one would want to cram on there. I'd imagine this will be sufficient until the hardware fails to support the latest Android versions or literally fails altogether.
@osm0sis I didn't have maguro last two months, I sold titanium colored one and now I bought a white one So, as promised, I will write a tutorial on how to flash new pit and how to untar the file @zzpianoman provided as soon as I get home from vacation. Also, I will make a new pit file (old one "borrowed" space only from /data), I will increase /system with another ~200MB (will take some from /cache and maybe some more from /data), so the /system partition will be around 1,2GB.
After examining a raw dump I did of the beginning of mmcblk0, the PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interestingly, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Edit 2: Fixed the command above. bs=1 when it's not a raw dump apparently. :good:
osm0sis said:
The PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=8 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interesting, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Click to expand...
Click to collapse
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice
bsmitty83 said:
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice
Click to expand...
Click to collapse
Okay, here's what I've worked out:
PIT structure -
block sizes, 4 bytes (little endian) from offsets:
system=1500
cache=1632
userdata=1764
block counts, 4 bytes (little endian) from offsets:
system=1504
cache=1636
userdata=1768
blocks / 2048 * 1024 = kB
Code:
system = 1,339,392 = 669696 (654 MB)
cache = 884,736 = 442368 (432 MB)
userdata("stock") = 31,174,656 = 15587328 (15222 MB) [maguro, ?]
userdata(bamtan2) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250]
userdata(osm0sis) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250M]
userdata(bsmitty83) = 59,944,926 = 29972463 (29269.9833984375 MB) [toro, SCH-i515]
userdata(musical_chairs) = 59,944,926 = 29972463 (29269.9833984375 MB) [toroplus, SPH-L700]
In all files but @mn.code's nothing changes but the block counts, even with the jump from 16gb to 32gb. He's changing block size on everything as well and I'm not sure that's something we should touch.. Maybe he can explain?
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Edit: I tried generating and flashing a PIT based on my own dump (seems like the safest approach to avoid exceeding chip capacity), simply by subtracting 304mb of blocks from cache and adding them to system (so cache at 128mb, and I also tried cache at 256mb), and I could not make it stable... Damn thing kept corrupting system when I'd format cache and vice versa.
So I then took the risk and flashed @mn.code's instead and it definitely works. So he'll have to tell us what the trick is with the math. In better news, there's no need to mess around with tar's like we thought, even if @Ziyan doesn't go back to file OTA-style flashable zips for awhile, since TWRP's new(ish) Resize FS command resolves it. Run it after flashing the ROM then you have your space added back to flash GApps, etc. Really, we want addon.d support though, so hopefully ZMoD will be switched back anyway, but this is a simple workaround for now.
Edit 2: Updated with the latest. A 16gb Verizon i516 and a couple maguros with different emmc chips (non-VYL00M) would be appreciated now. Thanks everyone!
osm0sis said:
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Click to expand...
Click to collapse
Here's a dump from a toroplus - is this what you are looking for?
@osm0sis Mine is like yours, but 1 month senior:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011
@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way
musical_chairs said:
Here's a dump from a toroplus - is this what you are looking for?
Click to expand...
Click to collapse
bamtan2 said:
@osm0sis Mine is like yours, but 1 month senior:
Code:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011
Click to expand...
Click to collapse
Thanks guys! I've updated my previous PIT structure post with your info. Now I just need a couple non-VYL00M EMMC chip devices.
mn.code said:
@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way
Click to expand...
Click to collapse
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!
osm0sis said:
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!
Click to expand...
Click to collapse
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.
mn.code said:
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.
Click to expand...
Click to collapse
I modified my dump using the values from yours and the MD5 matches yours, so I think the latest PIT Magic seems to write the userdata value correctly; the issue I had was that I didn't change cache's block size since I didn't understand that was a sum of the previous partition's values yet. Thanks! :good:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
With a universal solution still in mind, leaving userdata untouched for the various device variants and only adjusting system and cache, this afternoon I've whipped up the following:
/tmp/pitfix.sh:
Code:
#!/sbin/sh
# PIT Updater for Galaxy Nexus
# osm0sis @ xda-developers
#
# Alters cache to the preset length in MB, adding the difference to system
# new cache block count (length) in MB
cacheMB=128;
# block device containing the PIT
block=/dev/block/mmcblk0;
# offset of the PIT in the block device, and partition size offsets within the PIT; count offsets are size offset + 4
pit=17408;
sysOffset=1500;
cacheOffset=1632;
dataOffset=1764;
# functions for dumping then converting the little endian hexadecimal into decimal for calculations and converting then writing back
dump_hex2dec() { echo "$(printf '%d\n' 0x$(dd ibs=1 count=4 skip=$(($1+pit)) obs=1 if=$block 2>/dev/null | od -hv | head -n 1 | awk '{ print $3$2 }'))"; }
write_dec2hex() { echo -ne "$(printf '%08x\n' $2 | sed 's/.\{2\}/& /g' | awk '{ print "\\x"$4"\\x"$3"\\x"$2"\\x"$1 }')" | dd obs=1 count=4 seek=$(($1+pit)) of=$block; }
# create a backup of the existing, untouched PIT with a timestamped filename
dd if=$block of=/sdcard/backup-$(date +%Y-%m-%d.%H%M%S).pit bs=1 count=4096 skip=$pit;
# check the PIT against known values for userdata to ensure it is an original PIT for the device
dataSize=`dump_hex2dec $dataOffset`;
dataCount=`dump_hex2dec $((dataOffset+4))`;
if [ "$dataSize" != 2379776 ]; then
echo "Non-standard userdata size (start offset) detected... aborting."; exit 1;
fi;
case $dataCount in
28397534|59944926) ;;
*) echo "Non-standard userdata count (length) detected... aborting."; exit 1;;
esac;
# calculate new block counts (lengths) for cache and system; 1 MB is 2048 blocks
sysCount=`dump_hex2dec $((sysOffset+4))`;
cacheCount=`dump_hex2dec $((cacheOffset+4))`;
cacheCountNew=$((cacheMB*2048));
sysCountNew=$((sysCount+cacheCount-cacheCountNew));
# calculate new block size (start offset) for cache; the sum of the previous partition (system)'s block size and block count
sysSize=`dump_hex2dec $sysOffset`;
cacheSizeNew=$((sysSize+sysCountNew));
# abort if cache has already been altered
if [ "$cacheCount" == "$cacheCountNew" ]; then
echo "cache count (length) already matches target amount... aborting."; exit 1;
fi;
# write back the updated values to the PIT in the block device
write_dec2hex $cacheOffset $cacheSizeNew;
write_dec2hex $((cacheOffset+4)) $cacheCountNew;
write_dec2hex $((sysOffset+4)) $sysCountNew;
# zero minor gpt/efs areas of the PIT to mimic external editing and flashing
for i in 200 596 632; do
echo -ne "\x00\x00\x00" | dd obs=1 count=3 seek=$((i+pit)) of=$block;
done;
# format the altered partitions to their new lengths; userdata remains untouched
#make_ext4fs /system;
#make_ext4fs /cache;
# repair and resize all partitions as a final measure to ensure data integrity
#e2fsck /system;
#resize2fs /system;
#e2fsck /cache;
#resize2fs /cache;
#e2fsck /data;
#resize2fs /data;
echo "Success!";
exit 0;
This is pretty much the sum of all our knowledge thus far. I'm still working out the format commands so they're commented out, but otherwise it should work, though I'd caution against trying it just yet. It needs to be run from recovery, but I haven't decided what I want the delivery method to be (as a shell script update-binary, run by another shell script update-binary, or run by an EDIFY updater-script), beyond a flashable zip. Thoughts? Comments? Fixes?
Edit: Fixed the dump_hex2dec function to work in recovery. I also noticed in my function testing that even though the 15222 MB userdata block count value was passed to Odin using @mn.code's -osm0sisV2 PIT above, the value actually written (verified by dump) was the correct, lower one. I'm guessing Odin knows best so I've removed the higher one as an accepted value from the script. :good:
Edit 2: I've now thoroughly tested it by running it manually in recovery after adb pushing it to /tmp but I've run into a problem with recovery not acknowledging the new sizing in make_ext4fs, or resize2fs. I have dumped the adjusted PIT and it looks flawless, all the values were written correctly and nothing else was touched (verified in a hex editor). fastboot formats the partitions to the correct new sizes, but then recovery acts like they're corrupt, showing them as 0 until you format, at which point they go back to being their old sizes.
Edit 3: Odin flashing the dump of my adjusted PIT and then fastboot formatting resulted in recovery recognizing everything properly again. So there must be some voodoo we're not aware of that Odin does after flashing the PIT. Any ideas?
If all else fails, I guess I can just make a flashable zip to dump the original PIT as a backup, copy the backup file, adjust the values in the file only, then instruct a user to Odin flash their personalized PIT, but that's not nearly as fun or useful.
Edit 4: Add zeroing of minor gpt/efs areas as per this post. Still not registering in recovery though, as above.
osm0sis said:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
Click to expand...
Click to collapse
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?
mn.code said:
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?
Click to expand...
Click to collapse
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.
osm0sis said:
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.
Click to expand...
Click to collapse
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?
mn.code said:
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?
Click to expand...
Click to collapse
Official, which has worked perfectly for everything on my end when a working PIT is flashed via Odin, resizefs, format and everything. I was more hoping for your ideas on what Odin could be doing after flashing that I'm not when I edit the PIT directly.

Categories

Resources