Edit:
There has been no new information provided in this thread about how big /cache NEEDS to be. To me it seems like /cache is simply unused. It's there for Google to use with stock ROMs, which we do not use anymore. So redistributing storage from /cache to /system makes good sense to me. Exactly how much should be removed from /cache is up for debate, but if we take most of it away, that improves /system significantly, and still leaves ~100mb in /cache for contingency.
@osm0sis has contributed work which will allow an enthusiast to shrink their /cache and add that space to /system. Because of the way @Ziyan's ROMs are flashed, flashing a ROM will revert any change you make to /system back to the default size, until you "use 'resize FS' option in TWRP" according to @defy_owner. http://forum.xda-developers.com/gal...n-2-0-gnp2-t3147253/post64482777#post64482777
@osm0sis' flashable script UPDATE-GN.PIT.Editor.zip is an excellent bit of work contributed to the community of users and developers -- users can modify their devices themselves, and ROM developers now know better than ever how to make the switch to a bigger /system when necessary. Thank you ALL for your contributions to this thread!!!
Original post:
Per: http://forum.xda-developers.com/gal...eta-builds-t3060319/post60805347#post60805347 and http://forum.xda-developers.com/gal...eta-builds-t3060319/post61089590#post61089590 -- I'm an interested newbie and not a developer, but I will try to contribute where I can... firstly by starting this thread.
The /system and /cache partition sizes on Nexus devices have changed multiple times since Galaxy Nexus launched. By the time the 64bit Nexus 9 launched, /system had more than doubled in size! Luckily we don't need to worry about accommodating 64bit binaries. We also don't necessarily need to accommodate the giant Google Apps packages that Google includes, because Google provides most of those apps on the Play Store.
But we DO need more space if we want more FLEXIBILITY in the future. Flexibility to add features... flexibility to migrate to larger versions of Android... If we want Galaxy Nexus to survive to 2020, we need flexibility!
If the community developers for Galaxy Nexus can come to an agreement, we can all standardize on one partition layout to move forward with. That way, flashing back and forth between new OS distribuitions will be easy. So because I'm not a developer, I'll do my part by helping this discussion get started. I'll invite some developers to this thread who I've seen active. Anyone can of course participate. Then, when the smart people have said what they think, we can decide where to go from there. I'll begin by inviting: @Ziyan @bsmitty83 @MWisBest @Carlos_Manuel @zzpianoman @osm0sis @Fenix46 @theBlackEnd @sgspluss @aosp
The plan here is for you guys to debate with each other about this topic. To start the debate, I will start with an opening statement which is not necessarily my opinion. Then I will provide a list of helpful proposed tasks which volunteers can complete. I will then list a discussion agenda which you may or may-not want to talk about.
Opening statement: Galaxy Nexus partitions are out of date, but don't need much change (if any at all). But because F2FS on /data is faster and also requires a phone wipe, we should make both of these changes at once. The incremental hassle of doing one change on top of the other is minimal. The exact partition sizes don't matter. Because the Nexus 5 partition layout seems sufficient for future flexibility, we should just copy it. It was not designed for Android L or M like Nexus 6 was, but the Nexus 5 layout is close enough to "good enough".
Tasks v0.1:
0) What are the partition sizes of every Nexus device?
1) What are the partition sizes of popular 16GB NON-Nexus devices? Does Google have suggestive documentation for hardware developers?
Discussion Agenda v0.2:
-1)If F2FS is good for users, and a new partition layout is good for users, and BOTH require destructive migration of users' data, should we make both changes at the same time?
0) What are the obvious pros/cons of changing the partition layout? How do we balance the needs of the people who want LOTS of new space against those who need NO new space?
1) How long do we expect this size change to be sufficient for the community? 1 year or 10? Until Google stops supporting Nexus 5 or 6, or long after?
2) Decide how big /system should be. How much margin for future growth? Is there a rationale for any given size, or is choosing a round number ("1GB") as good as any?
3) Decide how big /cache should be. Is shrinking it a good idea? Would that saved space be large enough to matter? Is there risk of compatibility problems?
4) Reality check: Is it okay that we're thinking of shrinking the /data partition this much? What are the negative repercussions?
5) How do we maintain our development work and migrate all developers into the future? Do we maintain two different builds of the OS in parallel for some time? What timeline can we operate this way? 1 month or 1 year?
6) Does anyone absolutely OPPOSE attempting to standardize on a new partition layout for community development?
I invite any interested developer to speak about any topic. You are the people who really matter, because you volunteer to do the real work the rest of us need. Let's begin. Good luck to us all!
Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83
bamtan2 said:
Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83
Click to expand...
Click to collapse
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .
I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, and even that's probably just playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.
bsmitty83 said:
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .
Click to expand...
Click to collapse
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
osm0sis said:
I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, even playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.
Click to expand...
Click to collapse
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...
bamtan2 said:
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...
Click to expand...
Click to collapse
Galaxy Nexus: 654mb system, 432mb cache
Nexus 4: 840mb system, 560mb cache
Nexus 5: 1074mb system, 734mb cache
Nexus 6: 1946mb system, 248mb cache
Moto X (2013): 1611mb system, 1067mb cache
Moto G (2013): 1023mb system, 694mb cache
Until the N6, Nexus phone device /cache has been ~66.6% of /system. I think the N6's stance on it should be standard going forward however, especially for us, since as I mentioned in my previous post, /cache is only really used by OTAs in >ICS Android and the GN is certainly not receiving any more of those, making it mostly up for grabs. If it's still percentage-based, the N6 would make it only ~12.5%.
"Stock" PIT for our device: http://forum.xda-developers.com/galaxy-nexus/general/guide-restore-to-stock-unbrick-galaxy-t2065470
We can extract it to use as our starting point. I'm unsure whether we need a different PIT file for each variant (maguro vs. toro vs. toroplus), but I'll figure out the correct offsets for the command to extract the PIT from our emmc (a la this) and we can find out definitively.
@mn.code's 1gb system PIT: http://forum.xda-developers.com/showpost.php?p=60798125&postcount=663
A good example of a working modified PIT for the GN. I believe he reclaimed all of the added space from /data however, and he has still yet to provide his documentation on modifying and flashing the PIT file. Maybe he can provide it now in this thread to save us some time. :good:
Official ODIN download site: http://odindownloader.com/
PIT Magic: http://forum.xda-developers.com/showthread.php?t=1916936
Seems to be the defacto app for creating and modifying PIT files.
@zzpianoman's ZMoD 0518 tar: http://forum.xda-developers.com/showpost.php?p=60824237&postcount=704
Since mn reported ROM flashing reverted the partition size changes. Though this can maybe be worked around by flashing the new nightly, backing it up, formatting and then restoring. Better would be @Ziyan switching from block system.dat patching back to the old separate file-based zip format; Google hasn't even switched the newer (Nexus 4, 5 and 7) devices' OTA zips to the new block format, so there's no reason the GN needs to be on it, especially when it reverts our partition tinkering due to a specific partition size flag with which the system.dat has to be built. It would also allow addon.d to work again, which would be very handy with our new expanded /system partition.
This also solves the question of different builds, provided they merely remain small enough to fit on the base-size partition. With system.dat OTA-style out of the picture not even formatting should affect the size. At worst, GApps flashing would have to be sacrificed by those not willing to move forward if the standard ROM size kept increasing. As for the F2FS query, it never needs to be mandatory so I wouldn't worry about it. Some people seem to have some issues with it as well, but that could be an issue of devs using either too aggressive flags or not aggressive enough flags. F2FS is still pretty new for Android and I'm not sure anyone's really mastered the tweaks.
I expect increasing to a reasonable size like 1.2gb would allow for the proper expansion Lollipop has caused and M will likely also, as well as all the GApps one would want to cram on there. I'd imagine this will be sufficient until the hardware fails to support the latest Android versions or literally fails altogether.
@osm0sis I didn't have maguro last two months, I sold titanium colored one and now I bought a white one So, as promised, I will write a tutorial on how to flash new pit and how to untar the file @zzpianoman provided as soon as I get home from vacation. Also, I will make a new pit file (old one "borrowed" space only from /data), I will increase /system with another ~200MB (will take some from /cache and maybe some more from /data), so the /system partition will be around 1,2GB.
After examining a raw dump I did of the beginning of mmcblk0, the PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interestingly, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Edit 2: Fixed the command above. bs=1 when it's not a raw dump apparently. :good:
osm0sis said:
The PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=8 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interesting, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Click to expand...
Click to collapse
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice
bsmitty83 said:
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice
Click to expand...
Click to collapse
Okay, here's what I've worked out:
PIT structure -
block sizes, 4 bytes (little endian) from offsets:
system=1500
cache=1632
userdata=1764
block counts, 4 bytes (little endian) from offsets:
system=1504
cache=1636
userdata=1768
blocks / 2048 * 1024 = kB
Code:
system = 1,339,392 = 669696 (654 MB)
cache = 884,736 = 442368 (432 MB)
userdata("stock") = 31,174,656 = 15587328 (15222 MB) [maguro, ?]
userdata(bamtan2) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250]
userdata(osm0sis) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250M]
userdata(bsmitty83) = 59,944,926 = 29972463 (29269.9833984375 MB) [toro, SCH-i515]
userdata(musical_chairs) = 59,944,926 = 29972463 (29269.9833984375 MB) [toroplus, SPH-L700]
In all files but @mn.code's nothing changes but the block counts, even with the jump from 16gb to 32gb. He's changing block size on everything as well and I'm not sure that's something we should touch.. Maybe he can explain?
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Edit: I tried generating and flashing a PIT based on my own dump (seems like the safest approach to avoid exceeding chip capacity), simply by subtracting 304mb of blocks from cache and adding them to system (so cache at 128mb, and I also tried cache at 256mb), and I could not make it stable... Damn thing kept corrupting system when I'd format cache and vice versa.
So I then took the risk and flashed @mn.code's instead and it definitely works. So he'll have to tell us what the trick is with the math. In better news, there's no need to mess around with tar's like we thought, even if @Ziyan doesn't go back to file OTA-style flashable zips for awhile, since TWRP's new(ish) Resize FS command resolves it. Run it after flashing the ROM then you have your space added back to flash GApps, etc. Really, we want addon.d support though, so hopefully ZMoD will be switched back anyway, but this is a simple workaround for now.
Edit 2: Updated with the latest. A 16gb Verizon i516 and a couple maguros with different emmc chips (non-VYL00M) would be appreciated now. Thanks everyone!
osm0sis said:
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Click to expand...
Click to collapse
Here's a dump from a toroplus - is this what you are looking for?
@osm0sis Mine is like yours, but 1 month senior:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011
@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way
musical_chairs said:
Here's a dump from a toroplus - is this what you are looking for?
Click to expand...
Click to collapse
bamtan2 said:
@osm0sis Mine is like yours, but 1 month senior:
Code:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011
Click to expand...
Click to collapse
Thanks guys! I've updated my previous PIT structure post with your info. Now I just need a couple non-VYL00M EMMC chip devices.
mn.code said:
@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way
Click to expand...
Click to collapse
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!
osm0sis said:
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!
Click to expand...
Click to collapse
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.
mn.code said:
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.
Click to expand...
Click to collapse
I modified my dump using the values from yours and the MD5 matches yours, so I think the latest PIT Magic seems to write the userdata value correctly; the issue I had was that I didn't change cache's block size since I didn't understand that was a sum of the previous partition's values yet. Thanks! :good:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
With a universal solution still in mind, leaving userdata untouched for the various device variants and only adjusting system and cache, this afternoon I've whipped up the following:
/tmp/pitfix.sh:
Code:
#!/sbin/sh
# PIT Updater for Galaxy Nexus
# osm0sis @ xda-developers
#
# Alters cache to the preset length in MB, adding the difference to system
# new cache block count (length) in MB
cacheMB=128;
# block device containing the PIT
block=/dev/block/mmcblk0;
# offset of the PIT in the block device, and partition size offsets within the PIT; count offsets are size offset + 4
pit=17408;
sysOffset=1500;
cacheOffset=1632;
dataOffset=1764;
# functions for dumping then converting the little endian hexadecimal into decimal for calculations and converting then writing back
dump_hex2dec() { echo "$(printf '%d\n' 0x$(dd ibs=1 count=4 skip=$(($1+pit)) obs=1 if=$block 2>/dev/null | od -hv | head -n 1 | awk '{ print $3$2 }'))"; }
write_dec2hex() { echo -ne "$(printf '%08x\n' $2 | sed 's/.\{2\}/& /g' | awk '{ print "\\x"$4"\\x"$3"\\x"$2"\\x"$1 }')" | dd obs=1 count=4 seek=$(($1+pit)) of=$block; }
# create a backup of the existing, untouched PIT with a timestamped filename
dd if=$block of=/sdcard/backup-$(date +%Y-%m-%d.%H%M%S).pit bs=1 count=4096 skip=$pit;
# check the PIT against known values for userdata to ensure it is an original PIT for the device
dataSize=`dump_hex2dec $dataOffset`;
dataCount=`dump_hex2dec $((dataOffset+4))`;
if [ "$dataSize" != 2379776 ]; then
echo "Non-standard userdata size (start offset) detected... aborting."; exit 1;
fi;
case $dataCount in
28397534|59944926) ;;
*) echo "Non-standard userdata count (length) detected... aborting."; exit 1;;
esac;
# calculate new block counts (lengths) for cache and system; 1 MB is 2048 blocks
sysCount=`dump_hex2dec $((sysOffset+4))`;
cacheCount=`dump_hex2dec $((cacheOffset+4))`;
cacheCountNew=$((cacheMB*2048));
sysCountNew=$((sysCount+cacheCount-cacheCountNew));
# calculate new block size (start offset) for cache; the sum of the previous partition (system)'s block size and block count
sysSize=`dump_hex2dec $sysOffset`;
cacheSizeNew=$((sysSize+sysCountNew));
# abort if cache has already been altered
if [ "$cacheCount" == "$cacheCountNew" ]; then
echo "cache count (length) already matches target amount... aborting."; exit 1;
fi;
# write back the updated values to the PIT in the block device
write_dec2hex $cacheOffset $cacheSizeNew;
write_dec2hex $((cacheOffset+4)) $cacheCountNew;
write_dec2hex $((sysOffset+4)) $sysCountNew;
# zero minor gpt/efs areas of the PIT to mimic external editing and flashing
for i in 200 596 632; do
echo -ne "\x00\x00\x00" | dd obs=1 count=3 seek=$((i+pit)) of=$block;
done;
# format the altered partitions to their new lengths; userdata remains untouched
#make_ext4fs /system;
#make_ext4fs /cache;
# repair and resize all partitions as a final measure to ensure data integrity
#e2fsck /system;
#resize2fs /system;
#e2fsck /cache;
#resize2fs /cache;
#e2fsck /data;
#resize2fs /data;
echo "Success!";
exit 0;
This is pretty much the sum of all our knowledge thus far. I'm still working out the format commands so they're commented out, but otherwise it should work, though I'd caution against trying it just yet. It needs to be run from recovery, but I haven't decided what I want the delivery method to be (as a shell script update-binary, run by another shell script update-binary, or run by an EDIFY updater-script), beyond a flashable zip. Thoughts? Comments? Fixes?
Edit: Fixed the dump_hex2dec function to work in recovery. I also noticed in my function testing that even though the 15222 MB userdata block count value was passed to Odin using @mn.code's -osm0sisV2 PIT above, the value actually written (verified by dump) was the correct, lower one. I'm guessing Odin knows best so I've removed the higher one as an accepted value from the script. :good:
Edit 2: I've now thoroughly tested it by running it manually in recovery after adb pushing it to /tmp but I've run into a problem with recovery not acknowledging the new sizing in make_ext4fs, or resize2fs. I have dumped the adjusted PIT and it looks flawless, all the values were written correctly and nothing else was touched (verified in a hex editor). fastboot formats the partitions to the correct new sizes, but then recovery acts like they're corrupt, showing them as 0 until you format, at which point they go back to being their old sizes.
Edit 3: Odin flashing the dump of my adjusted PIT and then fastboot formatting resulted in recovery recognizing everything properly again. So there must be some voodoo we're not aware of that Odin does after flashing the PIT. Any ideas?
If all else fails, I guess I can just make a flashable zip to dump the original PIT as a backup, copy the backup file, adjust the values in the file only, then instruct a user to Odin flash their personalized PIT, but that's not nearly as fun or useful.
Edit 4: Add zeroing of minor gpt/efs areas as per this post. Still not registering in recovery though, as above.
osm0sis said:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
Click to expand...
Click to collapse
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?
mn.code said:
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?
Click to expand...
Click to collapse
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.
osm0sis said:
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.
Click to expand...
Click to collapse
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?
mn.code said:
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?
Click to expand...
Click to collapse
Official, which has worked perfectly for everything on my end when a working PIT is flashed via Odin, resizefs, format and everything. I was more hoping for your ideas on what Odin could be doing after flashing that I'm not when I edit the PIT directly.
Related
Hey Devs,
As you may have noticed, I recently released a ROM with 276 MB free space on /data.
I accomplished this by using the MTD hack discovered by firerat and lbcoder.
It changes the kernel cmdline and includes addresses where the partitions are located.
This way, we are able to change the sizes.
That's exactly what I did.
The bad thing is that you need a recovery with a cmdline, which matches the ROM's boot.img's one!
So, a user needs to flash a recovery first and can then flash the ROM after a reboot into the new recovery.
Now, if the user wants to change his ROM to something else, he will need to revert the recovery.
It would be a lot easier if some other developers here would be interested in the MTD hack, so there is no need to revert.
I hope that some other developers will jump on the bandwagon now.
Here is the command I use for creating boot.img's, with modified partition sizes:
Code:
mkbootimg --kernel kernel --ramdisk ramdisk --cmdline 'mtdparts=msm_nand:[email protected](misc),[email protected](recovery),[email protected](boot),[email protected](system),[email protected](cache),[email protected](userdata)' -o boot.img --base 0x19200000
This reduces the /cache partition to 4 MB and makes /data as big as possible (the space which is left).
Now, the Market would normally fail to download huge APKs, because /cache is so small.
The problem can be easily solved by symlinking /cache to /data/cache and mounting the native cache partition on /dev/cache.
This can be done even without modifying the ramdisk, just do it in a script which runs before the Android frameworks start, like a2sd scripts.
Here are the necessary commands:
Code:
mount -o rw,remount /
umount /cache
rm -rf /cache
mkdir /data/cache
ln -s /data/cache /cache
mkdir /dev/cache
mount /dev/block/mtdblock4 /dev/cache
chmod 777 /data/cache
mount -o ro,remount /
Regarding the recovery:
Klothius from the Sapphire section created an update.zip which automatically patches the current recovery to use the provided cmdline.
You can find it here and include it in your instructions:
http://www.megaupload.com/?d=FY6CBAEE
I hope this little documentation will help you and will make you interested in doing this for your ROM.
Nice work. We'll look at implementing this into the next FroydVillain release. My only reservation would be keeping a wee bit more space on /cache than 4MB since our OTA app uses it. We could always hack it to use a dir on /data I suppose.
Hacre said:
Nice work. We'll look at implementing this into the next FroydVillain release. My only reservation would be keeping a wee bit more space on /cache than 4MB since our OTA app uses it. We could always hack it to use a dir on /data I suppose.
Click to expand...
Click to collapse
Yea, that would be better.. The whole purpose of this thread is to make things more compatible, means same cmdline Also you wll need to write the instructions for the recovery to /dev/cache then, keep that in mind
I will also add the RecoveryMod and an explanation in the documentation now.
I think the limitation of needing to switch recovery when changing ROMs is quite bothersome. Is there anyway we can get a recovery which supports both modes?
erasmux said:
I think the limitation of needing to switch recovery when changing ROMs is quite bothersome. Is there anyway we can get a recovery which supports both modes?
Click to expand...
Click to collapse
Sure, it can be done, but that would take a looot of work!
maxisma said:
Sure, it can be done, but that would take a looot of work!
Click to expand...
Click to collapse
More than I think it would be worth, to be honest. It'd make more sense for all the ROM developers to pull together on this and just make it a fact that we're giving more space to data, since it also eliminates the need to create fancy hacks moving dalvik to cache or to sdcard or splitting it between 2-3 file systems to cater for people who like to have the entire Android Market on their phone.
This has my/Villain's full support Maxisma, it makes perfect sense. Might I suggest you PM the other ROM developers here, or at least the popular ones like Fusion etc, and refer them to this thread so we can all work together on it.
Now that it's safe to assume that HTC have pretty much ditched the Hero, it makes sense for us to fix/improve -everything- including partition layout. There is a lot of wasted space in /cache that only gets used if a ROM developer starts fudging about with where dalvik lives, but /cache isn't quite big enough to completely migrate dalvik cache to for people who like to have a lot of apps.
Is there any more work on this? :d
shinyfong said:
Is there any more work on this? :d
Click to expand...
Click to collapse
I believe maxisma stopped work on it due to it breaking something or causing something not to work. I think he mentioned it in his Rom thread...
pulser_g2 said:
I believe maxisma stopped work on it due to it breaking something or causing something not to work. I think he mentioned it in his Rom thread...
Click to expand...
Click to collapse
It was me who broke it ;-) I accdidentally booted with a different mtd layout and thus killed all my data.
This method is perfectly fine though.
But I switched to a Nexus and I don't have a Hero anymore.
Since I installed JPK on a test phone after claims that the RFS related lag has been fixed.. well.. I'd like to share my impressions:
- I have installed approx 100 apps after the flash (flashed / wiped / factory reset btw)
A this point, the phone while slightly slower than one with Voodoo is pretty fast
A few days later, it's still good enough.
- 5 days later, lag starts to be felt, and it's stronger every next day until it becomes annoying to use
That's exactly the same experience as I had on various Eclair roms without any lag-fix. I also had this on JP3 (early Froyo).
So yeah. I looked into the /init binary from samsung and it's supposed to make some file system checks from time to time when you restart the phone, but it does not appear to really carry on with that.
So I ran the checks myself:
- you need a rooted phone, and adb or a terminal
- find all RFS partitions:
$ su
# mount|grep -i rfs
- kill all processes, go flight mode and remount them read-only
$ su (if you're not root.. not going to repeat it again for subsequent commands)
# kill -9 <pid> of anything that use the patrition
# mount -oremount,ro /dev/block/....
- check the RFS filesystem and correct errors
# /system/bin/fsck_msdos -p -f /dev/block/....
Surprise, a million of RFS errors fixed such as:
/path/to/file starts with free cluster FIXED
Cluster XXX continues with cluster XXX in FAT 0 but is marked free in FAT 1 FIXED
Detected cluster chain loop head XXXX for p XXXX FIXED
FSNext block XXX is correct NumClusters XXX FIXED (weird one)
No LOST.DIR FIXED
Lost cluster chain at cluster XXX YY clusters lost FIXED (prolly lost files/data here!)
Repeat for each partition
reboot the phone at the end
SURPRISE! It doesn't lag much anymore.
Use the phone an hour, do the check again and you will see it's already full of errors.
Conclusion:
RFS is bugged (we knew that didn't we) but it looks like it's fixable, if ever Samsung figures out what is corrupting the file system exactly (it's closed source so we can't really find out easily)
It might be possible to figure it out by looking at VFAT sources too.
I would be very interested to see if that fixes the lag for everyone or if i'm an isolated case and all my RFS partitions are on bad hardware or if it's really software corruption as I am guessing
Small disclaimer:
im not responsible for any data loss etc. no warranties etc. fixing file system even readonly might cause data loss due to the bugs in RFS. you've been warned lol.
My God,
I think a lot of eyes will be on the BOUNTY
Could someone just create an app that does this?
bilboa1 said:
Since I installed JPK on a test phone after claims that the RFS related lag has been fixed.. well.. I'd like to share my impressions:
- I have installed approx 100 apps after the flash (flashed / wiped / factory reset btw)
A this point, the phone while slightly slower than one with Voodoo is pretty fast
A few days later, it's still good enough.
- 5 days later, lag starts to be felt, and it's stronger every next day until it becomes annoying to use
That's exactly the same experience as I had on various Eclair roms without any lag-fix. I also had this on JP3 (early Froyo).
So yeah. I looked into the /init binary from samsung and it's supposed to make some file system checks from time to time when you restart the phone, but it does not appear to really carry on with that.
So I ran the checks myself:
- you need a rooted phone, and adb or a terminal
- find all RFS partitions:
$ su
# mount|grep -i rfs
- kill all processes, go flight mode and remount them read-only
$ su (if you're not root.. not going to repeat it again for subsequent commands)
# kill -9 <pid> of anything that use the patrition
# mount -oremount,ro /dev/block/....
- check the RFS filesystem and correct errors
# /system/bin/fsck_msdos -p -f /dev/block/....
Surprise, a million of RFS errors fixed such as:
/path/to/file starts with free cluster FIXED
Cluster XXX continues with cluster XXX in FAT 0 but is marked free in FAT 1 FIXED
Detected cluster chain loop head XXXX for p XXXX FIXED
FSNext block XXX is correct NumClusters XXX FIXED (weird one)
No LOST.DIR FIXED
Lost cluster chain at cluster XXX YY clusters lost FIXED (prolly lost files/data here!)
Repeat for each partition
reboot the phone at the end
SURPRISE! It doesn't lag much anymore.
Use the phone an hour, do the check again and you will see it's already full of errors.
Conclusion:
RFS is bugged (we knew that didn't we) but it looks like it's fixable, if ever Samsung figures out what is corrupting the file system exactly (it's closed source so we can't really find out easily)
It might be possible to figure it out by looking at VFAT sources too.
I would be very interested to see if that fixes the lag for everyone or if i'm an isolated case and all my RFS partitions are on bad hardware or if it's really software corruption as I am guessing
Small disclaimer:
im not responsible for any data loss etc. no warranties etc. fixing file system even readonly might cause data loss due to the bugs in RFS. you've been warned lol.
Click to expand...
Click to collapse
VOLD is what does the RFS check at boot. It seems to run, although maybe it fails? It definitely does run though, you can check it running using 'ps' on phone boot. Maybe it's only cleaning up /sdcard though.
At any rate, running the disk check does help with speed, but it doesn't help that much. It's still slow. I think if you want to stick with RFS, you need to do a defrag as well as the filesystem check. After some use, RFS must be very very fragemented on most people's phones.
Even in perfect condition though, RFS still has some very nasty properties such as locking the entire disk when a write occurs, not doing buffering, etc etc.
INeedYourHelp said:
Could someone just create an app that does this?
Click to expand...
Click to collapse
An app can't do this, since the app would have to be running off RFS and would crash/have to be killed to perform the FS checks. It could be done on boot using some trickery and the playlogos / replace binary trick. Or it can be done by replacing the init script with some kernel hackery. But at that point, you might as well just use a real filesystem.
I guess a PC .bat file could be made that uses adb to do this for you and then reboot the phone... Doesn't seem worth the trouble though!
RFS is journelled. You sure the filesystem doesn't do the journaling properly?
Nice job figuring this out. Is someone forwarding all these findings to that Samsung dev?
It's not about RFS vs the other filesystems. I'm well aware of the performance of RFS. But it's decent enough when it's working properly. It's not nearly as fast as ext but fast enough that you don't get annoyed.
Thus fixing RFS would make the life of many people who aren't technically inclined better. If the RFS do get all corrupted everywhere, and Samsung figure that out and fixes it, it means a good thing for most people.
The rest of us will end up on ext anyways
And yeah I think the fscheck at boot fails, it must fail actually, i don't see how else it would happen.
bilboa1 said:
It's not about RFS vs the other filesystems. I'm well aware of the performance of RFS. But it's decent enough when it's working properly. It's not nearly as fast as ext but fast enough that you don't get annoyed.
Thus fixing RFS would make the life of many people who aren't technically inclined better. If the RFS do get all corrupted everywhere, and Samsung figure that out and fixes it, it means a good thing for most people.
The rest of us will end up on ext anyways
And yeah I think the fscheck at boot fails, it must fail actually, i don't see how else it would happen.
Click to expand...
Click to collapse
What firmware did you test on, btw? I've noticed that JPK does seem to take longer on the FS checks, so maybe they have it fixed already (doubt it though)?
RyanZA said:
What firmware did you test on, btw? I've noticed that JPK does seem to take longer on the FS checks, so maybe they have it fixed already (doubt it though)?
Click to expand...
Click to collapse
on JPK actually
im going to flash JM8 to see if its the same in fact, but i expect so
RyanZA said:
VOLD is what does the RFS check at boot. It seems to run, although maybe it fails? It definitely does run though, you can check it running using 'ps' on phone boot. Maybe it's only cleaning up /sdcard though.
Click to expand...
Click to collapse
I think VOLD only does the checks for the Internal SD and External SD, not the RFS partitions.
Interesting find though, about the FS errors.
I’ve been wondering why the lag appears after a couple of days. I suspected corruption in one way or another myself, as it stays after a reboot it could not have been RAM and there are no signs of running out of space. Thanks for your research and I hope it will lead to new fixes. Sad but true my fix is to reflash my phone almost weekly.
I run Doc's JPK super slim ROM which is really nice but still it lags, even with OCLF installed.
Yesterday I take my phone out of my pocket to take a quick photo. My phone wakes up and I sweep the glass. Halfway through the sweeping the animation stops. I wait patiently for a second or three and there my home screen is. (No widgets, no animations, just a single home screen with 12 icons on it of the applications I actually use). I click on the camera icon and I wait another 5 seconds for the camera to realize it is not supposed to sleep on duty. I make a photo, the actual moment is already gone by now but hey I have the thing in my hand. It just takes another 5 seconds to store the photo.
It is like being in a hurry with a toddler with you. You want to go quicker but you can’t get angry cause the little thing just can’t go faster. It has to stop and wonder about life every once in a while.
I like my phone and I am sure it will grow up.
Very interesting findings! Sure hope Samsung sees this or this is forwarded to some Samsung techs.
Maybe move this to the development forum then it might get more traffic from people that can actually help.
borchgrevink said:
Very interesting findings! Sure hope Samsung sees this or this is forwarded to some Samsung techs.
Click to expand...
Click to collapse
do you really think Samsung is reading every thread here on XDA?
Sent from my GT-I9000 using XDA App
matty___ said:
do you really think Samsung is reading every thread here on XDA?
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
No need to be uppity my friend ;-)
Some further info on possible RFS fixes here:
http://forum.xda-developers.com/showpost.php?p=8445244&postcount=143
With the sync issue fixed, and corruption fixed with checks, RFS might just work 'okay'! It won't be as fast as EXT, but it should still work fine if we can sort out all the bugs! And we have a much better chance of getting Samsung to include these fixes in a new firmware too.
If we could get hold of Samsung somehow, and manage to convince them that they should
1) disable the always sync behaviour
2) do full filesystem checks at boot up (or provide a utility to do the checks)
then RFS should be A LOT more usable!
Could someone perhaps turn this into a .bat-file?
RyanZA said:
Some further info on possible RFS fixes here:
http://forum.xda-developers.com/showpost.php?p=8445244&postcount=143
With the sync issue fixed, and corruption fixed with checks, RFS might just work 'okay'! It won't be as fast as EXT, but it should still work fine if we can sort out all the bugs! And we have a much better chance of getting Samsung to include these fixes in a new firmware too.
If we could get hold of Samsung somehow, and manage to convince them that they should
1) disable the always sync behaviour
2) do full filesystem checks at boot up (or provide a utility to do the checks)
then RFS should be A LOT more usable!
Click to expand...
Click to collapse
That quite right, I noticed the sync issue also.
As for the corruption, 2 things should be fixed:
- fsck from init should actually fsck the partitions properly on boot (we can sort of fix that by calling it in a script again)
- corruption should in theory not even happen so Samsung would have to work on RFS for that one
I ran JM8 for 2 days now and my rfs partitions are full of errors, like in JPK, just for confirmation if there needed to be one.
In fact it's still running fsck on /system as I'm writting and issues are filling the terminal .. its been running for 30s already lol
woeds said:
Could someone perhaps turn this into a .bat-file?
Click to expand...
Click to collapse
I haven't tried that but it might be easier to enable adb over usb (in development settings) then .. make sure you are rooted, type that:
> adb shell
$ su
#
<prompt on the phone for root, click allow>
reboot into recovery
> adb shell
$ su
# mount -oremount,ro /dev/block/stl9
# mount -oremount,ro /dev/block/stl10
# mount -oremount,ro /dev/block/stl11
<for safety i'm not including stl3 it's the EFS)
if there's any error due to "filesystem busy" them stop there, it means it doesn't work
otherwise:
#/system/bin/fsck_msdos -p -f /dev/block/stl9
#/system/bin/fsck_msdos -p -f /dev/block/stl10
#/system/bin/fsck_msdos -p -f /dev/block/stl11
# reboot
I've tested this and it works. I only have a class 2 SDcard, so my testing shows it's really slow, a faster card would probably help.
With just this class 2 SD card, the boot-from-sd process is really only useful to see if the ROM being tested will straight-up brick your device or not.
Edit!
A class 6 card works fabulous! I'm posting from a tweaked version of mmarz's port of ath3nos' port of cm7 running from my brand-new class 6 card right now.
This process is hacked together from multiple other devices' howtos here on xda, sorry I don't know who to credit for the bits and parts. All of it is pretty generic, actually, and might well be applicable to other devices when tried as a whole.
For the moment, I'm only posting a brief skeleton how-to without specific walkthroughs for the steps, and I don't really want to upload many files until more of the bugs are swatted.
The following info should be enough for you, the savvy dev, to put the ROM of your choice on your SD card and boot it (or watch it fail to boot) without risk of bricking your device!
noobs, don't you dare, bricking is always a risk if you don't know your way around fastboot and adb!
Of course, savvy dev or not, a fresh nandroid backup is MANDATORY before hacking at your phone like this.
Standard disclaimer:
There is always a possibility of bricking your phone when messing with adb and fastboot.
If you're not willing to take that risk, don't try this at home (or at work, or school....)
Here goes:
The basic plan I followed was:
1) make and format 3 extra ext2 partitions after the default fat32 on the SD card, in this order: data (at least 180Mb) , system (170Mb to match stock), and cache (102Mb stock).
2) split up the boot.img from your chosen ROM so you can mod the ramdisk.
edit: turns out this next step in the quote is not required, there is an easier way.
A nifty command called devwait for init.rc
3) compile a modified init, adding a "pause (5);" after the ANDROID text. It goes on line 569 in the gingerbread init.c from a recent repo of google source.
use the newly compiled init in place of the init from the ramdisk.
since getting the android source uses so much time and bandwidth, I'm being nice and attaching a modified gingerbread init. If you test it with a froyo ROM and it doesn't work, don't blame me. If it does work with froyo ROMS, let me know!
Click to expand...
Click to collapse
3) change your init.rc from the ramdisk as follows:
replace the "mount yaffs2 [email protected] /system" , cache, and data lines with
Code:
devwait /dev/block/mmcblk0p3
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime
mount ext2 /dev/block/mmcblk0p3 /system nodev noatime nodiratime ro
devwait /dev/block/mmcblk0p2
mount ext2 /dev/block/mmcblk0p2 /data nodev nosuid noatime nodiratime
devwait /dev/block/mmcblk0p4
mount ext2 /dev/block/mmcblk0p4 /cache nodev nosuid noatime nodiratime
be sure to comment out any other mounts which go to /system anywhere, like the "mount squashfs [email protected]/system/blahblah/blah.sqf /blah/blah" lines in the aospCmod init.rc
4) repack your boot.img, with the correct cmdline ("mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" works,) etc.
5) prepare the ext2 /system partition with your target ROM /system files.
How to load the ROM:
steps a) to f) done on your PC. step g) is done on the phone.
a) Unzip the ROM on your PC, to get at the files to copy to the new /system directory in adb, and allow you to modify the updater-script.
b) Mod that ./META-INF/com/google/android/updater-script as follows:
As an example, I removed the following from the aospCmodOV-5-16-11 updater-script:
everything except the "ui_print" "symlink" and "set_perm" lines, including the "unmount" line at the end of the script.
I thinned it down because the update-binary wants to write to MTD partition for /system, and I didn't want or need that to happen.
I did need it to install the symlinks and permissions, though.
Without those, the keyboards kept FCing, and the phone couldn't connect to the network.
c) rezip the ROM with the modified updater-script.
d) boot phone into recovery and connect to PC with cable.
e) adb push or otherwise copy the modified ROM.zip to your sdcard.
f) next, copy /system from the unzipped ROM to the new partition with adb
Code:
adb shell
mount -t ext2 /dev/block/mmcblk0p3 /system
exit
adb push /path/to/unzipped/ROM/system /system
g) on phone, in recovery: install .zip from sdcard
select the new ROM.zip you reassembled and pushed to the SD card
select yes, you really want to install the .zip
this should write the /system symlinks and permissions to the new /system partition.
Click to expand...
Click to collapse
6) back on your PC, still connected to phone with cable in recovery;
Code:
adb reboot bootloader
fastboot boot /path/to/boot.img
Step 6 is important!
this will boot from your modified boot.img without actually flashing it into the phone's NAND, so a reboot (or restart after battery pull in case of issues) just goes to your regular installed ROM in the phone.
That's about it for now. It's pretty much a hack as yet, but it's cool in a geeky way to be able to do this.
<reserved for future use>
Thanks! This should be very helpful to devs and those of us who are adventurous with our phones!
Very interesting concept, I'd love to see how many uses this could have!
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
bigsupersquid said:
bump...
with a class 6 card, ath3nos/blarf cm7 runs great off sd.
even feels a little snappier than running from the phone mtd partitions.
I must benchmark soon to verify.
I'm really surprised no-one seems interested, I guess this is too old-hat, like a debian chroot
Click to expand...
Click to collapse
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
I personally still have problems recompiling a ramdisk.
mmarz said:
This is good work, but I think people require a bit more hand holding. How about publishing some of the modded files?
Click to expand...
Click to collapse
I can post files one rom at a time, because the ramdisk and kernel will be a bit different for each rom, and so will the permission/symlink updater-script.
first, I'll have to get rom-dev permission, because they get modded, and also cause it's nice to have permission.
I personally still have problems recompiling a ramdisk.
Click to expand...
Click to collapse
fastboot provides a fantastic shortcut for that, if you cpio-gzip the ramdisk together after editing it to your specs
from http://android-dls.com/wiki/index.php
in a command line linux shell, from the directory containing your unpacked ramdisk,
Code:
find . | cpio -o -H newc | gzip > ../newramdisk.cpio.gz
will put newramdisk.cpio.gz in the directory above the one containing your ramdisk
Click to expand...
Click to collapse
fastboot can make the boot.img for you and either dump it into the phone (assuming your shell current directory contains your kernel (zImage) and ramdisk (newramdisk.cpio.gz))
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" flash:raw boot zImage newramdisk.cpio.gz
or boot your phone from the assembled image without actually flashing it in.
Code:
fastboot -c "mem=477M console=ttyMSM2,115200n8 androidboot.hardware=thunderc" boot zImage newramdisk.cpio.gz
those fastboot lines assume an optimus v (or s.)
looking at all the great work you've posted on the forums, I'm surprised you have trouble handling anything to do with these phones!
I'm slowly working up to a modded recovery to install-to-sd directly and avoid all this hacking.
Nice, thanks for that. I had no idea fastboot could do that.
mmarz said:
Nice, thanks for that. I had no idea fastboot could do that.
Click to expand...
Click to collapse
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
bigsupersquid said:
happy to share.
I only learned fastboot could do that when I seriously started in kernel testing after finally getting one to compile without errors. I didn't feel like bricking my phone to test the fresh kernel, and researching fastboot showed me its capabilities... I could've just typed fastboot with no arguments and that makes it list its parameters.
it's really helpful for this mod, because you can just reboot into your stock rom without having to reflash anything.
plus it's nice to just tweak the ramdisk and let fastboot do the work making a new boot.img for you when testing weird stuff like this.
Click to expand...
Click to collapse
Very cool i loved this idea when it was executed in a way over at Ubuntdroid for the intercept when i was doing alpha testing. Very convenient for many of the scarier tests when i wanted to keep my existing system intact. The big difference was that on the intercept the bootable partitions were written into the kernel. Downside you couldn't use it to test kernels but could switch kernels from the phone and flip-flop between completely isolated systems on the go without a computer. Which was great when i wanted to test a new ROM on the way to work but i needed my phone when i got there and had to have a painless method of switching to my usual running system. So i'd flash a "non SD" kernel and i was back up and running in just a minute or so.
I'm excited to play with this when i have some free time.
Thanks dude!
you could easily use an update.zip for each of the sd-boot and regular phone boot.img files and swap between working systems with those in recovery.
before getting the fast sd card, I was looking at this more as for testing foreign roms than two-system operation, but now I see that would work too.
for a second-system option I was looking at a market-enabled rom on sd since I don't use g-apps on my daily driver.
note: the GB rom I'm running on won't 'see' a card with more than four partitions properly, and won't mount the vfat partition with vold, although you can manually mount and access the first four partitions including the vfat. android just won't admit it's there. and I couldn't get at a fifth partition through the phone os, even though my pc could see it fine (to get 5 partitions I had to use an extended partition to contain most of the virtual partitions)
android could read virtual partitions in an extended partition just fine as long as the total number on the card didn't exceed four.
I learned a neat trick today.
going through the init.rc from the stock rom, I noticed a section at the beginning which was labeled 'on emmc' which would imply running from a card if the device hardware lacked nand flash memory.
I'd read on xda while researching this concept of boot-from-sd about an unspecified wait-for-device command in init.rc, but most of the info I've gotten on init scripting language has come from picking apart init scripts, as google has not documented the available commands very well.
long story short, I tried using devwait with the regular init and it worked just fine to mount the ext partitions. yay!
first post edited to reflect the improved method.
I've also been testing a recovery mod. if recovery.fstab is modified in the recovery ramdisk (or a fakeflash recovery) then the recovery will copy a tweaked rom.zip to the ext /system partition with some minimal modifications to the updater-script and a few commands in the adb shell.
the recovery with the modded recovery.fstab can also back up from and restore to the sdcard ext /data and /system partitions.
I will update the first post with more specific instructions as I hammer them into something consistent.
soon I'll be ready to release a modded fakeflash recovery to make experimentation easier, as well.
All Kaiser owners complain at one time or another about size limitations to the /data partition.
I had asked around about increasing the /data partition to a maximum, keeping the /system partition to a bare minimum.
Well, I figured out a way to maximize your /data partition and still give the /system partition some "wiggle room".
The size of your "androidinstall.tar" file is the key here.
Depending on how it is constructed, you can easily figure out how much room you'll need for the /system partition in one of two ways.
1. Build your kernel with ATools as usual. Give it "normal" settings for now because you'll be changing and installing the kernel once more. Install your build and let it finish. Once it does, open up "Terminal Emulator", usually found as part of the "Dev Tools" app.
In Terminal Emulator you'll need to type "df -h" without the quotes. This detects the filesystem partitions and gives it a "human" number, hence the argument "-h".
You are looking for "dev/block/mtdblock2" and the numbers following.
If you are using the "system_froyo_us_odex" file from SF, then your mtdblock2 "should" read 90.7MB. Other builds are larger or smaller, this is why we're looking at mtdblock2.
You will also notice mtdblock3... this is your /data partition.
Once you have your numbers in MB, you can go back and build your kernel based on the size of your /system partition as installed on the phone, releasing as much room as possible for /data.
2. Going back to the "androidinstall.tar" file, most recently Devs have been combining the "app" folder inside the system folder. In the past, the data folder was separate from the system folder and gave the install a "bloated" size, appearing larger than what actually ends up in the mtdblock2 partition.
Most of these builds are built this way, with a few exceptions.
"Packed" size of "system_froyo_us_odex" comes out to around 87MB as a .tar file, relatively close, but you can't be sure unless the file is unpacked to a new folder.
When I unpack mine, it comes out to 90.7MB.
See where this is going?
I go to ATools and build a kernel based on the actual size of the /system file, plus 1 or 2%. (wiggle room)
I build the kernel with 92MB /system and 10MB for /cache, giving me 112MB for /data.
As long and drawn out as it may seem, I strongly suggest option 1. This will give you the best judge of size vs. kernel build size and is a more accurate measurement.
I have gotten it smaller than that once I stripped KingShui's rom to the bare essentials, around 52mb.
I'm just trying to share my knowledge is all.
Hello.
Normally Im able to fix any issues just by searching and reading, but I'm not able to find answers to what I'm looking for. I have a HTC Desire Z, bough in Hong Kong. Also I'd like to point out that only recently I started exploring android system, however I am relatively knowledgeable around linux and PC's in general.
Phone has been working fine for nearly 3 years (running factory 2.2 and then 2.3 ROMs only - no root or other hacks - completly factory). In recent months my phone has been crashing more and more often to the point that it would freeze or crash several times a day and was barely usable. I decided to do format it and put some custom rom on it. I used guides on this forums to root and install 4ext (also tried CW). I tried few roms, some would not boot at all (stuck on boot screen), some would boot but also crash. Upon further investigation I believe the internal storage on the phone is damaged/corrupted. I did format the partitions several times to no avail. I have fun e2fsck, which sometimes shows bad block, sometimes not....
To prove that internal storage is corrupted, I do the following. Boot to 4ext, using adb I mount internal /data partition. I copy a large file from sdcard to /data partition. Once the /data is about 30-40% full, copy will fail with different error messages - segmentation fault, unable to write, and some other random ones. Then /data is inaccessible until i reboot and mount it again. I can format and repeat the procedure - same result. Same applies to /system partition, with the exception that it is about first 90% ok and only last 10% is damaged (will result in error if I try to copy data there)
Oddly enough, factory 2.2 rom works to some extent - because it's small and makes minimal use of /data (about 70MB), It will boot and essentially phone will work for a short while, as long as there's no need for using data partition - restoring sms messages or contacts would fail before completed, etc. Eventually it will crash once, twice, and eventually wont boot at all.
Anyway, I'm looking for answers for following questions:
1. e2fsck application that comes with 4ext and CWM does not actually perform check for bad blocks. It says it does, but it doesn't - it happens too quick. I think what it does is that it only verifies existing bad blocks. There should be another binary that comes with it called "badblocks". This executable does that actual write/read search of bad blocks, but this is not included. According to documentation of e2fsck theres a special parameter to force write/read badbblock detection of entire disk, but the e2fsck version does not support it. It appears that the "android" version of e2fsck is cut down and not the same as linux version. Is there a way to perform a proper bad block search on android ?
2. better solution would be to completely change ROM to use sdcard storage only, and do not use /system.or /data partitions at all (/dev/block/mmcblk0p25 and /dev/block/mmcblk0p26), and create those partitions on sdcard. knowing linux I'm pretty sure this is entirely possible, however I do not know how much work is involved to phone, boot partition and rom - I'm also not too familiar with android limitations. I cant find any related information, but maybe somebody can point me in the right direction ?
3. any other possibilities to make any use of this phone ? It's still a good phone, good screen, works fine, etc, its just that internal memory is corrupted hence the thing is useless at the moment.
Note this is the internal memory that is corrupted, not the SDcard.
Thanks in advance,
First thing i would do is check which emmc i have, there are two (that i know of) put in these phones and one is prone to failure
Code:
adb shell
cat /sys/devices/platform/msm_sdcc.2/mmc_host/mmc0/mmc0:0001/name
output will five you either SEM04G or M4G2DE, the M4G2DE is the bad one
or from fastboot run
Code:
fastboot oem check_emmc_mid
samsung = prone to failure and sandisk = not prone to failure
for running e2fsck i would boot to recovery (so you can unmount partitions) and run this
Code:
umount /system
umount /data
umount /cache
adb shell
e2fsck -n /dev/block/mmcblk0p25
e2fsck -n /dev/block/mmcblk0p26
e2fsck -n /dev/block/mmcblk0p27
also maybe play around a bit, check THIS OUT for an idea of other commands to try
I do think the fastboot command may also be of some help here, in fastboot try
Code:
fastboot oem rbchk
or
Code:
fastboot oem partition_test all
also im not sure what method you used to format your partitions, i would consider flashing one of these PC10IMG.zips i created, but be careful and read the post. highly recommend using fastboot rather than bootloader to flash it. and i do believe if you did not root with the gfree method to gain true radio s-off it could be dangerous.
also THIS is a great tool for complete wipe/format whatever partitions, read the thread for more info
as for booting from sd card, this is kinda a last result but very do able
Great Work by Catheral
if you need more help feel free to ask, its nice to see someone who has done a bit of research before asking questions so i applaud you for this as it is evident that you did which makes me more the willing to help :good:
Good Luck!
Wow, thanks for amazing reply.
To be hones I didn't expected much but this blew me away in terms of content and quality.
Im at work so I cant test everything at the moment, however so far I have did some of the tests you recommended:
1. I do have M4G2DE emmc. This is the bad one.... no surprise here.
2. I have fun e2fsck back to back with all sort of different options. I have even used it to mark large section of partition as bad and tried copying files to it -> fail
3. "fastboot oem rbchk" produced output, after which the phone froze. No response to adb/phone buttons. Removed battery and proceeded to #4
Code:
... INFOrbchk: scan blocks from 163841 to -1
INFOmsm_nand_block_isbad: unsupported block address, 0x1C205320
INFOmsm_nand_block_isbad: unsupported block address, 0x1114C80
INFOmsm_nand_block_isbad: unsupported block address, 0x32529900
INFOmsm_nand_block_isbad: unsupported block address, 0x17439260
INFOmsm_nand_block_isbad: unsupported block address, 0x2D75D840
INFOmsm_nand_block_isbad: unsupported block address, 0x1266D1A0
INFOmsm_nand_block_isbad: unsupported block address, 0x28991780
INFOmsm_nand_block_isbad: unsupported block address, 0xD8A10E0
INFOmsm_nand_block_isbad: unsupported block address, 0x23BC56C0
INFOmsm_nand_block_isbad: unsupported block address, 0x8AD5020
INFOmsm_nand_block_isbad: unsupported block address, 0x1EDF9600
INFOmsm_nand_block_isbad: unsupported block address, 0x3D08F60
INFOmsm_nand_block_isbad: unsupported block address, 0x1A02D540
INFOmsm_nand_block_isbad: unsupported block address, 0x30351B20
INFOmsm_nand_block_isbad: unsupported block address, 0x15261480
... some more similar lines here
INFOmsm_nand_block_isbad: unsupported block address, 0x2B585A60
4. ran "fastboot oem partition_test all" output: (I dont think it's doing what you intended)
Code:
... INFOTesting Partition[merge_emmc]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[misc]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[hboot]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[sp1]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[wifi]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[recovery]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[boot]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[mfg]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[sp2]...
INFOCan not allocate heap for flash_block_test
INFOTesting Partition[pdata]...
INFOCan not allocate heap for flash_block_test
OKAY
5. I have not tried your " PC10IMG.zips" as you suggest, however I did confirmed I have radio s-off, as per your thread.
6. I really want to try "G2/DZ AIO wipe tool". I have downloaded it but I cant see any install/usage instructions anywhere ?
7. CyanogenMod 7.1 booting from SDCARD is awesome !. Does exactly what I wanted. Only few things I need to find out now: how do I make it always boot from sdcard ? currently, once i boot this rom, then when i go reboot, it boots to the rom installed in the internal memory. Is there a way to force it to boo from sdcard every time ?
Unfortunetly to the best of my knowledge you'll will shortly have far to many bad blocks and multiple corrupt partitions, I have yet to see anything succesfully bring this chip back to life, if your talented at soldering (very fine soldering) there are suppliers who will send you a new emmc allready loaded.
The aio wipe tool is to be flashed like a rom within recovery, it has a great aroma installer to choose which wipe\formats to perform, bear in mind it will take 10 or more minuets to complete depending on what you choose
Kbeezie also created a modded version of catherals boot from sd mod, id say check it out (find in developer threads) it has a few more features (can't remember off the top of my head) unfortunetly I don't believe apply at boot is one of them
Good luck to you, and feel free to ask any further questions
Sent from my RubiX NonSense using xda app-developers app
Update on the issue.
I guess the good thing is that in my case, is that factory 2.2 rom does load and works (to some extent).
Catheralls modded CM7.1 ROm to boot from sdcard using app called "BootManager" and some other tricks.
This app allows you to MOD "any" custom roms to run from SDcard. Trouble is that when I install it on my stock 2.2 rom, the app does not work correctly. Mucking around a bit I was able to swap phone boot image with the one from catherall modded ROM. As a result the phone boots automatically from SD, and behaves just like it was in the internal OS.
There's a bit of trickery required to use recovery mode though. Once booted to recovery, /etc/fstab needs to be edited to mount the partitions from the sdcard instead of internal ones. Once I did that I was able to load gapps and they function just fine. I suspect I could even try loading different ROMS, with the added step where I would have to edit the boot image to reference correct partitions (not sure how to do that yet). If boot partition is damaged, there might be other ways to boot the rom from sdcard - adb, recovery, bootloader.
But otherwise my phone is working fine. CM7.1 mod is working very well and smooth - so far I have not found any issues and performance is great (mind you I have overclocked the CPU). The only drawback is that its not possible to mount sdcard as external drive in windows (as OS partitions are locking it).
I'll keep digging and update if I find something worthwhile.
Much appreciate the help. My plan is to keep this phone for few more months and once S4 comes down in price I'll get one.
Thanks.
Right on! If you are able to come with something truley special this community would love it! Let's say perminent booting from sd with full playstore access and multiple roms, you could start a thread in the dev section explaining how to do it, this way anyone with multiple bad blocks or fried emmc could benfit, great job!
Sent from my RubiX NonSense using xda app-developers app