Develop Bugs

[UTIL][OUTDATED] One Click Lag Fix 2.0

[size=+2]This version is now deprecated in favor of the 2.0 APK version. Please see this link: http://forum.xda-developers.com/showthread.php?t=784691[/size]
This version is not recommended for use any longer.
Details about what this fix does:
Creates a VIRTUAL EXT2 filesystem inside the stock RFS filesystem on the internal SD card, with a 4KB block size. This means that this lag fix creates a buffer between the real filesystem and the android system. This buffer should reduce the amount of disk I/O required for all operations by utilizing EXT2 buffering, as well as not writing file access times to disk, etc.
Folders that are currently supported:
/data/data
/data/system
/data/dalvik-cache
/data/app
/data/app-private as a symlink to /data/app/app-private
/dbdata/databases is not supported. It appears to be ROM backed, and can cause problems if overwritten.
Benefits over version 1
1.5GB of application data available, with no data loss.
e2fsck of the EXT2 partition on each boot.
Correct busybox version included! YES!
/app and /app-private directory included in the fix for faster application installs.
/dbdata/databases included in the fix, expected to give a big performance boost for apps that use it.
Mounts instead of symlinks for extra performance as OS does less work (about 100 or so more quadrant).
Benefits over other lag fixes
Open source, with full comments and ease of editing.
Works on any and all firmware versions, including any yet unreleased froyo versions (that don't change file structure).
Credits
Big thanks to mimocan for putting us all on the right track in how to sort out lag problems!
Big thanks to ykk_five for showing us all how well loopback filesystem mounting works!
Big thanks to cyanogen mod for e2fsck
Requirements for One Click Lag Fix 2.0
Rooted phone - http://www.addictivetips.com/mobile...t-samsung-galaxy-s-i9000-with-a-single-click/
Windows computer with SGS Drivers (Samsung Kies), or the ability to read through the batch file and run it yourself.
(Beta Release) The ability to reflash your device if something goes wrong.
No other lag fix installed. If you installed One Click Lag Fix 1.0, then use the uninstall function which came with that lag fix before running this lag fix. (Untested but assumed to be working, please help out here.)
1.5GB of freespace on Internal SD Card for swap files while the fix is working (/sdcard).
"Internal phone storage" in Settings->SD Card must read greater than 500mb (0.5GB) of free space.
How to run One Click Lag Fix 2.0
Place your phone into USB debugging mode: Settings->Applications->Development
Attach your device to your computer. Do not mount the drives.
Download the attached ZIP file.
Unzip to a folder of your choice.
Double click "lagfixme.bat".
Don't double click "unlagfixme.bat".
Wait for it to complete.
You will need your phone to be unlocked when it runs the script, so that you can accept the permissions request that will appear on your device.
How to remove One Click Lag Fix 2.0
Double click "unlagfixme.bat"
Wait for it to complete.
Known Issues For All Versions
Some custom firmwares use up all available space in /system. This fix requires some libraries to be placed in /system/lib. These libraries are used to create the filesystem properly, and to check it for errors on every boot.
If your firmware does not have the available space (around 1mb) in /system, do not use this fix! Your phone will not boot and will have to be restored from backup / reflashed.
Current known firmwares with this issue: None yet. Please provide the firmware version+mods if you encounter this issue. It will show up as an out of space error in the log, under 'Copying libraries'.
Known Issues 2-1, 2-2
Paid apps from the market have issues.
Google maps and other pre-installed ROM-backed applications have issues.
2-3 Changes from 2-2
/dbdata has been removed. This fixes maps issues.
/app-private is now a symlink to /app/app-private. This fixes paid apps issues.
Alternate installation methods for similar fixes
Tayutama has made an update.zip version that is easy to install - http://forum.xda-developers.com/showpost.php?p=7632258&postcount=208
Chainfire has a .NET version of this fix with some nice features - http://forum.xda-developers.com/showthread.php?t=751513

Frequently Asked Questions
Q: My apps are force closing while this fix is running, and I can't use my phone!
A: By design. The script has backed up/copied your apps over to the internal SD card (remember the requirement about 1.5GB of free space on the SD card?). It is now overwriting them with a 1.5GB file. As the file overwrites an app that is trying to do something, it will probably force close. This is normal.
Closing all running apps, and removing widgets before running this fix can make the process much smoother, though.
Q: The script can't transfer files to my phone / The script can't run / Help help I'm dieing!
A: Read the first post again.
Q: My paid apps from the market don't show up.
A: I will hopefully have a fix for this sooner or later. Hold tight! It's in the known issues. I don't have access to paid apps, so I can't test this.
Q: Does this need busybox?
A: No, busybox is included.
Q: I only have 200mb of free space now! What gives?
A: The lag fix has made a 1.5GB file, and is storing all of the data inside there. The side effect is that the free space meter is now incorrect. Sorry, this can't be helped.
You can check real free space by using ADB like this:
Code:
adb shell
su
busybox df -h
Q: When I use a backup tool, the backup is now 1.5GB big! It's taking forever!
A: The backup tool isn't designed to work with this fix. It will work, it just won't work well. Hopefully this fix will be short lived, and either Samsung will give us a new update, or someone will give us a good custom firmware that can natively mount what we need, where we need it. Or someone might come out with a better backup manager. Until then, we suffer.
Q: Will a reflash wipe this fix?
A: Yes, a reflash will wipe everything this fix did.
Q: Can this brick my phone?
A: If you know how to get to the download mode from power off (hint: volumedown+home+power), then almost nothing short of throwing your phone off a tall building can actually brick it. If you can't do this though, or don't know someone who can, then you're better off waiting for samsung to release a fix. Anything that moves files around on your device has the potential to break things, and this fix has no QA department.
Q: Why is /dbdata not included in your fix, but other people have included it?
A: Other people have included it in the same way my 2-2 fix includes it. However, /dbdata is ROM backed. This means that the real files are on ROM, and only the changes appear in the /dbdata folder. When copying or moving files from this folder, you would need to specify each folder by exact name to ensure that it was copied across, and each firmware can have their own names. (This is because RFS wildcard will not catch an unused ROM backed file.) In some cases, you can get lucky and have this work perfectly because you have already used all the files in /dbdata. There is no fail safe method to do this though, and /dbdata does not make a big difference to performance. (It is already on NAND flash.) If you want to try your luck, v2-2 is still available.
Q: Why does this lag fix work? Is it slowly destroying my phone?
A: Let's say an application counts from 1 to 10, and writes the value each time to disk.
Stock:
1 -> App tells RFS to write 1 to disk -> RFS writes 1 to disk -> RFS writes journal saying to changed the value on the disk.
2 -> App tells RFS to write 2 to disk -> RFS writes 2 to disk -> RFS writes journal saying to changed the value on the disk.
..
9 -> App tells RFS to write 9 to disk -> RFS writes 9 to disk -> RFS writes journal saying to changed the value on the disk.
10 -> App tells RFS to write 10 to disk -> RFS writes 10 to disk -> RFS writes journal saying it changed the value on the disk.
Total physical disk writes: 20. Speed: SLOW! Wear and tear on disk: HIGH!
Lag Fix:
1 -> App tells EXT2 to write 1 to disk -> EXT2 stores 1 in RAM.
2 -> App tells EXT2 to write 2 to disk -> EXT2 stores 2 in RAM.
..
9 -> App tells EXT2 to write 9 to disk -> EXT2 stores 9 in RAM.
10 -> App tells EXT2 to write 10 to disk -> EXT2 stores 10 in RAM.
..
EXT2 tells RFS to write 10 to disk -> RFS writes 10 to disk -> RFS writes journal saying it changed the value on the disk.
Total physical disk writes: 2. Speed: FAST! Wear and tear on disk: LOW!
This isn't exactly what is happening, but it gives you the general idea.
Q: Can this mod work on other Android devices? Would we see a performance boost on them as well? If not why is it limited to the Galaxy S?
A: SGS has very very good hardware, but it has some parts of it's hardware poorly implemented. The filesystem that samsung chose to use is custom-built using FAT32 as a base, RFS. It has a lot of the problems that FAT32 has, and should have been left back in the 90s, or even the 80s.
One of the big issues with it is how it handles multiple requests - it blocks. It blocks everything. When your mail app wants to read the mail you just tried to view, but your twitter app is busy writing a new tweat it just received, your mail app is forced to wait.
This is bad, but it could be worse! And it is... your twitter app didn't just get one tweat, it got 50 tweaks. It is busy writing the tweats one by one to the filesystem. This would be fine, since all modern filesystems will buffer writes, so instead of writing each one at a time, they will batch them together and write it as a big chunk. Uh oh - RFS does no buffering at all! After each write, it will also write an update to the grafted-on journal system. Guess what happened to your mail you were trying to view while all this happened? It 'lagged' and you got a black screen for half a second, before the mail popped into view.
Luckily the hardware on the device is so good that you usually don't even notice the problem until you have a lot of apps running, all writing their updates when you unlock the phone.
This is mostly speculation based on experiments done on RFS -- RFS is closed source, and we have no idea if the problems are just badly set settings (such as a block size that is too small), coding bugs in the implementation, or if RFS is just really that badly designed.
This fix just grafts a buffer on top of the RFS filesystem, using a very very simple and fast filesystem, EXT2. It fixes most of the issues by writing to RFS as seldom as possible.
So no, this fix won't fix other devices, since they're already running quite close to maximum speed. The SGS at stock is running nowhere near maximum speed, and this lag fix takes it a bit closer. You could probably speed up other devices by tweaking the filesystem settings to give them a big buffer or similar, but it isn't really needed. (I haven't actually tried to put an EXT2 onto any other Android phone, as I don't have any other Android phone, so this is just speculation.)
Q: My phone is fast now!
A: Yeah.

Oh, awesome. I managed to post this in the wrong forum. Doh!
Could a moderator please move this to Android Dev sub forum?

haha looks awesome dude but you are correct, wrong section im gonna try it now, will report in 5 mins!

A few people in the other thread said they had the first lag fix working on the captivate. Anything that might change that with this release? I came to download the old one, and here a new one is

abra-cadabra...
*POOF*
Done!

E_man5112 said:
A few people in the other thread said they had the first lag fix working on the captivate. Anything that might change that with this release? I came to download the old one, and here a new one is
Click to expand...
Click to collapse
The old one is still here. Nothing has changed that should stop it working on the captivate though, but it is completely untested. Use the 1.0 (which has had a lot of testing) until this one has been put through the paces.

sirphunkee said:
abra-cadabra...
*POOF*
Done!
Click to expand...
Click to collapse
Thanks.

RyanZA said:
The old one is still here. Nothing has changed that should stop it working on the captivate though, but it is completely untested. Use the 1.0 (which has had a lot of testing) until this one has been put through the paces.
Click to expand...
Click to collapse
E_man5112 said:
A few people in the other thread said they had the first lag fix working on the captivate. Anything that might change that with this release? I came to download the old one, and here a new one is
Click to expand...
Click to collapse
I'm running the manual ext2 fix with Q scores 2000+. I'll try 2.0 Util tonight and post feedback....

Someone got some benchmarks?
Maybe for xxjp3? I only reach 1900 in quadrant.. /:
Sent from my GT-I9000 using Tapatalk

gonna test report in 5minutes

RyanZA said:
The old one is still here. Nothing has changed that should stop it working on the captivate though, but it is completely untested. Use the 1.0 (which has had a lot of testing) until this one has been put through the paces.
Click to expand...
Click to collapse
I'm in the process of flashing back to stock on my Captivate so I can give this a legit test. I'll report back here and let you know how it goes. Assuming my Captivate doesn't melt I'll update my OP from the thread about the original fix in the Captivate section!

About what lag is this?
Sent from my HTC Desire using XDA App

Dominik06 said:
Someone got some benchmarks?
Maybe for xxjp3? I only reach 1900 in quadrant.. /:
Sent from my GT-I9000 using Tapatalk
Click to expand...
Click to collapse
Benchmarks are roughly the same as 1.0. You can expect about 100 points more on average though, at least on my phone.

I just removed the V1 version and finished the V2.
It finally finished and rebooted the phone. now it tells me my internal memory storage is full. None of my Widgets will load.
Looking i see only 14MB free in internal storage.
thoughts?

Zilch25 said:
I'm in the process of flashing back to stock on my Captivate so I can give this a legit test. I'll report back here and let you know how it goes. Assuming my Captivate doesn't melt I'll update my OP from the thread about the original fix in the Captivate section!
Click to expand...
Click to collapse
Just as long as you point out that this fix isn't heavily tested yet!

RyanZA said:
Just as long as you point out that this fix isn't heavily tested yet!
Click to expand...
Click to collapse
Oh I know, I'm going to emphasize that heavily, lord knows I don't want to get stuck on tech support all night

clubtech said:
I just removed the V1 version and finished the V2.
It finally finished and rebooted the phone. now it tells me my internal memory storage is full. None of my Widgets will load.
Looking i see only 14MB free in internal storage.
thoughts?
Click to expand...
Click to collapse
Thoughts? That's bad.
You should be seeing roughly 215mb free. Did you see any errors in the log at all?

Everything is working fine for me. First reboot after lagfix had some problems with downloading apps from market, but after a second reboot it got fixed.
"Biggest" problem now, is just the phone stating that my internal phone storage is too low, with an icon on notification bar that I cannot remove. Already deleted some .bak from /data folder, any more tips what I can delete to get rid from this message?
Regarding the fix. apps are indeed much more snappier, no lag on the system when installing apps from market, and overall if the phone would continue like this for the next 48 hours, lag fix form me is solved. I had some problems after some 24h, with some lag, even with the previous version of the lag fix.

clubtech said:
I just removed the V1 version and finished the V2.
It finally finished and rebooted the phone. now it tells me my internal memory storage is full. None of my Widgets will load.
Looking i see only 14MB free in internal storage.
thoughts?
Click to expand...
Click to collapse
Looking through my old V1 remove script, I believe I was leaving behind the .bak files! Nastyyy... I'll update V2 to remove those if they're there on install. Will hopefully clear up any problems.

Some New Information Regarding the KIN 2M Filesystem

I have been following the xda-developer website for awhile now, as I have a KIN 2m of my own and I have been interested in finding a way to root or replace the current OS with a new one. Since I haven't seen anymore progress with the phone itself, I decided to do some digging and see what I could do myself (I am a programmer and IT Specialist). What I found might be helpful.
According to other posts, and what is floating around the internet, the KIN 2m flash memory is based on the Samsung MoviNAND 8G architecture. The chip itself is the Samsung MoviNAND KLM8G4DEDD-B101 which supplies 8GB of flash memory to the phone. Since this is also a Flash based NAND memory, I decided to investigate into the actual filesystem (FS) which makes up the chip and which WindowsCE 6.0 is based.
Apparently, this particular NAND memory is based on the YAFFS2 (Yet Another Flash File System rev2.0) which supports both little- and big-endian (32 and 64-bit architecture and some 16-bit systems), respectively. Also, the operating systems that are built on this particular FS are WindowsCE, Android, Linux, pSOS, eCos, and ThreadX.
Going off what I had found, I discovered that with regard to WindowsCE in particular, there are four different parts to the KIN NAND set-up: A Portable YAFFS "Core", a YAFFS Direct Interface, the WindowsCE wrapper, and then of course the WindowsCE OS itself. For WindowsCE, the WindowsCE wrapper accesses the YAFFS Direct Interface, not the core directly. In order to write instructions to the NAND and the "core", a set of instructions in three different types are necessary. These types are a POSIX Application Interface, an RTOS Integration Interface, and finally Flash Configuration and Access Interface. I have attached (and pasted) below a diagram of the above description and I have attached a document which was provided by yaffs.net which also covers some of these details.
View attachment 1461518
The POSIX Application Interface allows execution of application code to access the filesystem. These commands that are executed are typically open, close, read, write, etc. The RTOS Integration Interface consists of functions which allows for YAFFS to access the RTOS system resources. The commands are things like lock, unlock, initialize, get time, set error, etc.. Finally, we have a Flash Configuration and Access Interface which allows YAFFS to access the NAND directly and it executes commands such as initialize, read chunk, erase block, etc.)
So this is what I have discovered so far, and I am currently working on seeing where I go from here. I am currently trying to mount my KIN and browse it as a YAFFS filesystem on my computer, and once I do that, I am thinking that I might be able to execute instructions to access the ROM and NAND chip. On the YAFFS.net website there are a lot of good documents on how this FS works and how commands are executed. I am currently trying to read all I can and see if there is anything I can do.
And the plus side is, after all of this digging and experimenting...my KIN still works!!

very interesting.
wouldn't instructions be executed on the ARM?
http://en.wikipedia.org/wiki/ARM_architecture#Instruction_set
the trick would just be getting something on there and running...
Edits:
actually we may have already found a way to execute. it involves XNA or Silverlight and its on these forums somewhere. now compiling something...
must note that this could probably end very poorly.
so how we write android to NAND?
http://source.android.com/
BOOM
http://infocenter.arm.com/help/topic/com.arm.doc.ddi0290g/DDI0290G_arm1156t2fs_r0p4_trm.pdf
if we can access a console on the actual kin device we can compile code. now, how exactly we could do that is a mystery to me. the kinOS is running on top of windows CE kernel, so there may be one in there...
that console would be "run23"
http://support.microsoft.com/kb/194302
http://developer.download.nvidia.com/tegra/docs/tegra_250_hw_setup.pdf
check out page 12. to put the kin in recovery mode, you hold u+s+b+power.
also relevant
http://forum.xda-developers.com/showthread.php?t=894130&page=2

BOOM
infocenter.arm.com/help/topic/com.arm.doc.ddi0290g/DDI0290G_arm1156t2fs_r0p4_trm.pdf
if we can access a console on the actual kin device we can compile code. now, how exactly we could do that is a mystery to me. the kinOS is running on top of windows CE kernel, so there may be one in there...
that console would be "run23"
support.microsoft.com/kb/194302
Click to expand...
Click to collapse
That is a good thought. However, according to what I have found, there is no reason to go looking for a console on the device itself with which to compile/execute code. What I have found, to date, is that because the device is a YAFFS filesystem, if we have the correct drivers installed on our host machine (which can be gained from the internet), we can mount the device as a drive and using Command Prompt on the host machine we can compile and execute the code from there. We can use the commands NVFlash, mount, write, flash_eraseall, etc.
Sorry if this doesn't seem to make an sense at the moment...I am trying to multitask, and I believe I am failing miserably. I will hopefully have more information later.

IT_Tech said:
we can mount the device as a drive and using Command Prompt on the host machine we can compile and execute the code from there. We can use the commands NVFlash, mount, write, flash_eraseall, etc.
Click to expand...
Click to collapse
ooh, that's a good one. i think JohnKussak was doing something like that using the NVidia tegra toolkit, but he was not able to connect for some reason.
http://forum.xda-developers.com/showthread.php?t=894130&page=2
now, from what i understand of YAFFS (which is admittedly very little) it's just a partition of the memory right? or is the YAFFS system on a completely separate piece of memory?
i was reading about the MPU (memory protection unit) in the ARM manual and it described the layout of memory. basically it supports up to 16 blocks, all with 32 bit addresses (4294967295 bytes = 4gb). since there's 8 gb of storage on the kin, it can probably be assumed that 2 of those blocks are used for storage. another block is probably for RAM (256mb). now, the YAFFS system has to be accessible to the ARM somehow (maybe), so there are several possibilities.
1) the YAFFS file system is on a partition of the 8gb storage space
2) the YAFFS file system is on a partition of the 256mb RAM (pretty sure this isn't the case)
3) the YAFFS file system has its own block of memory
regardless, the MPU can be disabled. when it's disabled, no permissions are checked (ever) and we can read/write anywhere we please, including the YAFFS, wherever it may be. i'm not totally sure this is necessary, but i know we've had problems accessing system folders in the past. i suspect disabling the MPU could fix that, if we could manage.
i also suspect that the YAFFS is accessible when the phone is in recovery mode (u+s+b+power), we just need the correct drivers to flash a Tegra 2600 APX chip. i believe i read on the tegra thread that somebody couldn't find that particular driver. it may need to be written.
edit:
just found this https://developer.nvidia.com/tegra-2-technical-reference-manual
you have to apply for access and it can take up to a month i guess. i'm working on that right now. the main item of interest is "16.0 NAND Flash Controller". i'm gonna try looking in some shady places and see if i can't dig that up...
double edit:
just realized that tegra 2 is different than tegra APX, NVidia does not offer an APX manual anymore, if they ever did.
triple edit:
ooooookay. http://viewsonic-gtablet-for-dummies.webs.com/nvflash.htm
just grabbed source for drivers. unfortunately, NVflash does not run on 64 bit systems, which is causing a bit of problems for me...

slimeq said:
now, from what i understand of YAFFS (which is admittedly very little) it's just a partition of the memory right? or is the YAFFS system on a completely separate piece of memory?
Click to expand...
Click to collapse
From what I understand, YAFFS is a partitioning system. It does not need to be stored as a separate system on a separate piece of memory. However, it does also have the capabilities to be partially RAM based... Which may end up confounding things. If you get a chance, you may have done this already I don't know, but read up on the YAFFS file system on the yaffs.net website--it has a ton of documents on how the system works and the commands it executes, its architecture, etc.
slimeq said:
i was reading about the MPU (memory protection unit) in the ARM manual and it described the layout of memory. basically it supports up to 16 blocks, all with 32 bit addresses (4294967295 bytes = 4gb). since there's 8 gb of storage on the kin, it can probably be assumed that 2 of those blocks are used for storage. another block is probably for RAM (256mb). now, the YAFFS system has to be accessible to the ARM somehow (maybe), so there are several possibilities.
1) the YAFFS file system is on a partition of the 8gb storage space
2) the YAFFS file system is on a partition of the 256mb RAM (pretty sure this isn't the case)
3) the YAFFS file system has its own block of memory
Click to expand...
Click to collapse
Let me begin here and differentiate between YAFFS types...there are two types of YAFFS formatting--YAFFS1 and YAFFS2. The KIN is formatted using YAFFS2 because it has 8g of space. Yaffs2 is different from Yaffs1 in the fact that it allows for memory sizes greater than 4GB because it supports 4KB pages rather than 512byte pages (Yaffs1). I will spare all the details as they are lengthy, but check out:
yaffs.net/documents/how-yaffs-works and yaffs.net/yaffs-original-specification (paying attention to Yaffs2 and how it relates to it foundation off of Yaffs1). Also, since Yaffs only uses a RAM based system for emulation purposes when the kernel is not being run on a true NAND (or NOR) storage, we can eliminate Option 2. Option 3, may be viable, but I am not sure as to how. Option 1 seems to make the most logical sense (pun intended), because of how the YDI (YAFFS Direct Interface) works with the YAFFS kernel and filesystem, as well as the WindowsCE Wrapper, etc.
Now as for the MPU. Yaffs has a built in code to handle MPU and a way to disable it through a console session on a host machine, but I don't remember where I read it, I believe it was on the yaffs.net website in one of the technical documents... I will work to remember where I found it.
I hope we can get some more people on-board helping with this YAFFS thing. If we could, it might make this go a bit faster, and have more heads working on it. Plus they might see something we don't.
P.S: These are the HARDEST captchas I have ever seen in my entire life!

kin
Hopefully you get access to the filesystem with this IT_Tech :fingers-crossed:

Every once in a while, it's fun to revisit Dev on the Kin
I remember going through the Tegra 2500 APX links on the wayback machine:
http://web.archive.org/web/20100813070722/http://www.nvidia.com/object/product_tegra_apx_us.html
If you click the Specifications tab, you can see that the 2600 and 2500 are virtually identical, aside from some video features.
Also, the ZuneHD is the only other product listed as using the 2500 chip. I don't have one, so I wouldn't know for sure, but I wonder if its drivers could be tweaked to allow access to the Kin, in the same way as the Zune. I remember trying to hack into other Windows Mobile drivers (for other WinMo 6.x devices I have) but never getting anything further, even when in other USB modes on the device.

Nandroid defrag

Hi
Before someone say it, yes, i know that in a nand doesnt matter if the files are or not fragmented. But i had read in some other places in the internet that it actually comes to a decrease in performance when its fragmented... If its a fact or not i dont know...
But i have a lot of apps that havent been updated in a while, and im concerned about my phone become slow...
So my question is:
if a make a nandroid backup and flash and restore (in case it become slow), this restore puts all the files defragmented? Or it just do an exact image of all the things including their location (some parts here other there...)?
Thanks

According to http://www.wizcode.com/articles/comments/flash_memory_fragmentation_myths_and_facts/:
Myth: Flash cards unlike hard drives do not have movable parts so defragmentation is useless.
On theory, since a flash drive does not have moving parts, its access time is independent on where the data is stored, which is why supposedly flash drives don't need defragmenting.
The fact is that flash memory is physically organized in blocks (or pages) of data, usually 128K or 256K large. Things get even worse from the fact that in order to change even one single byte, the entire page has to be first erased and then re-written with its contents again. In our example the time needed to change one byte of information is calculated the following way:
T = R + E + W
T is the total time, R is the time needed to read the entire flash page containing the byte we wish to change, E is the time required to erase the page and W is the time it takes for the data to be written back to the empty page. Not only we had to read 128KB in order to change a single byte but we also had to erase the entire block (which is very slow) and then write 128KB over again.
To complicate matters even further, it must be noted that there are additional layers between the flash card controller and the file system that cache pages being read and written to. The cache serves for improving performance. It is a simple trade-off between read/write performance and some RAM being used to cache the pages. It is most effective to read or write entire flash pages performance wise. When the operating system instructs the controller to read a particular sector on the card the cache normally retrieves the entire block and stores it internally. What this means is that information that is stored in a contiguous matter is more likely to be found in the cache than non-contiguous information.
Suppose we have a file that is 263892 bytes large and is fragmented. On a FAT32 file system using 1K cluster the file will occupy 260 clusters. In the worst case scenario the clusters will be dispersed across 260 different flash pages. If the file is contiguous all 260 clusters will fit inside 3 flash pages. Caching of the fragmented file will be impossible as it won't fit in the cache (260 pages will require 33MB of RAM to cache) while the defragmented file will fit in just 384K.
And finally the FAT file system stores folders the same way files are being stored - in cluster chains. A large folder that is fragmented and is not cached represents a huge performance penalty for standard file operations like listing the files in that folder, renaming or even deleting a file.
Conclusion: Fragmentation has a serious impact on flash card performance especially during write operations and when the file system is heavily fragmented across many different flash pages.
Click to expand...
Click to collapse
Now, about your question. It would seem logical, as at that time, every partition is written in order, thereby it should be defragmented. During normal usage, you change this here, change that there, thereby different blocks are edited, causing fragmentation.

^^
Hey thanks!!
I supose that this is why explore an album or images cause some lag.
Also, i had noted that not even putting all images in order (contiguous), the cache always creates a lot of chunks...

Using this thread...
I just formatted the sd-ext, and it appears in link2sd as ext2, but i had it as ext4.
Could it lead to problems? Do i need to reformat my sd card with an external reader? Or from the cwm?
Ive restored all my apps and they work, but im not sure...
Thanks

Linux on Android binding ./usr to a folder on extSdCard

Let me preface this by saying this is my first ever real post in a forum, so if im doing something wrong, please tell me. I would post this in the linux on android forum, but I am apparently not allowed to do that. Hopefully this helps some people out.
I have been experimenting with the ubuntu.sh script and various linux images. After getting an image to work I attempted to expand the image file by using dd. This would create a 12 gb (or whatever size) image, but on copying the file system from kali.img to the new .img, the new image failed to execute and ended up wiping my kali.img, my modified ubuntu.sh script and my new .img (my fault for rebooting with the images mounted). This and subsequent attempts all failed to create a 12 gb image that would run kali.
Before I go further, I should state that I'm mounting my images to /storage/extSdCard/mnt
So all this got me thinking on other ways to expand the file and I thought why not move ./usr to a folder on extSdCard and use mount to bind it to the kali.img file. That way you'd get about 2gb extra on the image and installed programs could be installed to space on the extSdCard. If this works it could be applied to the entire linux filesystem, and just use a small .img to mount everything to when you wanted to start it.
That being said I'm having difficulty implementing this. Kali will boot and I can execute commands from it after running:
busybox mount -o bind ../kali/usr/usr ./usr
And
busybox mount -o remount,exec ./usr
It would appear that things work, but I can't connect through vnc as vncserver isn't installed and I can't install things because of an error with dpkg: error setting ownership of ... Operation not permitted.
That is where I'm at right now. I figured I would make a post and hopefully get some more input on this, or maybe help someone out with the whole image file size thing.
Sent from my SM-N900V using xda app-developers app

I have made some progress, but this idea as a whole would appear to be impractical at this moment. The permission errors occur because of the exfat file system used on android devices. This implementation does work on ext(4) formatted sdcards, as tested by my linux box running backtrack. However, the android system, at least for the note3 does not easily support ext4 formatted anything, despite /system maybe being ext4. I haven't looked too far into that as it is read only and would also be a hassle to configure. This may become an effective means of running linux on android systems once ext4 is widely supported, but not likely until then.
If I were more skilled I would try to set up a raid system with ext4 image files, since exfat has a 4gb file size limit. Unfortunately though, I do not have the skill to do this at the moment nor the time to take on such a task.
I wish you all the best of luck with your projects.
Sent from my SM-N900V using xda app-developers app

Project: Galaxy Nexus Partition 2.0. GNP2!

Edit:
There has been no new information provided in this thread about how big /cache NEEDS to be. To me it seems like /cache is simply unused. It's there for Google to use with stock ROMs, which we do not use anymore. So redistributing storage from /cache to /system makes good sense to me. Exactly how much should be removed from /cache is up for debate, but if we take most of it away, that improves /system significantly, and still leaves ~100mb in /cache for contingency.
@osm0sis has contributed work which will allow an enthusiast to shrink their /cache and add that space to /system. Because of the way @Ziyan's ROMs are flashed, flashing a ROM will revert any change you make to /system back to the default size, until you "use 'resize FS' option in TWRP" according to @defy_owner. http://forum.xda-developers.com/gal...n-2-0-gnp2-t3147253/post64482777#post64482777
@osm0sis' flashable script UPDATE-GN.PIT.Editor.zip is an excellent bit of work contributed to the community of users and developers -- users can modify their devices themselves, and ROM developers now know better than ever how to make the switch to a bigger /system when necessary. Thank you ALL for your contributions to this thread!!!
Original post:
Per: http://forum.xda-developers.com/gal...eta-builds-t3060319/post60805347#post60805347 and http://forum.xda-developers.com/gal...eta-builds-t3060319/post61089590#post61089590 -- I'm an interested newbie and not a developer, but I will try to contribute where I can... firstly by starting this thread.
The /system and /cache partition sizes on Nexus devices have changed multiple times since Galaxy Nexus launched. By the time the 64bit Nexus 9 launched, /system had more than doubled in size! Luckily we don't need to worry about accommodating 64bit binaries. We also don't necessarily need to accommodate the giant Google Apps packages that Google includes, because Google provides most of those apps on the Play Store.
But we DO need more space if we want more FLEXIBILITY in the future. Flexibility to add features... flexibility to migrate to larger versions of Android... If we want Galaxy Nexus to survive to 2020, we need flexibility!
If the community developers for Galaxy Nexus can come to an agreement, we can all standardize on one partition layout to move forward with. That way, flashing back and forth between new OS distribuitions will be easy. So because I'm not a developer, I'll do my part by helping this discussion get started. I'll invite some developers to this thread who I've seen active. Anyone can of course participate. Then, when the smart people have said what they think, we can decide where to go from there. I'll begin by inviting: @Ziyan @bsmitty83 @MWisBest @Carlos_Manuel @zzpianoman @osm0sis @Fenix46 @theBlackEnd @sgspluss @aosp
The plan here is for you guys to debate with each other about this topic. To start the debate, I will start with an opening statement which is not necessarily my opinion. Then I will provide a list of helpful proposed tasks which volunteers can complete. I will then list a discussion agenda which you may or may-not want to talk about.
Opening statement: Galaxy Nexus partitions are out of date, but don't need much change (if any at all). But because F2FS on /data is faster and also requires a phone wipe, we should make both of these changes at once. The incremental hassle of doing one change on top of the other is minimal. The exact partition sizes don't matter. Because the Nexus 5 partition layout seems sufficient for future flexibility, we should just copy it. It was not designed for Android L or M like Nexus 6 was, but the Nexus 5 layout is close enough to "good enough".
Tasks v0.1:
0) What are the partition sizes of every Nexus device?
1) What are the partition sizes of popular 16GB NON-Nexus devices? Does Google have suggestive documentation for hardware developers?
Discussion Agenda v0.2:
-1)If F2FS is good for users, and a new partition layout is good for users, and BOTH require destructive migration of users' data, should we make both changes at the same time?
0) What are the obvious pros/cons of changing the partition layout? How do we balance the needs of the people who want LOTS of new space against those who need NO new space?
1) How long do we expect this size change to be sufficient for the community? 1 year or 10? Until Google stops supporting Nexus 5 or 6, or long after?
2) Decide how big /system should be. How much margin for future growth? Is there a rationale for any given size, or is choosing a round number ("1GB") as good as any?
3) Decide how big /cache should be. Is shrinking it a good idea? Would that saved space be large enough to matter? Is there risk of compatibility problems?
4) Reality check: Is it okay that we're thinking of shrinking the /data partition this much? What are the negative repercussions?
5) How do we maintain our development work and migrate all developers into the future? Do we maintain two different builds of the OS in parallel for some time? What timeline can we operate this way? 1 month or 1 year?
6) Does anyone absolutely OPPOSE attempting to standardize on a new partition layout for community development?
I invite any interested developer to speak about any topic. You are the people who really matter, because you volunteer to do the real work the rest of us need. Let's begin. Good luck to us all!

Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83

bamtan2 said:
Update 1: Adding a simple proposal which leaves /data untouched.
Premise: Nexus 6 has a smaller cache partition than Galaxy Nexus, which implies Google thinks we're wasting space on our /cache. Leaving /data untouched is very convenient. So instead of focusing on how large we want /system to be, we can focus on how small we want /cache to be, and then accept whatever benefit to /system that provides.
I do not have the exact numbers. But if Galaxy Nexus /cache is ~720MB, and Nexus 6 /cache is ~248MB, then that means we could add ~472MB to /system, which is a LOT.
Does anyone think this would be too much or too little? Do we need more /cache space than Nexus 6?
Can anyone find the exact numbers to compare what we would get from this change? I don't have a Nexus 6 to read the exact size of /cache.
We must be wary of taking away too much /cache space if we risk preventing new features in the future. For example: Moto X (2013) and Moto G (2013) both use F2FS, and F2FS may demand cache use. I am curious what their /cache partition sizes are.
CC @osm0sis @Ziyan @bsmitty83
Click to expand...
Click to collapse
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .

I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, and even that's probably just playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.

bsmitty83 said:
Gnex /cache is 432 mb , n6 on m preview has 256 mb . but I have far more apps on my n6 yet only 13 mb used , as opposed to 137 mb used on my gnex . so I think the cache size is fine . the n6 has triple the ram , double the cpu cores , and doesn't really have to cache data .
Personally I would wait until the time comes when a ROM is too big to fit on the current partition . I don't necessarily think its a bad idea at all , I just don't have a need to touch my partitions right now .
Click to expand...
Click to collapse
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
osm0sis said:
I've never seen the /cache partition on modern (>ICS) Android be used for anything but OTA temp storage, and the GN certainly isn't receiving any more OTAs so I agree we should at least take the overhead of 425.2-256 and add it to system. App caches are stored in /data/data, so I honestly don't think /cache needs any more than 128, even playing it safe.
I'll post on some of the other research I've been doing for this in my limited spare time, hopefully next week.
Click to expand...
Click to collapse
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...

bamtan2 said:
This is an excellent post! Exactly why we need to have this discussion! We want to leave no stone unturned. Your point about not changing until we need to is EXCELLENT. Sometimes we all need to put on the brakes and not fix what's not broken. PS: Of course I got the numbers wrong RIGHT after I had been looking at them *forehead smack* 200MB saved, not 400mb, of course.
In my experience the /cache has not really been used for anything. Now is a good time to collect information from all interested parties about how the cache can and will be used.
That is really informative, thank you!! I look forward to hearing more of what you've learned about /cache usage!
It sounds like we might be onto something here...
Click to expand...
Click to collapse
Galaxy Nexus: 654mb system, 432mb cache
Nexus 4: 840mb system, 560mb cache
Nexus 5: 1074mb system, 734mb cache
Nexus 6: 1946mb system, 248mb cache
Moto X (2013): 1611mb system, 1067mb cache
Moto G (2013): 1023mb system, 694mb cache
Until the N6, Nexus phone device /cache has been ~66.6% of /system. I think the N6's stance on it should be standard going forward however, especially for us, since as I mentioned in my previous post, /cache is only really used by OTAs in >ICS Android and the GN is certainly not receiving any more of those, making it mostly up for grabs. If it's still percentage-based, the N6 would make it only ~12.5%.
"Stock" PIT for our device: http://forum.xda-developers.com/galaxy-nexus/general/guide-restore-to-stock-unbrick-galaxy-t2065470
We can extract it to use as our starting point. I'm unsure whether we need a different PIT file for each variant (maguro vs. toro vs. toroplus), but I'll figure out the correct offsets for the command to extract the PIT from our emmc (a la this) and we can find out definitively.
@mn.code's 1gb system PIT: http://forum.xda-developers.com/showpost.php?p=60798125&postcount=663
A good example of a working modified PIT for the GN. I believe he reclaimed all of the added space from /data however, and he has still yet to provide his documentation on modifying and flashing the PIT file. Maybe he can provide it now in this thread to save us some time. :good:
Official ODIN download site: http://odindownloader.com/
PIT Magic: http://forum.xda-developers.com/showthread.php?t=1916936
Seems to be the defacto app for creating and modifying PIT files.
@zzpianoman's ZMoD 0518 tar: http://forum.xda-developers.com/showpost.php?p=60824237&postcount=704
Since mn reported ROM flashing reverted the partition size changes. Though this can maybe be worked around by flashing the new nightly, backing it up, formatting and then restoring. Better would be @Ziyan switching from block system.dat patching back to the old separate file-based zip format; Google hasn't even switched the newer (Nexus 4, 5 and 7) devices' OTA zips to the new block format, so there's no reason the GN needs to be on it, especially when it reverts our partition tinkering due to a specific partition size flag with which the system.dat has to be built. It would also allow addon.d to work again, which would be very handy with our new expanded /system partition.
This also solves the question of different builds, provided they merely remain small enough to fit on the base-size partition. With system.dat OTA-style out of the picture not even formatting should affect the size. At worst, GApps flashing would have to be sacrificed by those not willing to move forward if the standard ROM size kept increasing. As for the F2FS query, it never needs to be mandatory so I wouldn't worry about it. Some people seem to have some issues with it as well, but that could be an issue of devs using either too aggressive flags or not aggressive enough flags. F2FS is still pretty new for Android and I'm not sure anyone's really mastered the tweaks.
I expect increasing to a reasonable size like 1.2gb would allow for the proper expansion Lollipop has caused and M will likely also, as well as all the GApps one would want to cram on there. I'd imagine this will be sufficient until the hardware fails to support the latest Android versions or literally fails altogether.

@osm0sis I didn't have maguro last two months, I sold titanium colored one and now I bought a white one So, as promised, I will write a tutorial on how to flash new pit and how to untar the file @zzpianoman provided as soon as I get home from vacation. Also, I will make a new pit file (old one "borrowed" space only from /data), I will increase /system with another ~200MB (will take some from /cache and maybe some more from /data), so the /system partition will be around 1,2GB.

After examining a raw dump I did of the beginning of mmcblk0, the PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interestingly, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Edit 2: Fixed the command above. bs=1 when it's not a raw dump apparently. :good:

osm0sis said:
The PIT dumping command for our device is definitely:
Code:
dd if=/dev/block/mmcblk0 of=/sdcard/out.pit bs=8 count=4096 skip=17408;
My maguro dump is different from the maguro -0915.pit posted at the Stock PIT link I added above, so I'd definitely like to see a PIT dump from another maguro, a toro and a toroplus for me to compare. I'm going to open them up in PIT Magic and see what it registers the differences as next.
Edit: Interesting, the only difference was with userdata. "Block Count" was 28,397,534 with mine, and 31,174,656 with the supposed "Stock PIT". That certainly wouldn't account for 32gb vs. 16gb (which is something we will definitely have to accommodate), so my best guess is EMMC chip type?
Code:
d=/sys/class/block/mmcblk0/device;
echo "Name: `cat $d/name`\nManufacturer ID: `cat $d/manfid`\nFwRev: 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
I'm running everybody's favorite :
Code:
Name: VYL00M
Manufacturer ID: 0x000015
FwRev: 0x25
Date: 12/2011
So I'll want another VYL00M and non-VYL00M 16gb maguro to figure this out.
Click to expand...
Click to collapse
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice

bsmitty83 said:
I'll dump my 32gb toro when I get home in a little bit . most Toro's are 32gb anyways so it should suffice
Click to expand...
Click to collapse
Okay, here's what I've worked out:
PIT structure -
block sizes, 4 bytes (little endian) from offsets:
system=1500
cache=1632
userdata=1764
block counts, 4 bytes (little endian) from offsets:
system=1504
cache=1636
userdata=1768
blocks / 2048 * 1024 = kB
Code:
system = 1,339,392 = 669696 (654 MB)
cache = 884,736 = 442368 (432 MB)
userdata("stock") = 31,174,656 = 15587328 (15222 MB) [maguro, ?]
userdata(bamtan2) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250]
userdata(osm0sis) = 28,397,534 = 14198767 (13865.9833984375 MB) [maguro, GT-i9250M]
userdata(bsmitty83) = 59,944,926 = 29972463 (29269.9833984375 MB) [toro, SCH-i515]
userdata(musical_chairs) = 59,944,926 = 29972463 (29269.9833984375 MB) [toroplus, SPH-L700]
In all files but @mn.code's nothing changes but the block counts, even with the jump from 16gb to 32gb. He's changing block size on everything as well and I'm not sure that's something we should touch.. Maybe he can explain?
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Edit: I tried generating and flashing a PIT based on my own dump (seems like the safest approach to avoid exceeding chip capacity), simply by subtracting 304mb of blocks from cache and adding them to system (so cache at 128mb, and I also tried cache at 256mb), and I could not make it stable... Damn thing kept corrupting system when I'd format cache and vice versa.
So I then took the risk and flashed @mn.code's instead and it definitely works. So he'll have to tell us what the trick is with the math. In better news, there's no need to mess around with tar's like we thought, even if @Ziyan doesn't go back to file OTA-style flashable zips for awhile, since TWRP's new(ish) Resize FS command resolves it. Run it after flashing the ROM then you have your space added back to flash GApps, etc. Really, we want addon.d support though, so hopefully ZMoD will be switched back anyway, but this is a simple workaround for now.
Edit 2: Updated with the latest. A 16gb Verizon i516 and a couple maguros with different emmc chips (non-VYL00M) would be appreciated now. Thanks everyone!

osm0sis said:
I still need toro SCH-i516 (16gb Verizon), another couple maguros with different MMC chips, ideally one from the Play Store (maguro GT-i9250) too, and of course toroplus SPH-L700 (32gb Sprint) for a proper comparison.
Click to expand...
Click to collapse
Here's a dump from a toroplus - is this what you are looking for?

@osm0sis Mine is like yours, but 1 month senior:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011

@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way

musical_chairs said:
Here's a dump from a toroplus - is this what you are looking for?
Click to expand...
Click to collapse
bamtan2 said:
@osm0sis Mine is like yours, but 1 month senior:
Code:
1|[email protected]:/ $ su
blk0 of=/sdcard/out.pit bs=1 count=4096 skip=17408;
4096+0 records in
4096+0 records out
4096 bytes transferred in 0.442 secs (9266 bytes/sec)
[email protected]:/ # d=/sys/class/block/mmcblk0/device;
name`\nManufacturer ID: `cat $d/manfid`\nFwRev:
> 0x`cut -b 19,20 $d/cid`\nDate: `cat $d/date`";
Name: VYL00M
Manufacturer ID: 0x000015
FwRev:
0x25
Date: 11/2011
Click to expand...
Click to collapse
Thanks guys! I've updated my previous PIT structure post with your info. Now I just need a couple non-VYL00M EMMC chip devices.
mn.code said:
@osm0sis Your cache partition is getting corrupted because PITMagic, for some reason, is not saving new values for /userdata partition (so /cache and /userdata are overlapping). You have to change them manualy with hexedit after you change all other partitions (it's kinda tricky and a little pain in the a** ). Also, Block Size value of current partition must be the sum of Block Size and Block Count of the previous one (it is the point from which the current partition starts). New PIT is on the way
Click to expand...
Click to collapse
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!

osm0sis said:
Ahh nice! I'll give it another shot next chunk of free time I get. Can one of your new PIT files have /cache at only 128MB, /data untouched, and /system with the added ~304MB reclaimed from /cache? That's what I was going for so it'd be much appreciated for my testing purposes. Thanks!
Click to expand...
Click to collapse
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.

mn.code said:
Sure, will make one as soon as I catch some spare time
Here is the new pit, /system is 1280MB, /cache 248MB and the rest is /data.
Edit: Here is another pit, the one you requested, /cache is now 128MB, 304MB reclaimed to /system, /data untouched. INFO: if you used any of my pits before flashing this one, your /data might get corrupted and you won't be able to repair it with official twrp 2.8.7.0, you must use the one @Ziyan made (2.8.6.0 with f2fs support) and it will work fine.
Click to expand...
Click to collapse
I modified my dump using the values from yours and the MD5 matches yours, so I think the latest PIT Magic seems to write the userdata value correctly; the issue I had was that I didn't change cache's block size since I didn't understand that was a sum of the previous partition's values yet. Thanks! :good:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
With a universal solution still in mind, leaving userdata untouched for the various device variants and only adjusting system and cache, this afternoon I've whipped up the following:
/tmp/pitfix.sh:
Code:
#!/sbin/sh
# PIT Updater for Galaxy Nexus
# osm0sis @ xda-developers
#
# Alters cache to the preset length in MB, adding the difference to system
# new cache block count (length) in MB
cacheMB=128;
# block device containing the PIT
block=/dev/block/mmcblk0;
# offset of the PIT in the block device, and partition size offsets within the PIT; count offsets are size offset + 4
pit=17408;
sysOffset=1500;
cacheOffset=1632;
dataOffset=1764;
# functions for dumping then converting the little endian hexadecimal into decimal for calculations and converting then writing back
dump_hex2dec() { echo "$(printf '%d\n' 0x$(dd ibs=1 count=4 skip=$(($1+pit)) obs=1 if=$block 2>/dev/null | od -hv | head -n 1 | awk '{ print $3$2 }'))"; }
write_dec2hex() { echo -ne "$(printf '%08x\n' $2 | sed 's/.\{2\}/& /g' | awk '{ print "\\x"$4"\\x"$3"\\x"$2"\\x"$1 }')" | dd obs=1 count=4 seek=$(($1+pit)) of=$block; }
# create a backup of the existing, untouched PIT with a timestamped filename
dd if=$block of=/sdcard/backup-$(date +%Y-%m-%d.%H%M%S).pit bs=1 count=4096 skip=$pit;
# check the PIT against known values for userdata to ensure it is an original PIT for the device
dataSize=`dump_hex2dec $dataOffset`;
dataCount=`dump_hex2dec $((dataOffset+4))`;
if [ "$dataSize" != 2379776 ]; then
echo "Non-standard userdata size (start offset) detected... aborting."; exit 1;
fi;
case $dataCount in
28397534|59944926) ;;
*) echo "Non-standard userdata count (length) detected... aborting."; exit 1;;
esac;
# calculate new block counts (lengths) for cache and system; 1 MB is 2048 blocks
sysCount=`dump_hex2dec $((sysOffset+4))`;
cacheCount=`dump_hex2dec $((cacheOffset+4))`;
cacheCountNew=$((cacheMB*2048));
sysCountNew=$((sysCount+cacheCount-cacheCountNew));
# calculate new block size (start offset) for cache; the sum of the previous partition (system)'s block size and block count
sysSize=`dump_hex2dec $sysOffset`;
cacheSizeNew=$((sysSize+sysCountNew));
# abort if cache has already been altered
if [ "$cacheCount" == "$cacheCountNew" ]; then
echo "cache count (length) already matches target amount... aborting."; exit 1;
fi;
# write back the updated values to the PIT in the block device
write_dec2hex $cacheOffset $cacheSizeNew;
write_dec2hex $((cacheOffset+4)) $cacheCountNew;
write_dec2hex $((sysOffset+4)) $sysCountNew;
# zero minor gpt/efs areas of the PIT to mimic external editing and flashing
for i in 200 596 632; do
echo -ne "\x00\x00\x00" | dd obs=1 count=3 seek=$((i+pit)) of=$block;
done;
# format the altered partitions to their new lengths; userdata remains untouched
#make_ext4fs /system;
#make_ext4fs /cache;
# repair and resize all partitions as a final measure to ensure data integrity
#e2fsck /system;
#resize2fs /system;
#e2fsck /cache;
#resize2fs /cache;
#e2fsck /data;
#resize2fs /data;
echo "Success!";
exit 0;
This is pretty much the sum of all our knowledge thus far. I'm still working out the format commands so they're commented out, but otherwise it should work, though I'd caution against trying it just yet. It needs to be run from recovery, but I haven't decided what I want the delivery method to be (as a shell script update-binary, run by another shell script update-binary, or run by an EDIFY updater-script), beyond a flashable zip. Thoughts? Comments? Fixes?
Edit: Fixed the dump_hex2dec function to work in recovery. I also noticed in my function testing that even though the 15222 MB userdata block count value was passed to Odin using @mn.code's -osm0sisV2 PIT above, the value actually written (verified by dump) was the correct, lower one. I'm guessing Odin knows best so I've removed the higher one as an accepted value from the script. :good:
Edit 2: I've now thoroughly tested it by running it manually in recovery after adb pushing it to /tmp but I've run into a problem with recovery not acknowledging the new sizing in make_ext4fs, or resize2fs. I have dumped the adjusted PIT and it looks flawless, all the values were written correctly and nothing else was touched (verified in a hex editor). fastboot formats the partitions to the correct new sizes, but then recovery acts like they're corrupt, showing them as 0 until you format, at which point they go back to being their old sizes.
Edit 3: Odin flashing the dump of my adjusted PIT and then fastboot formatting resulted in recovery recognizing everything properly again. So there must be some voodoo we're not aware of that Odin does after flashing the PIT. Any ideas?
If all else fails, I guess I can just make a flashable zip to dump the original PIT as a backup, copy the backup file, adjust the values in the file only, then instruct a user to Odin flash their personalized PIT, but that's not nearly as fun or useful.
Edit 4: Add zeroing of minor gpt/efs areas as per this post. Still not registering in recovery though, as above.

osm0sis said:
Hmm, I also see that your PITs use the odd-one-out value for userdata count (=15222 MB), from the -0915 "Stock PIT" file. Any idea where that comes from? I'm still waiting on some non-VYL00M chip devices, but no maguro we've checked so far has that, and come with the slightly lower number I reported instead.
Good news is your PIT, and only stealing from cache, seems to work perfectly. dexopting seems to take the same amount of time and everything.
Click to expand...
Click to collapse
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?

mn.code said:
Well, my PITs use that value for userdata because I base them on -0915 "Stock PIT" file. I didn't dump mine before I edited it the first time, I just flashed that one since it came with stock firmware (Odin.stock.takju.jwr66y+4.3.zip)
When you flash my PIT, is your /userdata size changing?
Click to expand...
Click to collapse
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.

osm0sis said:
Answered that in my first edit above. I'm wondering if you have any insight into what I mention in my second and third edits as well.
Click to expand...
Click to collapse
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?

mn.code said:
Are you using official TWRP 2.8.7.0 or the one with f2fs that Ziyan made (2.8.6.0)?
Click to expand...
Click to collapse
Official, which has worked perfectly for everything on my end when a working PIT is flashed via Odin, resizefs, format and everything. I was more hoping for your ideas on what Odin could be doing after flashing that I'm not when I edit the PIT directly.