Related
Does anyone know how to make a complete memory dump (RAM) of the XDA, apart from the m? commands. These commands give a memory, hex, text breakdown. What i'm looking for is a command that get's 32MB of data and stores is as 32MB of DATA (the the invoking and storing the dump is obvious, so no tips on those please, i only want the commands get this raw memory dump).
Regards,
René
I just read up on some of the other stuff in here.
Somebody mentioned the XDA has 32MB of RAM "AND!!!" 32MB of flash ROM.
So, is this true? And if so. Does this mean that the organizer data is held in flash ROM allways? (this would in my opinion not be correct with some of the functions of the XDA, but OK). And if so..... How do i make a raw dump of this memory?
Cheears,
René
dumping ram
Hi,
I'm not aware of a means to dump RAM using the bootloader functions. However I'm considering developing something for it. It shouldn't be too hard with a bootloader patch like we developed to get around 5.17 limitations.
What I'm looking for is info on how data is organized in RAM such as the filesystem. The 32 RAM contains semi persistent data while the ROM contains what is needed to reinitialize the device and system binaries. However, as far as I understand, applications you install additionally and any application data is only stored in RAM. Anyone knows how the filesystem works on the lower level? I'm looking to implement a tool that can read such a RAM dump and make sense outof it.
Ok, so i came up with the idea of using the m? functions for dumping the data (what is the difference between those anyway?), into a huge file (you get 80bytes for each 16 bytes of data, so 160MB for a 32MB model and a hellufa lot of time, say 4 hours at 105200), and than parse these back to the original 32MB, based on the hex dump supplied.
As to your question, for which i'd like the answer myself, about the file system: as it's windows, i hope it will be some form of FAT or NTFS, but what i've seen so far, i'm afraid it's the system MS developed for storing their office documents....
Does anyone know if parts of the data are stored encoded/crunched/ encrypted?
Anonymous said:
As to your question, for which i'd like the answer myself, about the file system: as it's windows, i hope it will be some form of FAT or NTFS, but what i've seen so far, i'm afraid it's the system MS developed for storing their office documents....
Does anyone know if parts of the data are stored encoded/crunched/ encrypted?
Click to expand...
Click to collapse
I doubt if its FAT or NTFS, as it needs to handle having its size limitations changed dynamically as the device manages objectstore<->memory changes.
And its all stored compressed.
..Chuck..
Well, I recently managed to work out what RAM belongs to the file system, and dump that ram only. And within that RAM I've found some files, and their compressed data, and used the BinaryDecompress calls to decompress it and verify the contents.
However I still can't work out the overall structure that provides the pointers/identifiers to the location of the files, I just happened to find certain files by searching on the filename (which is stored uncompressed BTW).
Has anyone else been trying this?
..Chuck..
I am a newbie and just bought my first PDA phone - O2 Atom
Trying to learn as much about it so that I can fully utilize it.
Reading the posts in this site is my first step.
I know this may sound stupid to the experienced users but
I've been reading posts asking about unlocking the Ext. Rom
What is the purpose of doing this ? What are the additional things I can do if I can unlock my ext. Rom vs just leaving the device "as is" ?
Anyone care to enlighten me ?
When you unlock the ext-rom you can see what's in it? Also you can turn it writeable, so you can do anything you like with it, make a copy of the content first. As it it persistent storage on most devices the content won't be erased by hard-reset.
On my magician for instances the ext-rom is concatenated to the internal storage folder enlarging it from 7 to 27MB.
Regards, M
Basically, the ext (extended) rom includes extra settings like operator network settings, extra operator programs and operator visual customisations. Really, if you are new to this, it is best to leave as-is, as unlocking only allows you to edit the Ext rom, which you will probs not want to do yet.
Thanks all for the info. I guess I need to find out more before I try this.
There seems to be 3 kinds of memory on my device:
1. Storage
2. Memory Card
3. Device (which itself seems to be broken into 2 ? One for programs and another for program storage ?)
One of the reasons I asked this question is because I encountered a low memory warning but when I checked the Settings/memory 1 & 2 still showed lots but on the General tab it showed Storage as .5KB left not sure how to interprete the info.
The memory on your device is called RAM & is devided dynamically, by the OS, in a storage & a program part. When you install programs Via Act.sync & choose default they end up in ram, program files folder, this will cost precious space to run an application. So try to install to SD or storage folder. that's why BS is nice. You can use cabinstl to choose where you want to install when you install a cab directly on your magician.
Got the idea
You mentioned the operator blah blah.. does that means that i can unlock my simcard by unlocking the ext rom?
no extented rom and sim locks are 2 different things
Actually, oltp is a bit mistaking in his description of the memory regarding your specific phone.
The Atom is Windows Mobile 5 device.
On WM 5 phones there are two physical types of memory:
RAM and FLASH.
Unlike in older models, on WM 5 the RAM also called 'program memory' is only used to run programs, just like RAM in your PC.
The ROM or flash memory (slower, but isn't erased when power is lost) is divided in to 3 partitions:
ROM / OS - locked partition where the operating system resides. Only special tools like ROM upgrade utilities can write to this section.
Ext ROM - a locked partition where the operator / vendor of the phone stores some extras which are automatically installed after hard reset. As you already know you can unlock it and use as regular storage. Any data stored there will not be erased during hard reset.
Storage memory - this is the part of internal memory (flash) where you can install apps and store your documents and other files. Just like a PC hard drive.
...
Is there a sticky or a thread somewhere about editing ROM/Ext.Rom?
Thank you.
It's all in the WiKi!
I'm assuming this is associated with the recent phenomenon of hardware ID's changing everytime a new ROM is installed. Apparently MS uses the same hardware ID when it encrypts files on SD cards. What this means is you will lose all of your files on the SD card (including backup files) if you have encryption turned on, the files get encrypted, and then you switch ROM's.
So.. uh.. this is just a little warning, and it might be obvious to everyone but me... don't use SD encryption unless you know you're going to stick with a ROM.
I have no idea why MS doesn't just use the IMEI, but... they don't.
From what I read about the encryption, the key is generated after a hard reset, so basically you can't hard reset the device once data is encrypted.
Do you know wether there is an option to backup ones key to a file, save it to ones PC, and then reimport it once one has finished hardresetting the device?
If I were MS I'd see the vast usefullness of such an option and integrate it at once
the encryption key is created when you turn the Setting on...
and when flashing a new ROM or a HardReset the key is desteroid...
i am still yet to find the location... still looking...
Providing you remember, can't you just turn off the setting before a flash or hard reset and restore all the files to there unencrypted state?
Once the ROM has been flash and everything hard-reset you can just encrypt them again?
Percz said:
Providing you remember, can't you just turn off the setting before a flash or hard reset and restore all the files to there unencrypted state?
Once the ROM has been flash and everything hard-reset you can just encrypt them again?
Click to expand...
Click to collapse
No, because turning it off doesn't decrypt existing encrypted files. Just like turning it on doesn't encrypt the normal files. It will decrypt them as you open and resave them.
:-\
walshieau said:
the encryption key is created when you turn the Setting on...
and when flashing a new ROM or a HardReset the key is desteroid...
i am still yet to find the location... still looking...
Click to expand...
Click to collapse
OK; that makes sense. I just realized that after I hard-reset I restored most of my settings with the data from a backup (Sprite Backup). I wasn't seeing the encoded files problem because I was restoring from a non-encrypted file.
ugh.
y2whisper said:
From what I read about the encryption, the key is generated after a hard reset, so basically you can't hard reset the device once data is encrypted.
Click to expand...
Click to collapse
That makes perfect sense, actually. That way someone can't hard reset your phone to get at the data.
Too bad it also means the real owner can't get to his own data..
Some FAQs from the horse's mouth: http://blogs.msdn.com/windowsmobile...ows-mobile-6-storage-card-encryption-faq.aspx
What you can do is ActiveSync your Device and then drag and drop all the files you want to keep before the hardreset. And then when you finish installing your ROM and Hardresetting your device, just transfer back the files via activesync. I know its tedious and long if you have like 1 gig of **** in the SD card, but thats the only way i've found.
just lost files to encryption
Been reflashing my 8525 with new versions of custels and vanilla and have never lost files to encyption. However just flashed to Black 3.01 and lost all my stuff. If i flash back to my previous ROM is it conceivable that the same key will be created and i will regain access to my files?
Unfortunately, I was also unaware of this. I presumed MS would use a key based on the hardware or something like that.
Anyway, is there any way of breaking the encryption and get back the files?
Thanks!
Keshen
As the DataProtection API as in WinXP and Win2003 is used, it is AES-128 by default.
"The design and strength of all key lengths of the AES algorithm (i.e., 128, 192 and 256) are sufficient to protect classified information up to the SECRET level. TOP SECRET information will require use of either the 192 or 256 key lengths."
So brute force attack is not really an option….
As the key is stored somewhere in the flash I think this will be the only feasible way to decrypt the files.
But as the key is lost because of the hard-reset during upgrade, there is not much hope...
Only if we can get more information on how the keys are generated, maybe this will reduce
the complexity of an attack.
You won't have good luck trying to crack the encryption. Which, is actually a good thing since the purpose is to keep your data safe in the wrong hands. I prefer to use a 3rd party encryption solution as it allows more choices and control.
MrGAN said:
As the DataProtection API as in WinXP and Win2003 is used, it is AES-128 by default.
"The design and strength of all key lengths of the AES algorithm (i.e., 128, 192 and 256) are sufficient to protect classified information up to the SECRET level. TOP SECRET information will require use of either the 192 or 256 key lengths."
So brute force attack is not really an option….
As the key is stored somewhere in the flash I think this will be the only feasible way to decrypt the files.
But as the key is lost because of the hard-reset during upgrade, there is not much hope...
Only if we can get more information on how the keys are generated, maybe this will reduce
the complexity of an attack.
Click to expand...
Click to collapse
Breaking AES is pretty hopeless unless you have access to the NSA's computer systems.
The big problem, in my opinion, is MS doesn't give you an easy way to back up the key and restore it. Now that I've given it more thought, it makes perfect sense that the key gets nuked on a hard-reset: otherwise someone could just hard-reset your phone and have access to your data. In most cases, it's better to permanently lose the data than to allow someone else to have access to it.
Another thing that MS would be wise to offer is a "mass decrypt" function that would go through the entire SD and decrypt all of the encrypted files.
Other than that -- the way I've been dealing with it is by using SpriteBackup (I'm sure SBP does this too) and back up the SD card along with the main memory. Since the encoding happens in the OS level, the backup software sees the normal unencrypted files. So this way I can restore the SD backup after a rebuild (I don't recommend restoring the main ROM image, since so much changes between each release).
j
keshen said:
Unfortunately, I was also unaware of this. I presumed MS would use a key based on the hardware or something like that.
Anyway, is there any way of breaking the encryption and get back the files?
Click to expand...
Click to collapse
Once you hard-reset your device and lose that key, your files are gone, unless you somehow have access to some very very high powered computer systems that can break AES -- and even then it would take several years.
I think the NSA in the US has managed to break AES (that's the rumor I've heard), but it takes quite some time.
j
AES-128 offers a sufficiently large number of possible keys, making an exhaustive search impractical for many decades.Correctly implemented AES-128 is likely to protect against a million dollar budget for at least 50 – 60 years and against individual budgets for at least another 10 years.
But as there are many pitfalls to avoid when encryption is implemented, and keys are generated.
So if the key is easy to guess it doesn’t matter if AES is secure ot not.
Creating good and strong keys is a surprisingly difficult problem and requires careful design when done with a computer. The challenge is that computers are notoriously deterministic, but what is required of a good and strong key is the opposite – unpredictability and randomness.
Provided that the implementation is correct, the security provided reduces to a relatively simple question about how many bits the chosen key, password or pass phrase really corresponds to. Unfortunately this estimate is somewhat difficult to calculate, when the key is not generated by a true random generator.
To make a long story short: if at all then the key generation might be the weak spot...
So i've to realyze that i've lost all the data of mi SD card. That sounds incredible!!, i don't know if i will be able to recover all the changes made.
I installed Mobsync, wich makes me backups of the SD data to avoid SD corruption, but the files are also encrypted on my PC.
WM5torage
I'm curious if anybody has tried turning encryption on, and using WM5torage to transfer files to/from the Hermes. Does it properly encrypt/decrypt the files, or does that work only when using ActiveSync?
-- Joe
I have been following the xda-developer website for awhile now, as I have a KIN 2m of my own and I have been interested in finding a way to root or replace the current OS with a new one. Since I haven't seen anymore progress with the phone itself, I decided to do some digging and see what I could do myself (I am a programmer and IT Specialist). What I found might be helpful.
According to other posts, and what is floating around the internet, the KIN 2m flash memory is based on the Samsung MoviNAND 8G architecture. The chip itself is the Samsung MoviNAND KLM8G4DEDD-B101 which supplies 8GB of flash memory to the phone. Since this is also a Flash based NAND memory, I decided to investigate into the actual filesystem (FS) which makes up the chip and which WindowsCE 6.0 is based.
Apparently, this particular NAND memory is based on the YAFFS2 (Yet Another Flash File System rev2.0) which supports both little- and big-endian (32 and 64-bit architecture and some 16-bit systems), respectively. Also, the operating systems that are built on this particular FS are WindowsCE, Android, Linux, pSOS, eCos, and ThreadX.
Going off what I had found, I discovered that with regard to WindowsCE in particular, there are four different parts to the KIN NAND set-up: A Portable YAFFS "Core", a YAFFS Direct Interface, the WindowsCE wrapper, and then of course the WindowsCE OS itself. For WindowsCE, the WindowsCE wrapper accesses the YAFFS Direct Interface, not the core directly. In order to write instructions to the NAND and the "core", a set of instructions in three different types are necessary. These types are a POSIX Application Interface, an RTOS Integration Interface, and finally Flash Configuration and Access Interface. I have attached (and pasted) below a diagram of the above description and I have attached a document which was provided by yaffs.net which also covers some of these details.
View attachment 1461518
The POSIX Application Interface allows execution of application code to access the filesystem. These commands that are executed are typically open, close, read, write, etc. The RTOS Integration Interface consists of functions which allows for YAFFS to access the RTOS system resources. The commands are things like lock, unlock, initialize, get time, set error, etc.. Finally, we have a Flash Configuration and Access Interface which allows YAFFS to access the NAND directly and it executes commands such as initialize, read chunk, erase block, etc.)
So this is what I have discovered so far, and I am currently working on seeing where I go from here. I am currently trying to mount my KIN and browse it as a YAFFS filesystem on my computer, and once I do that, I am thinking that I might be able to execute instructions to access the ROM and NAND chip. On the YAFFS.net website there are a lot of good documents on how this FS works and how commands are executed. I am currently trying to read all I can and see if there is anything I can do.
And the plus side is, after all of this digging and experimenting...my KIN still works!!
very interesting.
wouldn't instructions be executed on the ARM?
http://en.wikipedia.org/wiki/ARM_architecture#Instruction_set
the trick would just be getting something on there and running...
Edits:
actually we may have already found a way to execute. it involves XNA or Silverlight and its on these forums somewhere. now compiling something...
must note that this could probably end very poorly.
so how we write android to NAND?
http://source.android.com/
BOOM
http://infocenter.arm.com/help/topic/com.arm.doc.ddi0290g/DDI0290G_arm1156t2fs_r0p4_trm.pdf
if we can access a console on the actual kin device we can compile code. now, how exactly we could do that is a mystery to me. the kinOS is running on top of windows CE kernel, so there may be one in there...
that console would be "run23"
http://support.microsoft.com/kb/194302
http://developer.download.nvidia.com/tegra/docs/tegra_250_hw_setup.pdf
check out page 12. to put the kin in recovery mode, you hold u+s+b+power.
also relevant
http://forum.xda-developers.com/showthread.php?t=894130&page=2
BOOM
infocenter.arm.com/help/topic/com.arm.doc.ddi0290g/DDI0290G_arm1156t2fs_r0p4_trm.pdf
if we can access a console on the actual kin device we can compile code. now, how exactly we could do that is a mystery to me. the kinOS is running on top of windows CE kernel, so there may be one in there...
that console would be "run23"
support.microsoft.com/kb/194302
Click to expand...
Click to collapse
That is a good thought. However, according to what I have found, there is no reason to go looking for a console on the device itself with which to compile/execute code. What I have found, to date, is that because the device is a YAFFS filesystem, if we have the correct drivers installed on our host machine (which can be gained from the internet), we can mount the device as a drive and using Command Prompt on the host machine we can compile and execute the code from there. We can use the commands NVFlash, mount, write, flash_eraseall, etc.
Sorry if this doesn't seem to make an sense at the moment...I am trying to multitask, and I believe I am failing miserably. I will hopefully have more information later.
IT_Tech said:
we can mount the device as a drive and using Command Prompt on the host machine we can compile and execute the code from there. We can use the commands NVFlash, mount, write, flash_eraseall, etc.
Click to expand...
Click to collapse
ooh, that's a good one. i think JohnKussak was doing something like that using the NVidia tegra toolkit, but he was not able to connect for some reason.
http://forum.xda-developers.com/showthread.php?t=894130&page=2
now, from what i understand of YAFFS (which is admittedly very little) it's just a partition of the memory right? or is the YAFFS system on a completely separate piece of memory?
i was reading about the MPU (memory protection unit) in the ARM manual and it described the layout of memory. basically it supports up to 16 blocks, all with 32 bit addresses (4294967295 bytes = 4gb). since there's 8 gb of storage on the kin, it can probably be assumed that 2 of those blocks are used for storage. another block is probably for RAM (256mb). now, the YAFFS system has to be accessible to the ARM somehow (maybe), so there are several possibilities.
1) the YAFFS file system is on a partition of the 8gb storage space
2) the YAFFS file system is on a partition of the 256mb RAM (pretty sure this isn't the case)
3) the YAFFS file system has its own block of memory
regardless, the MPU can be disabled. when it's disabled, no permissions are checked (ever) and we can read/write anywhere we please, including the YAFFS, wherever it may be. i'm not totally sure this is necessary, but i know we've had problems accessing system folders in the past. i suspect disabling the MPU could fix that, if we could manage.
i also suspect that the YAFFS is accessible when the phone is in recovery mode (u+s+b+power), we just need the correct drivers to flash a Tegra 2600 APX chip. i believe i read on the tegra thread that somebody couldn't find that particular driver. it may need to be written.
edit:
just found this https://developer.nvidia.com/tegra-2-technical-reference-manual
you have to apply for access and it can take up to a month i guess. i'm working on that right now. the main item of interest is "16.0 NAND Flash Controller". i'm gonna try looking in some shady places and see if i can't dig that up...
double edit:
just realized that tegra 2 is different than tegra APX, NVidia does not offer an APX manual anymore, if they ever did.
triple edit:
ooooookay. http://viewsonic-gtablet-for-dummies.webs.com/nvflash.htm
just grabbed source for drivers. unfortunately, NVflash does not run on 64 bit systems, which is causing a bit of problems for me...
slimeq said:
now, from what i understand of YAFFS (which is admittedly very little) it's just a partition of the memory right? or is the YAFFS system on a completely separate piece of memory?
Click to expand...
Click to collapse
From what I understand, YAFFS is a partitioning system. It does not need to be stored as a separate system on a separate piece of memory. However, it does also have the capabilities to be partially RAM based... Which may end up confounding things. If you get a chance, you may have done this already I don't know, but read up on the YAFFS file system on the yaffs.net website--it has a ton of documents on how the system works and the commands it executes, its architecture, etc.
slimeq said:
i was reading about the MPU (memory protection unit) in the ARM manual and it described the layout of memory. basically it supports up to 16 blocks, all with 32 bit addresses (4294967295 bytes = 4gb). since there's 8 gb of storage on the kin, it can probably be assumed that 2 of those blocks are used for storage. another block is probably for RAM (256mb). now, the YAFFS system has to be accessible to the ARM somehow (maybe), so there are several possibilities.
1) the YAFFS file system is on a partition of the 8gb storage space
2) the YAFFS file system is on a partition of the 256mb RAM (pretty sure this isn't the case)
3) the YAFFS file system has its own block of memory
Click to expand...
Click to collapse
Let me begin here and differentiate between YAFFS types...there are two types of YAFFS formatting--YAFFS1 and YAFFS2. The KIN is formatted using YAFFS2 because it has 8g of space. Yaffs2 is different from Yaffs1 in the fact that it allows for memory sizes greater than 4GB because it supports 4KB pages rather than 512byte pages (Yaffs1). I will spare all the details as they are lengthy, but check out:
yaffs.net/documents/how-yaffs-works and yaffs.net/yaffs-original-specification (paying attention to Yaffs2 and how it relates to it foundation off of Yaffs1). Also, since Yaffs only uses a RAM based system for emulation purposes when the kernel is not being run on a true NAND (or NOR) storage, we can eliminate Option 2. Option 3, may be viable, but I am not sure as to how. Option 1 seems to make the most logical sense (pun intended), because of how the YDI (YAFFS Direct Interface) works with the YAFFS kernel and filesystem, as well as the WindowsCE Wrapper, etc.
Now as for the MPU. Yaffs has a built in code to handle MPU and a way to disable it through a console session on a host machine, but I don't remember where I read it, I believe it was on the yaffs.net website in one of the technical documents... I will work to remember where I found it.
I hope we can get some more people on-board helping with this YAFFS thing. If we could, it might make this go a bit faster, and have more heads working on it. Plus they might see something we don't.
P.S: These are the HARDEST captchas I have ever seen in my entire life!
kin
Hopefully you get access to the filesystem with this IT_Tech :fingers-crossed:
Every once in a while, it's fun to revisit Dev on the Kin
I remember going through the Tegra 2500 APX links on the wayback machine:
http://web.archive.org/web/20100813070722/http://www.nvidia.com/object/product_tegra_apx_us.html
If you click the Specifications tab, you can see that the 2600 and 2500 are virtually identical, aside from some video features.
Also, the ZuneHD is the only other product listed as using the 2500 chip. I don't have one, so I wouldn't know for sure, but I wonder if its drivers could be tweaked to allow access to the Kin, in the same way as the Zune. I remember trying to hack into other Windows Mobile drivers (for other WinMo 6.x devices I have) but never getting anything further, even when in other USB modes on the device.
OK, this thread is going to be a work in progress, intended to serve as a reference for the work I've been doing on LVM partition remapping.
My work was done initially on a Find 7, but this should eventually be usable on many other devices (I have the Find 5 and N1 in mind for when I return from vacation). Also, this would not have been possible without the work Steven676 did years ago on the Nexus S, which has been used by all AOSP-derivative projects to support the Samsung Aries (Galaxy S) family for quite some time now.
The current state of things is that the patches are solid and work very well for the system side of things, but there is still a bit of work needed on the recovery side of things. This is due to TWRP having an architectural limitation I need to work on - Whether a device uses emulated storage or not is set at compile time, which is a problem if your design requires automatic detection of configuration at run time.
One of the key design goals here was to support both normal and LVM configurations automatically with a single build that detects which configuration is present on a device at run time.
A second key design goal was that the underlying partition table of the device is not touched in any way. Touching the partition table of a mobile device in the field is a fundamentally dangerous operation, as many partitions contain data that is device-unique or will render a device unbootable if altered. Recovery methods that involve DDing partition images to nonstandard partitions is asking for trouble due to typos... There's no protection against a user typoing the name of a critical partition.
Initially, I'm going to dump the contents of an email I wrote to someone giving them documentation on how to integrate LVM into their project. Over time I'll clean up and reorganize this post, including adding some more links. Also, since this email was written, I've added a LOT of comments to each patch explaining what is going on.
For additional documentation, especially a more user-oriented view of things (such as how to set this up if you want to use it with Omni nightlies) - see the Omni nightlies thread on XDA.
So here goes:
How it's implemented - the complete patch set is at:
https://gerrit.omnirom.org/#/q/topic:find7_lvm - Expect this to periodically change as work on this feature continues (Note: All patches required to support nightly builds of Omni have been merged. At this point, all remaining work that I expect is on polishing up TWRP.)
With the rest of this post, I'll talk about each individual patch and what it does.
https://gerrit.omnirom.org/#/c/9273/ - This is a patch against frameworks/base which adds an alternative to storage_list.xml called storage_list_lvm.xml - The frameworks will choose storage_list_lvm.xml instead of storage_list.xml if the property ro.lvm_storage is set to 1 - The device init scripts will set this property if they detect an LVM configuration.
https://gerrit.omnirom.org/#/c/9207/ - This is an Omni-specific patch. Omni builds for both the Find 7 and OnePlus One (also known as find7op) and both share a common device tree. The LVM patches do not apply to the find7op, so we move init.recovery.rc out of the msm8974-common tree - You likely don't have to worry about this unless you also have a -common tree for find7 and find7op
https://gerrit.omnirom.org/#/c/9276/ - Normal Android kernel ramdisks do not include busybox or any form of shell, making it impossible to run shell scripts without /system mounted. Since we need to run a shell script prior to mounting partitions, we need to add busybox to the ramdisk. This patch does that. For legal reasons you may wish to replace busybox with system/core/toolbox and system/core/sh - I have not tried doing so. If you choose to stay with busybox, you will have to provide the busybox source code in order to comply with the GPL.
https://gerrit.omnirom.org/#/c/9205/ - This adds the LVM binary and LVM configuration file to the ramdisks of both normal boot and recovery. This patch does not actually begin doing anything with the binaries, I separated it out from the other patches as a way to keep things organized so I could start working with the binaries when I began this project. The original source code and documentation for the binary is at https://github.com/steven676/android-lvm-mod
One change I made in lvm.conf that differs from the Samsung aries family (galaxysmtd, fascinatemtd, captivatemtd, etc.) is that I changed the filter line to only allow the userdata and sdcard partitions. This prevents LVM's vgscan from accidentally determining another partition is a physical volume, and also prevents users from accidentally running pvcreate on a critical partition.
https://gerrit.omnirom.org/#/c/9206/ - This is where all of the "heavy lifting" is done. I'm going to work on adding more comments to the init scripts and shell scripts to document them tonight and tomorrow, but I'll try to explain things here.
Android's init system is a bit limited in that it's very difficult to have conditional behavior defined in init.rc - which appears to be why Qualcomm loves to use shell scripts called from init. Similarly, much of the LVM magic happens in three shell scripts (which execute at three different phases within the boot sequence).
In the early-init phase, the two "wait" blocks ensure that the underlying block devices are ready before vgscan/vgchange are called. This will probably slow down booting by a few fractions of a second unfortunately.
vgscan will scan the volumes defined in lvm.conf (in this case, only the userdata and sdcard partitions) for LVM physical volumes. If LVM physical volumes are detected and form a proper volume group, vgscan will create appropriate device nodes. With the configuration I'm using, the device node will be /dev/lvpool/userdata - which consists of a single logical volume that merges the sdcard and physical userdata physical volumes (partitions). The configuration of lvm.conf prevents LVM commands (especially pvcreate) from altering partitions we don't want to alter. If someone accidentally tries to, for example, run pvcreate on the system partition, it will give an error indicating that the partition was not part of the filter.
vgchange will activate the physical volumes detected by vgscan
lvm_init.sh will check to see if /dev/lvpool/userdata exists, and copy fstab.qcom.lvm to fstab.qcom, init.fs.rc.lvm to init.fs.rc, and twrp.fstab.lvm to twrp.fstab if it does. If it does not, it selects fstab.qcom.std, etc.
In the "on init" section, the init script exports all environment variables from init.fs.rc, and creates all storage-related directories and symlinks needed for both configurations (except for when they conflict). lvm_symlinks.sh will create directories/symlinks that must be configuration-specific. Just like lvm_init.sh - it decides what to do based on whether /dev/lvpool/userdata exists
In the "on fs" section - we do an SELinux restorecon on /dev/mapper/lvpool-userdata (/dev/lvpool/userdata would probably work here too). If it doesn't exist, this will fail gracefully without causing any issues.
In "on early-boot" - lvm_setprop.sh uses /system/bin/setprop to set ro.lvm_storage to 0 or 1 depending on the detected configuration. The property service is not available until early-boot - so this cannot be in lvm_init.sh or lvm_symlinks.sh This propery is used by the frameworks/base patch above to determine which storage_list to choose.
At the end of the init.qcom.rc, the fuse daemon for emulated storage is added for all configurations. (I could not figure out a good way to make this conditional based on whether LVM was present or not). In a non-LVM configuration, it runs but is harmless - it maps /data/media (which is empty) to /mnt/shell/emulated (which nothing is looking at due to the environment variables and symlinks set in the "on init" section )
You will probably notice that Omni's standard storage configuration is fairly different from ColorOS - this is due to the way KitKat storage works, but it allowed us to get away without using Oppo's ext4 permissions hacks in our kernel (by remapping permissions instead, in a manner similar to how the emulated storage system works) The way we handle our /sdcard partition does interoperate without issues with the ColorOS approach.
https://gerrit.omnirom.org/#/c/9279/ is a patch specifically for TWRP. TWRP currently determines whether to use emulated storage (/sdcard on /data/media) at build time instead of at run time. Until I have time to fix this, the patch here operates as a workaround. It is similar to the behavior of the fuse sdcard daemon in the previous patch - it maps /data/media to /sdcard whether the configuration is actually emulated storage or not. If the device is not using emulated storage (LVM), mapping of /data/media to /sdcard is still mostly harmless. However it does result in undesirable changes to TWRP's user interface. DO NOT USE THIS APPROACH IN PRODUCTION RELEASES. It's a horrible hack. You'll need to figure out how to properly do /data/media handling depending on whether LVM is present or not based on how your own recovery architecture works.
https://gerrit.omnirom.org/#/c/9281/ adds "raw" sdcard and userdata partition entries to the partition table for the LVM configuration. This allows a user to return their device to a standard configuration by formatting the underlying sdcard and userdata partitions directly, instead of the removelvm ZIP at the beginning of this email. - To be abandoned, this patch was squashed into 9206
FAQ
Q: Coldbird already had repartitioning support. Why did you create this different approach?
A: Even before he started work, I strongly recommended that he not touch the partition table of the device. It's a really bad idea and is fundamentally dangerous. It's pure luck that someone hasn't hardbricked yet. (A number of people have come close.) If you read through his thread and the ColorOS 2.0.2 thread, you'll see that the repartitioning approach fails frequently, and in multiple ways. (Missing partition contents, partition table ending early, etc. The latter is really scary, one person had the process fail at mmcblk0p19 - what if someone else's partition table write operation aborted even earlier?.) Also, nearly everyone that has implemented support for that approach has needed a separate build to support it. (Oppo is the first to manage autodetection.) I also provided him all of the reference information from Steven676's work.
LVM is far safer. The underlying partition table is not touched in any way. Instead, LVM remaps sectors on the fly so that two partitions that are not adjacent to each other on the physical storage appear as a single contiguous partition to the filesystem drivers. Linux has supported LVM for on the order of a decade, if not more. I've been using LVM on my file server since 2006. (Yes, the system is 8 years old and still working other than needing a new power supply after a thunderstorm. Nothing to do with LVM. ) In addition, the lvm.conf configuration used here provides protection against accidental typos causing damage. Undoing the changes is as simple as doing a wipe of /data and /sdcard from any standard recovery and can be done in seconds, not of running a special batch file that runs a bunch of fastboot commands and takes 4-5 minutes. Similarly, the LVM setup process currently described in the Omni thread involves flashing a single ZIP from recovery that takes only 10-15 seconds, and most of that process is flashing an LVM-aware recovery. (The only limitation currently is that the ZIP must be on external storage - USB OTG or MicroSD)
To put it simply, it Just Works. No need to back up a pile of partitions other than /data and /sdcard because those partitions are never touched or altered.
Q: I have a device with a ridiculously oversized /system partition, can I get some of that back for /data?
A: Yes, you can. Add the physical /system partition to the lvm.conf filters and add it to the lvpool when creating it, then create a smaller /system LV out of this big pool. (see updater.sh in device/samsung/aries-common/ of any AOSP-derivative for hints here.) Be careful though - leave enough spare space for growth (new Android versions, etc.) While it should be possible to use some of the LVM tools along with ext4 resize tools to reorganize the LVs without wiping, this is very difficult and you'll probably have to make users wipe /data if you want to alter /system.
*reserved 2*
Nice work, I hope all the patches can be widely used on some other devices and other roms.
systop said:
Nice work, I hope all the patches can be widely used on some other devices and other roms.
Click to expand...
Click to collapse
Yup. I know Andre from PA was working on it last week but I haven't heard from him lately.
My priority when I return from vacation will be fixing up the TWRP side of things. It's working for now, but the user interface on non-LVM configs is a little funky thanks to RECOVERY_SDCARD_ON_DATA being compile time. This has never been a problem before since a single TWRP binary never had to support two different configurations before. I plan on either doing a property-based approach or fstab-based like CWM. (It should be possible for someone to make a CWM build that automatically detects configuration without any modifications to CWM, based on reading the code - but I haven't tried it myself.)
Once TWRP is in better shape, I plan on doing the Find 5 and N1. These will have the challenge of not having a MicroSD slot, so I may have to change TWRP so that it use /tmp instead of /sdcard when doing "adb sideload", or at least gives the user that option.
Good stuff :good: I don't really need it as of yet, but when my new device is provided (warranty) I will surely give this a try.
I hope ayysir will merge the LVM support very soon ^^
Find 7u PA 4.6 beta 1
Awesome work mate. I have avoided other methods because I'm always the guy that will have a device fail at very bad timing; like during boatloader or SBL stage.
I'm really glad you have continued to work on this. I have hit thanks a few times but would also like to thank you here
tork987 said:
I hope ayysir will merge the LVM support very soon ^^
Find 7u PA 4.6 beta 1
Click to expand...
Click to collapse
He had issues with merging support, hopefully now that I've added more documentation he can try again.
how are the *.std files created?
atm this is tough for me to port from omni to cm base which AOSPA Oppo trees
ayysir said:
how are the *.std files created?
atm this is tough for me to port from omni to cm base which AOSPA Oppo trees
Click to expand...
Click to collapse
the std files are also part of the device tree
https://github.com/omnirom/android_device_oppo_find7/tree/android-4.4/configs
ayysir said:
how are the *.std files created?
atm this is tough for me to port from omni to cm base which AOSPA Oppo trees
Click to expand...
Click to collapse
For the fstabs - they are simply moves/renames of the fstab files and other storage-related items from the standard Oppo configuration (they should appear as renames/moves in the Gerrit commit...)
For the init.fs.rc file - all of the "export <blah>_STORAGE" lines from init.qcom.rc/init.find7.rc are cut out of the RC file and put into .std
Obviously, the .lvm versions of the files are the ones where the fstab has been altered to support a single data partition with emulated storage.
Amazing work and amazing posts. Thanks a lot for your sharing. ?
I've got a question related to your configuration (/data and /sdcard merged) : are the LV hot-resizables?
Wendigogo said:
Amazing work and amazing posts. Thanks a lot for your sharing. ?
I've got a question related to your configuration (/data and /sdcard merged) : are the LV hot-resizables?
Click to expand...
Click to collapse
In theory, you could probably use some of the ext4 resizing tools to do something like this, but I haven't looked into it as there isn't much point in the current config (since the LVM userdata volume is allocated to use all space on the volume group).
Something like that might be more useful if someone ever uses LVM to regain some of the wasted /system partition space on certain excessively bloated devices (like some GS4 units).
Entropy512 said:
In theory, you could probably use some of the ext4 resizing tools to do something like this, but I haven't looked into it as there isn't much point in the current config (since the LVM userdata volume is allocated to use all space on the volume group).
Something like that might be more useful if someone ever uses LVM to regain some of the wasted /system partition space on certain excessively bloated devices (like some GS4 units).
Click to expand...
Click to collapse
Thanks for your answer.
Seems I misunderstood the way it's implemented here. All space is allocated to /data? So there's no more internal sdcard right?
But in that case an external sdcard is mandatory. How is it managed when there's no sdcard?
Enjoy!
Wendigogo said:
Thanks for your answer.
Seems I misunderstood the way it's implemented here. All space is allocated to /data? So there's no more internal sdcard right?
But in that case an external sdcard is mandatory. How is it managed when there's no sdcard?
Enjoy!
Click to expand...
Click to collapse
Android has supported emulated storage (where /data/media is mapped to /sdcard with a special FUSE daemon that makes /sdcard have DOS-like permissions despite an underlying ext4 partition) since ICS. It's pretty much the standard in all new devices - the Find 7 is to my knowledge the only device launched in 2014 not to use emulated storage. Most devices in 2013 also did - Oppos were again the rare exception.
As I understand it - for some reason Chinese users prefer the legacy pre-ICS partitioning scheme. My guess is due to UMS vs. MTP - MTP is required for access to emulated storage, UMS can't be used, but a lot of older desktop OSes have issues with MTP. So Oppo finds themselves in conflict between their home market (China) and expanding in the West. That said, the Find 7 was kind of a screwup in achieving this goal, since the internal sdcard partition was ext4 which meant UMS was a no-go for it.
Entropy512 said:
Android has supported emulated storage (where /data/media is mapped to /sdcard with a special FUSE daemon that makes /sdcard have DOS-like permissions despite an underlying ext4 partition) since ICS. It's pretty much the standard in all new devices - the Find 7 is to my knowledge the only device launched in 2014 not to use emulated storage. Most devices in 2013 also did - Oppos were again the rare exception.
As I understand it - for some reason Chinese users prefer the legacy pre-ICS partitioning scheme. My guess is due to UMS vs. MTP - MTP is required for access to emulated storage, UMS can't be used, but a lot of older desktop OSes have issues with MTP. So Oppo finds themselves in conflict between their home market (China) and expanding in the West. That said, the Find 7 was kind of a screwup in achieving this goal, since the internal sdcard partition was ext4 which meant UMS was a no-go for it.
Click to expand...
Click to collapse
I've got it now. Thanks for your explanations
I saw that Oppo phones didn't follow Android guidelines (yet?) by not using the emulated_storage mounting method but I didn't know why.
And your right, mtp doesn't work in Windows XP (or is hard to make working) and there's a lot of Asian people still using it. Obvious once you said it...
And that's also why only external sdcard is accessible in UMS mode in recovery.
Thanks again for your enlightenment. ?
Reading some of the comments on G+ it looks like Oppo might be using this solution for their KitKat release. I would be so pleased if they did.
Sent from my X9076 using Tapatalk
kishd said:
Reading some of the comments on G+ it looks like Oppo might be using this solution for their KitKat release. I would be so pleased if they did.
Sent from my X9076 using Tapatalk
Click to expand...
Click to collapse
You could be pleased...
Wendigogo said:
You could be pleased...
Click to expand...
Click to collapse
Had some problems with camera focus on earlier versions of omnirom for find 7. Now those have been addressed. I installed Omni and am on the nightlies with lvm. My find 7 and find 7a will not see another rom again.