Several of us are exploring how to change an Android build so that it unmounts the /data partition cleanly when shutting down. This thread is dedicated to discussion of how to do that. Solutions may eventually appear here, but will work their way into the various build and/or kernel threads.
reserved
Watch this space for a summary of answers and solutions.
Roughly how it works today
The Shutdown thread stops applications, and then tells the Volume Manager to shutdown. After that, the Shutdown thread calls reboot() which calls _reboot() which turns off power.
I think the key is in Volume Manager.
Volume Manager's Java code works with the C++ application "vold".
Volume Manager gets a list of mounted filesystems from vold.
Vold returns a list that it manages, not the same as "df" or "mount" from the command line. That list (at least in Fresh Froyo) does not include /data or /cache.
Volume Manager then, for each filesystem in the list, tells vold to unmount the filesystem. Each filesystem has a timeout, and Volume Manager eventually either finds success or gives up.
The "vold" application can be controlled with a command-line executable called "vdc". Commands like : "vdc volume unmount /sdcard" or "vdc volume list", use the same syntax as Volume Manager uses to talk to vold.
Perhaps the answer to all of this is to get /data registered with vold at startup.
What I have tried, with great failure, is a terrible hack where I modified the implementation of reboot() to call system("/system/bin/rosysdata") before calling _reboot(). Then I created /system/bin/rosysdata to remount /system and /data as r/o. That worked for a couple of shutdowns, but after that caused extreme filesystem failure a few times. It was a bad hack anyway, so I have never committed that.
If we can get the /data filesystem mounted by vold instead of just mounting it in the bootup scripts, it should get unmounted at shutdown, since VolumeManager gets a list of mounted volumes from vold and unmounts each of them.
I am trying this but don't have much success yet.
Another idea is to get android to execute a shutdown script as it shuts down.
The old initrd create severl problems with mounting because there was several binding to the data partition.
Now there is only one /data mounted to the partition.
The old script files has the problem with the binding but now they should work.
I think that vold can't be used with data so we need to exec:
Code:
sync
mount -o remount,ro /data
in a shutdown script.
energy cut
what about when the system crashes and battery has to be removed?
anergy cut ...
What about use different partitions?
/system, /data and /cache as different partitions...
the highest activity of reading and writing happens in /cache. Am I right?
I'll change the nbh and sysinit.rc to meet this.
tiagoclc said:
what about when the system crashes and battery has to be removed?
anergy cut ...
What about use different partitions?
/system, /data and /cache as different partitions...
the highest activity of reading and writing happens in /cache. Am I right?
I'll change the nbh and sysinit.rc to meet this.
Click to expand...
Click to collapse
Isn't /cache a symlink to /data/tmpcache ? couldn't we redirect to /mnt/sdcard/cache ? (or anywhere else ?)
Due to these issues I can reboot normally only if I clear Dalvik cache. Maybe it could be automatic on boot lol
drvitorino said:
Due to these issues I can reboot normally only if I clear Dalvik cache. Maybe it could be automatic on boot lol
Click to expand...
Click to collapse
I second this or how about we manually enter mount -o remount, ro /data on terminal before turning it off?
ohohoh, apparently L1qu1d fixed my issue with the new kernel and the update from http://forum.xda-developers.com/showthread.php?t=848921
Unfortunately wi-fi and data connection is still not working with the update in my kaiser. But L1qu1d may be happy to know that his code inside the update has something that fix data corruption when turning off/on the device.
drvitorino said:
ohohoh, apparently L1qu1d fixed my issue with the new kernel and the update from http://forum.xda-developers.com/showthread.php?t=848921
Unfortunately wi-fi and data connection is still not working with the update in my kaiser. But L1qu1d may be happy to know that his code inside the update has something that fix data corruption when turning off/on the device.
Click to expand...
Click to collapse
In my test, some data corruption still exists (when lots of widgets and data partition close to full) but is easily corrected by clearing the dalvik-cache, but some apps are lost in the launcher. I still think that unmounting the data partition before turning off or reboot is the best way to prevent this.
n2rjt said:
The Shutdown thread stops applications, and then tells the Volume Manager to shutdown. After that, the Shutdown thread calls reboot() which calls _reboot() which turns off power.
I think the key is in Volume Manager.
Volume Manager's Java code works with the C++ application "vold".
Volume Manager gets a list of mounted filesystems from vold.
Vold returns a list that it manages, not the same as "df" or "mount" from the command line. That list (at least in Fresh Froyo) does not include /data or /cache.
Volume Manager then, for each filesystem in the list, tells vold to unmount the filesystem. Each filesystem has a timeout, and Volume Manager eventually either finds success or gives up.
The "vold" application can be controlled with a command-line executable called "vdc". Commands like : "vdc volume unmount /sdcard" or "vdc volume list", use the same syntax as Volume Manager uses to talk to vold.
Perhaps the answer to all of this is to get /data registered with vold at startup.
Click to expand...
Click to collapse
Since our data is installed in NAND, I don't think "vold" can mount Nand partition. Not too sure though.
clemsyn said:
Since our data is installed in NAND, I don't think "vold" can mount Nand partition. Not too sure though.
Click to expand...
Click to collapse
from: http://osdir.com/ml/android-porting/2010-06/msg00279.html
Vold doesn't mount Nand.
Mounting of Nand partitions is done in init.rc
mountd: mount all fs defined in /system/etc/mountd.conf if started,
receive commands through local socket to mount any fs. The source is
in device/system/bin/mountd.
-
Sreekanth
On Jun 24, 5:47 pm, "Dennis.Yxun" <[email protected]> wrote:
> HI Community:
> I'm using donut branch, and my system have two block devices, I want them
> all mounted
> 1) nand flash: /dev/block/sdb
> 2) MMC/SD card: /dev/block/mmcblk0
>
> Current, I successfully use vold mount the MMC/SD card
> (/dev/block/vold/179:0 -> /dev/block/mmcblk0)
>
> My question here is :
> Does vold support to mount the Nand Device too? like
> mount /dev/block/sdb to /sdcard/nand
> mount /dev/block/mmcblk0 to /sdcard/mmc
>
> Dennis
is this true?
Related
Hi guys,
Now that several lagfixes are using the ext4 filesystem, perhaps we should look at optimizing ext4 for performance and lifespan of the flash memory.
One of the main things we can change is the journaling method to data=writeback. This should reduce writes and improve performance at a slight expense of reliability. Quote:
"In data=writeback mode, ext4 does not journal data at all. This mode provides a similar level of journaling as that of XFS, JFS, and ReiserFS in its default mode - metadata journaling. A crash+recovery can cause incorrect data to appear in files which were written shortly before the crash. This mode will typically provide the best ext4 performance."
One explanation here:
http://blog.smartlogicsolutions.com...ions-to-improve-ext4-file-system-performance/
Reference here:
http://git.kernel.org/?p=linux/kern...02ac5fa36d7f4c07856fe9cf89391e08986f7;hb=HEAD
HOW TO
1. Change the default mount option of the partition using tune2fs. You need to use a version of tune2fs that supports ext4, the one in busybox 1.17 does NOT. A working version is attached to this post.
- Push the tune2fs file to the phone's SD CARD:
adb push tune2fs /sdcard/
- Copy the tune2fs file to /data/
adb shell
su
cp /sdcard/tune2fs /data/
Now change the options:
/data/tune2fs -o journal_data_writeback /dev/block/mmcblk0p2
To verify:
/data/tune2fs -l /dev/block/mmcblk0p2
Look for the line that says:
Default mount options: journal_data_writeback
* Repeat the above for all other ext4 partitions.
It should take effect after a reboot. The next time it will be mounted automatically with data=writeback. You might have to do the tune2fs settings and reboot twice to get it to stick - I'm not sure why, but I had to.
You can verify this using the mount command:
# busybox mount | grep ext4
/dev/block/mmcblk0p2 on /data type ext4 (rw,noatime,barrier=0,nobh,data=writeback,noauto_da_alloc)
/dev/block/stl10 on /dbdata type ext4 (rw,noatime,barrier=0,nobh,data=writeback,noauto_da_alloc)
2. In addition to that, you can also edit the mount options to include the nobh option, which is a further minor optimization for data=writeback mode.
I personally use the options noatime,barrier=0,nobh,data=writeback. Voodoo already uses some of them like noatime and barrier=0.
I do it using a startup script (you need to know how to use/modify a startup script), with the following commands:
for k in $(busybox mount | grep ext4 | cut -d " " -f3)
do
sync
busybox mount -o remount,barrier=0,nobh $k
done
EDIT: Attached tune2fs that supports ext4. Works on Froyo kernels with ext4 support.
I posted the same request in another thread and i though it will be better if i do it here.
I found e2fsprogs-ext4.zip, but not sure if this is the correct zip.
it will be great if you can provide it here.
For others willing to try it, please change the /dev/block/mmcblk0pX to whatever you are using now.
my system is using /dev/block/mmcblk0p4
s88 said:
I posted the same request in another thread and i though it will be better if i do it here.
I found e2fsprogs-ext4.zip, but not sure if this is the correct zip.
it will be great if you can provide it here.
Click to expand...
Click to collapse
I've attached it here, it worked for me with the Universal Lagfix Froyo kernel that supports ext4.
hardcore said:
I've attached it here, it worked for me with the Universal Lagfix Froyo kernel that supports ext4.
Click to expand...
Click to collapse
And did you notice any improvements with new mount options?
busybox 1.17.1 says invalid option -o on tune2fs
==
i need the version attached to this thread, busybox version only supports ext2 and ext3
danzel said:
busybox 1.17.1 says invalid option -o on tune2fs
Click to expand...
Click to collapse
copy tune2fs into /system/xbin and chmod 755
Write back will cause data loses on crashes and it will increase read time on read miss since you need to write the block back to the main mem before reading a new block.
how & where do i edit mount options?
mdalacu said:
And did you notice any improvements with new mount options?
Click to expand...
Click to collapse
I didn't do any benchmarks, but in theory it should be better - reduced periodic journal writes (better battery life, flash lifespan and performance).
I don't know about performance but my SGS feels like a little bit snappier.
Thanks for this info.
chanw4 said:
Write back will cause data loses on crashes and it will increase read time on read miss since you need to write the block back to the main mem before reading a new block.
Click to expand...
Click to collapse
There is always a risk of slight data loss during crash, even with data=ordered. And we are already using some even more 'risky' options like barrier=0 and noauto_da_alloc anyway.
This setting increases the risk a bit more.
s88 said:
copy tune2fs into /system/xbin and chmod 755
Click to expand...
Click to collapse
He's referring to the busybox 1.17.1 version. It doesnt support the ext4 options. For the version attached here, you have to copy it to somewhere else like /data/ before u can execute it. You don't really need to copy it to /system/xbin.
hardcore said:
There is always a risk of slight data loss during crash, even with data=ordered. And we are already using some even more 'risky' options like barrier=0 and noauto_da_alloc anyway.
This setting increases the risk a bit more.
Click to expand...
Click to collapse
Yeah but I think at some point you're just throwing away the safety of EXT4, and might as well just use EXT2 since it writes faster even with these options on. (Can anybody confirm this? I did a check and EXT2 is still faster even with these mount options... while EXT4 reads faster. But writes are more important than reads on MoviNAND from my usability experience, since the reads are generally fast enough anyway.)
RyanZA said:
Yeah but I think at some point you're just throwing away the safety of EXT4, and might as well just use EXT2 since it writes faster even with these options on. (Can anybody confirm this? I did a check and EXT2 is still faster even with these mount options... while EXT4 reads faster. But writes are more important than reads on MoviNAND from my usability experience, since the reads are generally fast enough anyway.)
Click to expand...
Click to collapse
Hi Ryan, it's different. ext4 with data=writeback still uses journaling but only for metadata, which supposedly "provides a similar level of journaling as that of XFS, JFS, and ReiserFS".
So it's still safer than ext2 but without the speed (and more importantly, battery or flash write lifespan) impact full data+metadata journaling.
More detailed information in the sources here:
http://git.kernel.org/?p=linux/kern...02ac5fa36d7f4c07856fe9cf89391e08986f7;hb=HEAD
hardcore,
how do you change the mount options? like buffer from 1 to 0 and nobh, etc
s88 said:
hardcore,
how do you change the mount options? like buffer from 1 to 0 and nobh, etc
Click to expand...
Click to collapse
I've made a clearer procedure in the first post. To change some options you might need a startup script though.
hardcore said:
I didn't do any benchmarks, but in theory it should be better - reduced periodic journal writes (better battery life, flash lifespan and performance).
Click to expand...
Click to collapse
That sounds very promising
Followed the procedure - had to do it twice as well to make it stick...dont see any obvious changes in the last 5 minutes. Will update after using it for a while.
PS: how persistent is that setting? what circumstance does it get reset? When a new kernel is installed?
Does it work with 2.1 too?
bobbel said:
Does it work with 2.1 too?
Click to expand...
Click to collapse
yes, it does. I am using JM9, and it feel faster.
take note i only change the data to write_back, haven't figure out a way to do the rest.
for write back, it can be done on the terminal emulator.
s88 said:
yes, it does. I am using JM9, and it feel faster.
take note i only change the data to write_back, haven't figure out a way to do the rest.
for write back, it can be done on the terminal emulator.
Click to expand...
Click to collapse
Cool, I give it a try.
One Question:
When I enter the mount cmd I have:
/dev/block/mmcblk0p4 /data ext4 rw,noatime,...
So I have to use
/data/tune2fs -o journal_data_writeback /dev/block/mmcblk0p4 (instead mmcblk0p2)
Right?
SOLVED: This was fixed by adding and using e2fsck, see the link in post #3. I had issues using adb push to install the tools, see the CyanogenMod thread. I do NOT believe the parted files he linked were intended to be used on ext4, so YMMV, but it did convert my fscked up ext4 partition to a over-writable ext2.
*********************************
I am having a problem with my sd-ext partition being unable to mount. After a relatively routine procedure, I suddenly have this problem (below), for example when I tried to Nandroid Restore my sd-ext partition.
Code:
E:Can't mount /dev/block/mmcblk0p2
(File exists)
Setup: HTC Desire; HBOOT 0.93, S-ON; ClockWorkMod v2.5.0.7; CyanogenMod-6.0.2; DTapps2sd
I was trying the relatively simple procedure of removing a couple of system Apps, according to the instructions on the CyanogenMod Wiki:Barebones.
I had minor problems with capitalization since I'm using Windows.
I removed Calculator, Email, and Calendar. Upon restart, all of my Apps are gone, except the few system apps and GApps that were not moved via dtapps2sd.
The phone still "works", I can send/receive SMS, make calls etc., but it's pretty highly crippled.
I've tried a Nandroid Restore, but it gives this message above (as did my attempt to do a Nandroid Backup).
The dtapps2sd "repair" command also said something about "cannot mount".
I tried to wipe and reflash CyanogenMod, which worked (in that it did re-flash). But when I tried to then Restore, I got the same "Can't mount" problem.
I also tried the following fastboot command I found in another thread, with no effect
Code:
fastboot oem enableqxdm 0
fastboot reboot
I have noticed that whey I use Terminal Emulator, the Permissions on mmcblk0p2 don't match the other blocks. Could this be related? Sadly, my Linux karate is weak :/
Code:
$ cd /dev/block
$ ls -al
..
..
brw------ .. .. .. .. .. mmcblk0p1
brw-r--r--.. .. .. .. .. mmcblk0p2
brw------ .. .. .. .. .. mmcblk0p1
..
I guess I haven't yet tried the HTC RUU, which is always a possible fallback. Or I could try to flash a stock PB99IMG.ZIP file. But those seem like just a generic "solution" that doesn't really involve any understanding. And it certainly doesn't help others with this same problem, or to figure out what I did to create this problem, so others could avoid that.
UPDATE: Link for PB99IMG.ZIP. It's ~140MB, and ROM+Radio specific, so no I won't post it here.
http://shipped-roms.com/shipped/Bravo/
Still trying
I just tried another suggestion from these threads:
http://code.google.com/p/cyanogenmod/issues/detail?id=1614
http://forum.cyanogenmod.com/index.php?/topic/253-test-builds/page__st__380__p__6124#entry6124
Code:
adb shell
# su
# tune2fs -O ^huge_file /dev/block/mmcblk0p2
# sync
# reboot
But that did nothing for me.
More info
Following along from raskolnik
http://forum.cyanogenmod.com/topic/...ternal-storage/page__view__findpost__p__56396
I now have parted (and others) added to my ClockWorkMod Recovery
Below is the partition info for my SD card
Code:
parted /dev/block/mmcblk0
GNU Parted 1.8.8.1.179-aef3
Using /dev/block/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: SD USD (sd/mmc)
Disk /dev/block/mmcblk0: 8017MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 7444MB 7444MB primary fat32
2 7444MB 7979MB 535MB primary ext4
3 7979MB 8011MB 32.9MB primary linux-swap(v1)
I followed that up with a check with e2fsck
Code:
~ # e2fsck /dev/block/mmcblk0p2
e2fsck /dev/block/mmcblk0p2
e2fsck 1.41.6 (30-May-2009)
e2fsck: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear<y>? n
no
e2fsck: Illegal inode number while checking ext3 journal for /dev/block/mmcblk0p2
I decided NOT to go through with clearing the journal, as I don't really understand what implications that would have.
UPDATE:
Actually, I could only do adb push for the fiiles parted et al. after I had rebooted into CWM Recovery. When I rebooted normally, now the files are gone. I suppose this has something to do with the fact that I am still S-ON, since when the adb push failed, I got a response like "... read-only file system"
More Info:
I just did adb pull /etc/fstab. It looks like this
Code:
/dev/block/mtdblock4 /cache yaffs2 rw
/dev/block/mtdblock5 /data yaffs2 rw
/dev/block/mtdblock3 /system yaffs2 rw
/dev/block/mmcblk0p1 /sdcard vfat rw
/dev/block/mmcblk0p2 /sd-ext auto rw
[SOLVED] kinda
LEARN BY DOING!!!
So, I got anxious and just went ahead with the e2fsck command from above.
There were a LOT of problems (so many I had to .zip the attachment, it was waay over the forum .txt size limit) But, probly the tool wasn't meant for ext4...? I know that the actual Linux version does support ext4, but this one...?
Anyway, see the attached log if you want to check out more details. I don't really know WHAT all happened, whether anything good or bad went down.
I haven't really "Learned" anything, but the problem is fixed, more or less. After doing e2fsck, I was able to do a Nandroid Restore, and the mmcblk0p2 problem(s) were gone.
Spoiler: Semi-related minutia
Of course, since I had been careless and not done a Backup in about a week, actually ended up restoring last weeks' backup, then restoring last nights' backup sans sd-ext, then doing dtapps2sd: repair, remove, reinstall, repair. Then some manual app reinstalls, a couple widgets, and some shortcuts. Overall, tho, I think I only lost one SMS through the whole deal.
ScottHW said:
LEARN BY DOING!!!
So, I got anxious and just went ahead with the e2fsck command from above.
There were a LOT of problems (so many I had to .zip the attachment, it was waay over the forum .txt size limit) But, probly the tool wasn't meant for ext4...? I know that the actual Linux version does support ext4, but this one...?
Anyway, see the attached log if you want to check out more details. I don't really know WHAT all happened, whether anything good or bad went down.
I haven't really "Learned" anything, but the problem is fixed, more or less. After doing e2fsck, I was able to do a Nandroid Restore, and the mmcblk0p2 problem(s) were gone.
Spoiler: Semi-related minutia
Of course, since I had been careless and not done a Backup in about a week, actually ended up restoring last weeks' backup, then restoring last nights' backup sans sd-ext, then doing dtapps2sd: repair, remove, reinstall, repair. Then some manual app reinstalls, a couple widgets, and some shortcuts. Overall, tho, I think I only lost one SMS through the whole deal.
Click to expand...
Click to collapse
hi, i'm facing problem more or less just like you. I'm trying using e2fsck but it keep saying the super block can not be read or does not describe a correct ext2 system.
do you have any idea?
semi-brick.
Linking the problems
Just trying to link the thread involving my problem issue to this one as both are similar.
http://forum.xda-developers.com/showthread.php?p=44617022#post44617022
e2fsck seems to be the key to solve it as S2E log could automatically solve it. S2E is not compatible with higher android version ROMs that's all.
**** EXT4 for DATA (SD) install on second post *****
I've been slowly working on our Kaiser Kernel hoping to get extra life from it. Here are the changes I made so far.
1. Updated to the latest initrd and made a few changed in the init file
2. Changed the default I/O scheduler to noop.
3. Added block data integrity support and v4 support
4. Updated the latest yaffs2 filesupport up to changes dated 1/14/2011
5. Added PM module for Qualcomm 7500 chip
Will try to add BFQ I/O scheduler and see how the performance is compared to noop.
1/29/11 ** UPDATED KERNEL for better YAFFS2 best suited for Kaiser Samsung Memory chip **
Here is the nbh. This is type2 screen and keyboard1.
Will release my latest changes and update to git (need help on this one) when stable.
Please reinstall ROM and wipe data after flashing the kernel. The changes in yaffs2 code might cause issues. You can restore your data from BACKUP after installing the ROM.
************ TO FORCE CHECKPOINT IN DATA PARTITION PLEASE DO THIS IN TERMINAL *******************
su
echo 6 > /sys/module/yaffs/parameters/yaffs_auto_checkpoint
sync
This will change auto checkpointing to 2 and create a checkpoint.
This is just temporary, the checkpoint is deleted in reboot so the commands have to be done before every reboot.
One solution is to use Gscript.apk and type the commands in Gscripts. Execute via Gscript before reboot
**** To verify if you have checkpoint in Data Partition Please do in Terminal ******
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
scroll down and look for "blocks_in_checkpt" should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)
Update 2/7/2011
******************EXT4 FILESYSTEM FOR SD INSTALL***************
Because of bad blocks on my Kaiser, I have to give all NAND partition to install on system while installing data on SD with EXT4 filesystem. Here are the requirements:
1. An SD card with at least class 4
2. A way to partition and format your SD with FAT/ext2/Ext4 (ext2 should be as small as possible), I use Ubuntu for this.
3. Kernel with ext4 support (this will format your whole NAND partition to use system and use EXT4 on SD for Data)
Instructions:
1. FORMAT SD (I use Ubuntu DiskUtil) with FAT/ext2/EXT4, on my 8GB SD I use 7gb for FAT/8mb for ext2/1gb for EXT4 but it's your call. The most important part is that you only give the second partition (ext2) the least amount of size.
2. Download the KAISIMG.zip, extract and you will get a file KAISIMG.NBH
3. use Atools to edit accordingly with your Kaiser's screen type and keyboard (NBH is using screen 2 and keyboard 1)
4. Flash the KAISIMG.NBH
HERE IS WHERE IT GETS TRICKY!!! PLEASE FOLLOW THESE INSTRUCTIONS CAREFULLY!!!!
a. PRESS CENTER D PAD to enter ANDROID Installer
b. Choose (*) Sys on NAND
Choose (*) Data on SDCard Partition
c. Scroll Down to Install System and choose it
d. This will wipe your Existing system! Are you sure CHOOSE (YES)
e. Would you like to clear data too (Suggested) CHOOSE (NO) ***DO NOT CLEAR DATA OR THIS WILL CHANGE YOUR THIRD PARTITION TO EXT2 and will not work.
That's it. After installation reboot and you have an optimized ext4 on your SD for data. The mount options for ext4 in this kernel are:
data=writeback, barrier=0, nobh, noauto_da_alloc
Will use barrier=1 for better data security on next release, just enjoying the speed of barrier=0 for now
clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
Yes, Clemsyn If it's not a big problem please update 2.6.25 too. Thank you.
updates kernel 2.6.25
clemsyn said:
Will try to update our 2.6.25 kernel too if I get some space for my hard drive.
Click to expand...
Click to collapse
This would be nice. Thank you
mt1976 said:
This would be nice. Thank you
Click to expand...
Click to collapse
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.
Finally got a checkpoint in my userdata partition...instructions posted on first post, no update on yaffs needed
clemsyn said:
It would take awhile though since I have to remove my 2.6.32 kernel to put 2.6.25 since I'm limited in space. Donations for a laptop HD would be appreciated
UPDATE:
1. I think one of the reasons of data failure is that YAFFS refuses to write a checkpoint for our data partition. It writes it in the system partition with no issues. The new yaffs2 (updated 1-27-11 which I will update shortly) allows forcing checkpoint so I'll try that one and see if I can create a checkpoint on our data partition.
Click to expand...
Click to collapse
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol
aceoyame said:
I have a 20 GB Sata laptop drive I could get to you. Used to be in a 360 and then broke the enclosure open for a hdd when the one in my laptop died. Drive works just fine, just small lol
Click to expand...
Click to collapse
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).
clemsyn said:
Sorry about your phone. I was hoping you could test out my new kernel. ANyways, I think yaffs2 was having issues creating a checkpoint on the data partition since yaffs was only creating a checkpoint on proper unmount (which was not done in our kaiser). The latest yaffs can make us force a checkpoint by simply typing the command in the first post and typing "Sync" on terminal automatically creates a checkpoint in data partition. I'm currently testing it now and seeing how it goes.
BTW, I'll take that donation re:HD since I can use it as an external HD and put the 2.6.25 kernel there and improve it (that is if wife permits).
Click to expand...
Click to collapse
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.
aceoyame said:
lol she's really not that strict, she just thinks my development for our devices is a waste of time. Just PM me where to send it and ill get it out to you.
Click to expand...
Click to collapse
Agreed, that's what my wife thinks too I'll PM you my location
BTW, please make sure that the location of yaffs_auto_checkpoint is in /sys/module/yaffs/parameters (some ROM might have it on modules).
UPDATE:
Checkpointing is working great for me so far. Command on first post has to be executed in every reboot to create a checkpoint so typing the commands in first post before rebooting will create a checkpoint and save data partition (me thinks).
You can install Gscript and run the script before reboot
To verify if you have yaffs2 checkpoint in the data partition Type the following in terminal
su
cat /proc/yaffs
Scroll down to Device 2 "userdata"
blocks_in_checkpt should be 1 or 2 (not 0, if it is zero then you have no checkpoint in data partition)
Can anyone help adding this to the shutdown script?
I've done 5 restarts, 2 battery pulls and 2 resets, made sure data partition was checkpointed before doing so....So far data partition has been OK with no FC's or errors, no lost apps with 18mb's left in data partition The thing that surprises me is that Phone Storage size hardly changes in size in every reboots so it's good news I have a feeling we finally nailed down this data corruption issue, but only time can tell since I will shift to prolong use before doing any restarts...
I have sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion
clemsyn said:
sent a PM to scooter hoping he can help me with a shutdown script so we won't need a Gscript. So far, Gscript is doing it for me. Upong reboot, I run the script and make sure I have a checkpoint in data partiion
Click to expand...
Click to collapse
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.
scooter1556 said:
I've got it working in a script, i have it in a script which is run on startup and then again on a successful shutdown. I'll upload a couple of updates in a bit
EDIT: I have attached 2 updates to the post, one is for cyanogenMod based builds and the other is for all other builds such as fresh froyo etc.
Clemsyn, i have the latest yaffs source compiled into the kernel, i will commit it to git in a bit. With the latest yaffs a checkpoint is created every time sync is executed which makes scripts easier!
Click to expand...
Click to collapse
Thanks scooter, I'll try it out. I got stuck on a fake vsync and can't get out of it that I was unable to create a checkpoint and upon reboot without a checkpoint, everything was messed up. This is perfect, getting a checkpoint on startup and another in shutdown Thanks a lot.
Krazy-Killa said:
I downloaded the CM update, and it didn't enable the checkpoint, still had to enable it manually through terminal.
Testing the kernel right now however, and so far liking it.
Click to expand...
Click to collapse
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?
scooter1556 said:
That's weird! So it didn't make a checkpoint on startup? I'm running the same script and init.rc and it works fine. I did push the file with adb though, did you fix permissions after updating?
Click to expand...
Click to collapse
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.
Krazy-Killa said:
Ooooooh.... Probably not, lol. I'll do that, and edit this post with my results.
*EDIT* Ok, I feel stupid. I still had an old androidupdate.tgz file in my sdcard andboot folder, and the boot menu used it instead of the correct .tar file... Hm, that explained why my Market stopped working, lmao.
Click to expand...
Click to collapse
lol, rookie error
OK, it works on boot up since I get a checkpoint 1 every boot not sure about shutdown but if scooter says it works, I bet you it works in shutdown too
Scoot, I'm going to try to do a git diff and post it here so all the changes I've made in the config will be there.
Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^
Krazy-Killa said:
Even with that little debacle, I had 0 corruption of data, or missing apps in my /data partition, and had to reboot 4-5 times, each time I used GScript to execute the command manually before shutdown.
And camera still works. ^.^
Click to expand...
Click to collapse
That's great news and if my memory serves me right, you easily have data corruption in shutdowns and reboot. IMO, scooter's script is the best option we have for this issue and you don't have to remember to use gscript (especially from an old person like me with slight dementia Anyways, I'll continue on with my testing with this script. If it is still working after a few days, I'll try to add Namespace support, CIFFS, PPOE in our kernel.
BTW, here is my git diff and .config file. Please put it up in GIT. Thanks.
Y.G. said:
I formated my sd card to Ext4 and when insert it in to my phone, it says that's it's blank and has unsupported files. Any reasons for that?
Sent from my SPH-L710 using xda premium
Click to expand...
Click to collapse
The SPH-L710 Samsung Stock LJ7 TW 4.1.1 Android doesn't understand/support the change to ext4 external SD card (microSD) without a few things being done.
I'm working this out right now. So far I have manually been able to mount the newly created ext4 partition on the microSD card through adb, and after some chown/chmod I was able to go back to "Settings and Storage" and the "Mount SD Card" picked it up, and I was up and running ext4. But this didn't persist after a restart. So I'm looking into: /etc/vold.fstab MODS to keep it after restart right Now !!
If Some one else already has this perfected please chime in. I'm wanting to do most of the devices in the house this way when I get time because better performance, having a file system with a journal, and getting rid of thins like 4 Gig per file limitations is pretty Sweet in my humble opinion *Grin*
0) Assuming you already have your microSD card formatted ext4. I also happened to label mine extSdCard for the volume label within gparted
1) Can mount with:
mount -w -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard/
2) To get the correct owner and permissions run:
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
3) Should make the extSdCard owner/permissons match the regular internal sdcard you can verify this like so:
cd /storage/ && ls -l
drwxrwxr-x root sdcard_rw 2013-01-12 18:16 extSdCard
drwxrwxr-x root sdcard_rw 2013-01-12 17:05 sdcard0
4) After that you can go to the "Settings and Storage" to run "Mount SD" and you will have ext4 extSdCard Show up and it bring up the File System Status !! --> Until you reboot and it goes to crap because I don't have the vold.fstab edit/MOD complete _yet_ ... So, for now a boot script has been put in place to bring our external SD card back online during restart, so the system will acknowledges it, making the world a better place.
Example of how things look file system wise: mount | grep extSdCard
/dev/block/mmcblk1p1 /storage/extSdCard ext4 rw,relatime,user_xattr,barrier=1,data=ordered 0 0
5) Have not been able to resolve the vold.fstab to make this ext4 extSdCard matter fully Legit (in my opinion), but I did manage to make it remount the card on boot, so its online when the system comes up instead of having to manually mount it. Did this by -->
Added the following lines to the very bottom of: /etc/init.qcom.post_fs.sh
## sponix MOD to match with ktoonz kernel for better power management
stop mpdecision
## sponix MOD to mount extSdCard prior to GUI work around to make ext4 function
## read and write extSdCard mount
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
mount -w -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard
chown root:sdcard_rw /storage/extSdCard
chmod 775 /storage/extSdCard
## if you want read only extSdCard mount
## mount -r -t ext4 /dev/block/mmcblk1p1 /storage/extSdCard
Still attempting to automate the process so the Stock+root LJ7 can pick up the extSdCard _normally_ without having to do the mount command manually, but so far its kicking my butt. Also this is more a "General, or Question and Answer type Topic" the Kernel(s) obviously support ext4 the system fs uses/requires it *Grin*.. So we might get Our Friendly Neighborhood Moderator to Migrate it to the proper place to help others. Just hoping to get the last few bumps smoothed out, or find someone that already documented the process that I've overlooked *Grin*..
Current Known Issues: If you unmount the card through the "Settings | Storage | Umount SD" or by hand with umount, you will either need to reboot for it to reattach through the /etc/init.qcom.post_fs.sh boot script script addition, or will have to mount it manually if you want to keep the system up and running. Guess you could also probably just run the /etc/init.qcom.post_fs.sh as root from a terminal emulator (or adb).
Still searching for vold.fstab bits of wisdom but that will have to continue next weekend -->
Sexy and You Know it,
Keep on Flashing,
sponix2ipfw (sponix
:fingers-crossed:
Ha! Sorry. Deleted: Didn't understand that you had it running on boot (can't read properly )
Great idea
Am I really the only one who also thinks this idea is the nuts?
Am I the only one who longs to transform the mess that passes for a filing system on the internal sd using symbolic links into a beautifully organized, encrypted and cloud synced system on my external sd?
Is it just me and a few others that want to be able to achieve the above so that we can move from one ROM to another or recover from a lost phone with the minimum of fuss?
Are we freaks? :cyclops:
Say it isn't so XDA!! :crying:
I'm gonna try this on my international S3 running Null_ Rom 25 JB 4.1.2
PS do you have any idea how the entire ExtSD or just a folder can be enrypted using Cryptonite and automatically mounted at boot time?
emp111 said:
PS do you have any idea how the entire ExtSD or just a folder can be enrypted using Cryptonite and automatically mounted at boot time?
Click to expand...
Click to collapse
Is this along the lines of what you're looking for?
http://forum.xda-developers.com/showthread.php?t=1141467
Also your idea is pretty insane, but also genius
If you get that to work please do come back and share
Insane ideas are the best lol
CNexus said:
Is this along the lines of what you're looking for?
http://forum.xda-developers.com/showthread.php?t=1141467
Also your idea is pretty insane, but also genius
If you get that to work please do come back and share
Click to expand...
Click to collapse
Thank you for your prompt reply, yes it is asking a lot I know but I think that it can be done.
Now if you really thought that idea was insane......check this out:
Imagine that we asked every Android app developer to submit the various paths used for their config (and config backup) files to a central database and had the ability to add or own custom paths (which could be added to the central database once approved).
We could build an script/app that would retrieve a list of currently installed apps on your phone then automatically build a symbolically linked file system (and/or backup file system) in the location of your choice that you could either encrypt and/or sync using your current tools or even incorporate this functionality into the app itself along with the ability to choose what was encrypted/backed up and how i.e. either synced to the Cloud or (S)FTP or SMB as either a dd copy or even a cwm flashable zip.
Could I dare hope for a Tasker module or the ability to add custom scripts?
I wish I could do this myself but my coding skills are non existent
Anyway the LUKS manager app won't automatically mount a file system, but I really like it anyways, thank you for pointing me to it!
And on the Ext4 front, the mount command (yes the 1st one ) failed, maybe the op could offer a suggestion. :angel:
BTW is there a place for people to suggest ideas for apps here?
Wait really it wont? I couldve sworn I remembering that it did
But dude....
You need to learn yourself some java and start whipping stuff up
Idk about the whole central database thing, but the rest could definitely be done with root access
I think the main problem with that is the proprietary aspects...i mean even here on XDA where binaries released are supposed to be GPL compliant, many of them arent and its sad because it deteriorates the overall quality of work thats released afterward
This whole thing is just hard work!
CNexus said:
Wait really it wont? I couldve sworn I remembering that it did
Click to expand...
Click to collapse
Doesn't seem to unfortunately
But dude....
You need to learn yourself some java and start whipping stuff up :D :D[/QUOTE said:
You make it sound soooo easy lol, and at another point in my life maybe it would have been but right now I'm operating at a reduced level due to some unforeseen circumstances that have left me lacking focus, motivation etc
You know all the things you need to be creative, learn etc lol
Anyway back to the matter at hand, I have got my ext4 SD card to the stage where I have to manually mount it from within the Settings/Storage as I'm using the international S3 and don't have the init.qcom.post_fs.sh, I think the qcom refers to Qualcomm chipset in US S3's.
As for modifying vold.fstab so we can avoid the above workaround it would seem that maybe thats a dead end as according to a German guy on android-hilfe. de, Vold may have been modified by Samsung to only deadl with exFAT on External SD's.
Looks like I'm not gonna be in Android nirvana for a while :crying:
Unless anyone else on XDA fancies getting in on this !!!!
Click to expand...
Click to collapse
Got it working ..... kinda
Got an app called ezymount (by ezynow) that automounts my ext4 64GB microSD at boot time.
I have to wait a few seconds for the boot process to complete but it's automatic, am pretty happy!!
Now gotta get symlinks, encryption and cloud synchronization sorted :/
On ASUS stock rooted system, we can make use of fdisk to create a primary partition then use make_exr4fs to format that partition to ext4.
Once done and the /dev/block/mmcblk1p1 partition mounted as /storage/MicroSD it is possible as root to interact with it, but as non root you can only read data from there, not write.
How do you set selinux permissions correctly to allow writing without root ? The folder is set to 777, it cannot be a file permissions problem, it leaves us selinux policies to fix.
I tried to apply the selinux policy used for exfat external volumes but it still failed for writing.
Thanks to @Chainfire and Red Hat selinux documentation, I was able to resolve this without modifying the system, patching rules, disabling selinux or anything like that.
A two lines script can take care of this, requiring supersu and tools that are already on the device.
su --mount-master -c "mount -t ext4 -o discard /dev/block/mmcblk1p1 /storage/MicroSD"
chcon -R ubject_r:media_rw_data_file:s0 /storage/MicroSD
I'm still unsure to what each things are related to at 100%, but the context source of 3rd party application can write to media_rw_data_file , and this is what is applied on user's media folder inside data partition.
Some applications may still fail at understanding this is the external storage, but as my favorite file manager Fx does, that's what I want needed.
For some reasons the lost+found folder remains inaccessible, but the recursive part of chcon should take care of it on next boot.
Hope this will be useful to someone else.
Will your script work with another stock-ROM too? I've a Sony phone. Never had a phone I could get an ext4-formated sdcard working. Only with cyanogenmod installed ext4-formated cards work out of the box.
I'm a noob so I don't about scripting or understand details of file systems.
It will depend on where the Sony custom built system is mounting the micro sd card, if it mounts it at the same place, it will work.
But your message should have been a comment to my answer and not an answer proposal as you don't bring an alternative solution