I have a backup made with TWRP 2.6.0.0 of boot/data/system that were saved as .ext4.win files by booting into TWRP and doing a backup.
However, when I tried to browse the file using Nandroid Manager 1.4.1, most (but not all) of the app directories in /data/data/{packagename} are missing their actual data. They have a lib dir that's a <link>, but nothing else. However, some apps DO appear to have their normal and expected data. I could understand if NONE of them did... but I'm mystified as to how SOME could have data, but others that I know for a fact store data in /data/data/{packagename} are coming up empty.
I can think of a few possibilities:
* For some bizarre reason, TWRP didn't back up the files for certain apps. Unlikely, unless Illusion 4.2.2 (based on SlimRom/AOSP 4.2) was doing something weird with symlinks that confused it.
* Nandroid Manager 1.4.1 mangled the backup files while uncompressing them. Oh ${deity}, I hope this isn't the case. It never even occurred to me that it might try to uncompress the originals "in-place" and delete the original copy when I ran it and gave it the OK to uncompress them first.
* Nandroid Manager 1.4.1 uncompressed them fine, but for whatever reason can't explore the uncompressed backup properly.
I copied the .ext4.win files to Windows and tried finding something that can explore them, but I'm coming up blank. I tried ext2explore from Sourceforge, but it silently ignores me when I try to browse to the .ext.win file as a filesystem to mount.
Any ideas what to try next, either Android-side (Nandroid Manager 1.4.1 itself is the problem, and hopefully didn't mangle the backups themselves by attempting to uncompress them) or Windows-side?
Update: I have another theory. At the time I did the backup, my phone was bootlooping (it made it up to the converging-diverging RGB spotlight animation, then stayed there forever). To reboot, I had to 3-finger it (power+home+down or up, don't remember which one I used at the time).
I'm wondering whether it might be a case of TWRP backing up the raw image of a dirty ext4 partition that had uncommitted data in the journal, and the recovery tools that I'm using to explore it for individual files are blindly mounting it like an ext2 volume & causing any file written by an actively-running application to show up as "gone" instead of showing the old version.
If that's the case, would treating the .ext4.win file like the output of 'dd', then running an ext4 repair tool on it, be likely to fix it? I'm somewhat familiar with issues like this, but I've never actually had to deal with one head-on, so I'm a total n00b when it comes to the details of mounting a Linux filesystem from a dump file and allowing Linux to fix it as if it were rebooted into a dirty filesystem.
Related
My HD2 can use clockworkmod from the bootloader to create a backup of sys & boot. Even windows had sprite backup to make a image of the rom.....so what do tilt users have to backup their rom so they can get back to a good copy of it or try a new rom without losing everything?
famewolf said:
My HD2 can use clockworkmod from the bootloader to create a backup of sys & boot. Even windows had sprite backup to make a image of the rom.....so what do tilt users have to backup their rom so they can get back to a good copy of it or try a new rom without losing everything?
Click to expand...
Click to collapse
You can backup data from within install menu(dont remember how its properly called)
And system well if you have a flashed image on sd card than there is no point to back it up.
-Arturo- said:
You can backup data from within install menu(dont remember how its properly called)
And system well if you have a flashed image on sd card than there is no point to back it up.
Click to expand...
Click to collapse
Some of us change our system as well such as custom kernels, customized framework's and additional tweaks to files such as the build.conf etc we'd rather not have to keep doing over.
famewolf said:
Some of us change our system as well such as custom kernels, customized framework's and additional tweaks to files such as the build.conf etc we'd rather not have to keep doing over.
Click to expand...
Click to collapse
Well the thing with system is that after you flash it ..it stays in the memory card then you can move it(back it up) to your computer and change it accordingly. So for example you have changed the framework in the system on your phone.... you can do the same with the backed up system on computer and it will take you not more then 15 seconds. You just open it with 7zip and the world is yours you can pretty much do whatever you want with it remove/add files replace them and so on....and that is all the files and folders in the system image. If you have ever removed/replaced/added files to a simple rar archive then you pretty much know how easy it is to edit a rom on your computer.
backing up your kaiser
Android is just Linux so you can do almost anything you can do in Linux if you can get to the shell. I use a program called Terminal Emulator from a guy called jackpal that I got from the Market. It will give you a command prompt so you can perform any Linux commands you like and if you type su it will act as super user and give you a # prompt.
I also went as far as writing my own userinit.sh that runs each time I boot and copies files from my data partition on NAND to the SD card. I just copy the /data/data, /data/app, /data/local, and /data/system folders to a folder named backup on the root of the SD card. The script also moves the previous backup to a folder called backup.old and if backup.old already exists it deletes it first.
This way if my phone reboots and data is corrupt I can use Root Explorer to copy the files in either backup or backup.old on my SD card back to NAND and my phone is restored.
You can also us the backup routines that are in the menu that comes up when you press the DPAD during the boot process but that backs up the entire data partition to the tune of 256MB and if there is corruption it really doesn't do much good.
As a side note you can mount the file created by the backup on any linux box using the mount command and a loop device. Once mounted you can just copy files that you need out and transfer them back to your phone using the SD card.
Ok, these are specifically for MIUI, but they can easily be adapted to any other rom by editing the shell scripts. I'll explain how they work and it should be fairly apparent how one would need to modify for their particular rom/needs. Don't know if anyone will actually use these, but I thought I'd share. I also think it's a really good example of how CWM can be used for more than just flashing stuff.
So I've been running MIUI for a while now and I'm really happy. One little issue that popped up was every Friday when the rom was updated, the OTA update app icon gained a little "1" badge, indicating an update was available. While it's nice to be notified of the update, I occasionally didn't have the time or interest in updating so the little badge remained. I put that app in a folder, but then the folder had the little 1 badge. After a while, I started deleting the update app after install since I could always download and install updates through ROM Manager just as easily. Then I started deleting the update app from the downloaded ROM zip, along with several other apps I didn't want (Facebook, Twitter, the MIUI FileExplorer, etc) before flashing it. But this entailed downloading the Friday update on my phone, transferring it to my desktop, opening the zip in 7 Zip, deleting the unwanted files, uploading it back to my phone and then flashing. Quite a PITA.
Another issue I ran into was that the built-in MIUI backup app failed to backup my SMS/MMS messages. Actually, I don't know if it was failing to back them up, or failing to properly restore them, but at the end of the day, it wouldn't keep my text messages. I know I could avoid this by just installing each update over the current ROM, but I've always believed cleanly installing an OS helps avoid issues. I prefer to wipe everything, install clean and then restore my data.
So, the solution(s) I came up with was to mod the downloaded ROM zip file right on the phone, and manually backup and restore my data. While it took a little while to get set up, I find it actually works considerably faster and smoother than using MIUI's backup/restore app. I now have 3 simple shell scripts that are run through Clockwork Recovery just like installing any zip - one to mod the ROM after downloading, and two to backup and restore my data. Here they are:
MIUI-Mod.zip
This contains a shell script and a zip binary to handle the adding and deleting of files to/from the downloaded rom zip. When run through CWM, it copies zip and the script to /tmp and runs the script. Because the ROM zip is named differently every Friday, I figured the easiest solution was just to rename it manually. So, this script assumes the ROM zip is named "miui.zip" and is located in the root of the SD card. It's a simply matter to download the ROM on Friday, move it to /sdcard and rename it miui.zip. Here's what the script does in 2 commands:
/tmp/zip -d /sdcard/miui.zip /data/app/Twitter.apk /data/app/Facebook.apk /system/app/Email.apk /system/app/FileExplorer.apk /system/app/Gmail.apk /system/app/MIUI.us_Forums_1.1.2_AdSupported.apk /system/app/Superuser.apk /system/app/Updater.apk /system/app/MIUIStats.apk /system/media/lockscreen/* /system/media/wallpaper/*
/tmp/zip /sdcard/miui.zip /system/app/Gmail.apk
Click to expand...
Click to collapse
The first line simply opens the zip and deletes the listed files: Twitter, Facebook, Email, etc. It also removes all the wallpapers for the lock and main screens (I don't use them, so I didn't want them, but it's a good example of the use of wildcards like *).
The second line adds in the Gmail.apk from my current ROM. The latest versions of the Gmail app included in MIUI lock the mail database, which conflicts with a widget I use, so I just replace it with an older version. Every time I run the script, it removes the included Gmail app and adds in my current one. If you didn't want to add anything to the ROM, you could just delete this line entirely.
Data-Backup.zip
This just runs a shell script that creates a tar file in the root of your sdcard containing the listed files and directories. Tar files retain file permissions and owner information, so everything can be restored seamlessly. Here's how it works, it's pretty simple:
tar -c -f sdcard/backup.tar data/app/com.adobe.flashplayer-1.apk data/data/com.adobe.flashplayer data/app/com.speedsoftware.rootexplorer-1.apk data/data/com.speedsoftware.rootexplorer data/app/com.touchtype.swiftkey-1.apk data/data/com.touchtype.swiftkey data/data/com.android.browser data/data/com.android.deskclock data/data/com.android.launcher data/data/com.android.mms data/data/com.android.phone data/data/com.android.providers.contacts data/data/com.android.providers.settings data/data/com.android.providers.telephony data/data/com.android.providers.userdictionary data/data/com.android.thememanager data/data/com.android.vending data/data/com.google.android.gsf data/data/com.miui.camera data/data/com.miui.notes data/data/com.noshufou.android.su data/system data/misc/wifi
Click to expand...
Click to collapse
This creates the tar file and includes Adobe's Flash player and it's data, Root Explorer and it's data, and SwiftKey and it's data (I paired it down to just these three for this example). After that, it backs up a bunch of other data directories which include the current Browser settings, alarms, launcher settings and home screen layouts including widgets, phone logs, the current theme settings, any MIUI notes, current Wifi connections and, yes, SMS and MMS messages.
Obviously, you'd want to add in your specific apps and data directories. Sure, it takes a little work to set this up, but as I said, backing up is very quick and easy afterwards.
Data-Restore.zip
This just runs a shell script that restores all the files from the backup.tar on your SD card:
tar -xf /sdcard/backup.tar
Click to expand...
Click to collapse
Doesn't need much explanation.
So, the process is:
1-Download the Friday update through ROM Manager, move it to the root of my SD card and rename it to miui.zip
2-Reboot into CWM and run Backup-Data.zip to back up my apps and data
3-Run MIUI-Mod.zip to modify the downloaded MIUI rom
4-Wipe everything (cache/data/system)
5-Install the modded MIUI rom
6-Run Data-Restore to restore apps/data
7-Reboot
It's literally about 5 minutes after downloading the ROM and I'm updated and rebooted with a CLEAN os install and all my apps, data and settings intact, my home screen layouts, keyboard, wallpapers, wifi settings and everything retained.
Some notes:
1-Before running these in CWM, you have to make sure you have system, data and sdcard mounted since the scripts will be reading and/or writing to all of these. When you first reboot to CWM, just go to mounts and manually mount each one. The scripts could certainly be set up to mount them automatically, but mount points vary from phone to phone, and it's not difficult to mount through CWM. Also, after flashing a ROM, the ROM install script may unmount some of these, so before you restore your data, you might have to manually remount.
2-If you're adding anything from your current ROM to the zip (like how I add in my current Gmail.apk), make sure you wipe everything *after* running the mod script, otherwise you're deleting everything and then trying to copy file(s) that were just deleted.
3-I would highly recommend using Notepad++ to edit scripts, since it properly saves Unix line breaks.
4-as always, use at your own risk and a full CWM backup is highly recommended before messing with these.
That's pretty much it. Again, it does take a little while to get set up, but once it is, it's really simple. If you install a new app between updates, you just have to edit the backup script, or you can just keep the scripts maintaining a base setup and reinstall various apps through the market each time. This specific process is probably most useful for ROMs that are updated regularly - nightly or weekly, although I'm sure people can find other uses for running scripts through CWM.
Hope someone might find these useful.
This is really cool. I run Miui and would really like to be able to do a fresh install each week, but hate the hassle of resetting up my phone, even using Titanium backup, it's still a time consuming process. So needless to say, I flash the new rom over top of the old version without wiping just to save time.
So, basically I can open up the data restore zip in 7 zip, add in a few of my apps/data such as Tasker and so forth, close it and flash? If so, that will be great.
Thanks for sharing this!!!
You don't need to touch the Data-Restore.zip, all that does is extract the backup.tar file created with the backup script. What you'd do is open the Data-Backup.zip and edit the backup file (it's just a text file). Add in any of your apps and their corresponding data directories and when you flash the Data-Backup.zip in CWM, it'll back up those files/folders.
To find the files you want to backup, use something like Root Explorer and look in /data/app. There you'll find all the 3rd party apps you've installed. Add each one to the tar command in the backup file just like the ones I included. Then go into /data/data and find each app you want to back up's corresponding data folder. Add them to the tar command too.
Save the file, re-zip it and when you flash it through CWM, it'll back up the new files.
Just what i wanted
I also always install everything from clean system so setting up everything is a pain.
Would this work for Internal storage? I have a Motorola RAZR XT910 and running JBX roms.
Google does install all the apps when i sign in into the new rom but then i always had to move each folder to the correct destination manually using some file explorer. was thinking of making something like this but didnt know how to start. Thanks, I will now try and mod these to suit my phone :good:
Hi,
I flashed last night Temasek's CM11 4.4.2 ROM (http://forum.xda-developers.com/showthread.php?t=2596887), wiping user data, caches, etc (I was on a rooted stock 4.1.2 ROM).
Everything is working great so far, I love this ROM, I just noticed a problem that it's quite annoying: it seems that the internal user data file system (which would be the /sdcard folder in my old ROM) is all messed up and replicated. Here;s what happens:
I created a backup of the current ROM with Online Nandroid backup. If I use a file explorer from my phone (X-plore) and go to the "partition" /storage/emulated/0 I can see the backup created in /storage/emulated/0/clockworkmod/backup/.
However, if I connect the phone to my PC and browse the contents, the partition that I see isn't that same partition, it's an older one with older files (for example it has a clockworkmod folder but the backup isn't there, and some folders are different). What;s worse is that I can't find this "old" partition when browsing my phone with X-plore. These are the palces where I've looked at:
- /storage/emulated/0 (shows the partition where the backup I just created went, I can see the backup there)
- With root privileges:
- /data/media (shows an older second partition with different folders)
- /data/media/0 (shows the "right" partition, where the backup is)
- /sdcard (again, shows the "right" partition)
- /storage/sdcard0 (same here, shows the "right" one)
So to sum up, it looks like I see at least 3 diffent partitions:
1. From my PC, I see an older one that I can't see from X-plore (I tried Cyanogenmod file explorer too)
2. From my phone, I see the one that has the backup I just created
3. From my phone too, I see a second old partition (different from the one I see from my PC)
So what I would like to know is:
1. How can I transfer the backup to my PC if I can't see the partition that holds it from there?
2. Is it safe to just delete all those folders in both old partitions? I'm sure they're taking space unnecessarily
Any help would be much appreciated!
After flashing any CM/AOSP Rom. They actually create a different folder set up. That's because of the ability of different users...
You need to apply the restore SD card after downgrade.
Found HERE:
http://forum.xda-developers.com/showthread.php?p=36873319
After that^^^^^^^^^ just make sure you copy everything you want.. Then you can delete all those other folders you don't need.
SENT FROM MY SAMMY i317 USING TAPATALK
Bajanman said:
After flashing any CM/AOSP Rom. They actually create a different folder set up. That's because of the ability of different users...
You need to apply the restore SD card after downgrade.
Found HERE:
http://forum.xda-developers.com/showthread.php?p=36873319
After that^^^^^^^^^ just make sure you copy everything you want.. Then you can delete all those other folders you don't need.
SENT FROM MY SAMMY i317 USING TAPATALK
Click to expand...
Click to collapse
Thanks for the reply! I'll try this as soon as I get home!
Bajanman said:
After flashing any CM/AOSP Rom. They actually create a different folder set up. That's because of the ability of different users...
You need to apply the restore SD card after downgrade.
Found HERE:
http://forum.xda-developers.com/showthread.php?p=36873319
After that^^^^^^^^^ just make sure you copy everything you want.. Then you can delete all those other folders you don't need.
SENT FROM MY SAMMY i317 USING TAPATALK
Click to expand...
Click to collapse
Hey! So it seems that did it!!! Now I can see ALL the contents in the /sdcard partition (X-plore shows it as /storage/emulated/0)
So my question is (I'm still pretty noob at all this): all the files/folders shown there shouldn't be critical for the OS to work, right? They should all be stuff saved by apps, that if they get deleted the worst thing that will happen is that those apps will lose that info, right? I ask because I can recognize most of them, but not all of them, so I'm not sure, and I would like to remove as much stuff as possible.
One file that makes me doubt specially is /sdcard/clockworkmod/download/download2.clokworkmod.com/recoveries/recovery-clockwork-touch-6.0.4.3-t0lteatt.img. Is that the actual CWM recovery that I have in my phone? Or just the installation image that can be removed?
Thank you very much for your help!
I ended up deleting everything from the sdcard partition and restoring the stuff that mattered to me, and everything seems fine now. The only weird thing is that when I create a backup with Online Nandroid backup (which gets stored in /sdcard/clockworkmod/backups), I can't see it when connecting my phone to my PC. I can see everything else in the sdcard except the backup. I've tried unplugging and plugin again the phone, rebooting it, rebooting my PC... but nothing. However, if I zip the backup folder with X-plore, leaving the zip file in the same folder as the backup folder, my PC can see it...
Weird thing but oh well!
Deleted op
Ok, now I'm thinking that it has nothing specifically to do with those roms, rather with having installed the TWRP app as I was rebooting from TWRP into the fresh installs.
It is my belief that TWRP somehow absorbed my TWRP folder in the microSD. There is still a (vacant 3.7GB) system image bouncing around in the 'sdcard' or internal storage, though the original TWRP restorable backup has been replaced by a -500 hats- folder called BACKUPS, inside the TWRP folder.
All happening after flazing the SuperSU zip
I had originally installed and started TWRP as system read only, and created the TWRP restorable and also an image backup, which may be what I sill find kicking around. Currently flaking around with this system_image.emmc.win image file which Windows believes is about 3.9GB but unzips after renaming it, to a 58.8 MB [SYS] folder containing the 61.2MB 'Journal' file and a lost+found empty 0.0 kb folder. Which according to theandroidsoul.com/htc-one-m9-twrp-recovery-issue-how-to-backup-and-restore-stock-system-fix/ (would have been) going to be possible somehow to restore wit.
Totally FUpped. Some link to micro SD probably the blame for the missing backup and B. Cubbins BACKUP folder, after so many data format yes's because of this bs encryption we all dealin wit.
nothing good to report with this phone. That backup was the only thing it had going for it. And that got deleted by ????'s ghost
I tried
adb backup -all
but the resulting file is 60MB, but should be 100GB.
Edit: Some more information:
I'm running HavocOS, but the apps on this Rom are unstable. They are stable for almost 24h every time I flash most recent GApps, but by now I don't really think it's GApps anymore.
Aside from EVERY app being unstable copying files is not stable either. I just copied 150GB of images and videos from internal as well as "external" SD-card to my computer without any interrupt and that worked without any issues. If I try to copy a file from internal to external SD card within the phone, then this works in one out of three tries for every single file.
So next thing is try another ROM, then maybe an older ROM and then search for Hardware defects, before I want to try next ROM I want to create a backup.
There is A LOT of config files here and alot of them are linked to images all over the place, so I will likely have to switch to the backup a few times because if I just save the parts part by part now I will 100% forget a few important things.
This seems like a job for TWRP backup. Oh and btw, I did a TWRP-backup just when I migrated to this ROM, but it is defect! I tried to create a full backup withing TWRP >10 times today and yesterday and it all failed. Oh yes, moving files withing this device is buggy right now. But I can do it via adb, can't I? How?
andy01q said:
I tried
adb backup -all
but the resulting file is 60MB, but should be 100GB.
Edit: Some more information:
I'm running HavocOS, but the apps on this Rom are unstable. They are stable for almost 24h every time I flash most recent GApps, but by now I don't really think it's GApps anymore.
Aside from EVERY app being unstable copying files is not stable either. I just copied 150GB of images and videos from internal as well as "external" SD-card to my computer without any interrupt and that worked without any issues. If I try to copy a file from internal to external SD card within the phone, then this works in one out of three tries for every single file.
So next thing is try another ROM, then maybe an older ROM and then search for Hardware defects, before I want to try next ROM I want to create a backup.
There is A LOT of config files here and alot of them are linked to images all over the place, so I will likely have to switch to the backup a few times because if I just save the parts part by part now I will 100% forget a few important things.
This seems like a job for TWRP backup. Oh and btw, I did a TWRP-backup just when I migrated to this ROM, but it is defect! I tried to create a full backup withing TWRP >10 times today and yesterday and it all failed. Oh yes, moving files withing this device is buggy right now. But I can do it via adb, can't I? How?
Click to expand...
Click to collapse
try using the migrate app
Which migrate app?
andy01q said:
Which migrate app?
Click to expand...
Click to collapse
The app's name is migrate..search on playstore or xda thread
Got it, thank you.
It's not working though for three reasons:
1st: It's not a full backup. I got what I need for migrating, I want a full backup to return to the current state including filesystem, tree and random files. Ideally A full copy of all filesystems which wouldn't fit onto any of the filesystems even if this was what the app did.
2nd: It's not via ADB. Backing up from inside the phone doesn't work as I already wrote.
3rd: Doesn't recognize my SD card (yes it's mounted as R/W) I don't know why I even checked that as the app is useless for my case anyway.