Develop Bugs

Best way to create and re-flash userdata image on latest N (ryu-nrd91n)

First of all, thanks so much to everyone for all the hard work creating and updating utilities for the Pixel C.
I have a rooted (but otherwise stock) Pixel C running the November update of Android 7/N (ryu-nrd91n) and am trying to:
1. Create an image of the userdata partition
2. Read/modify files
3. Re-flash the modified partition back to the Pixel C
So far I've been able to create a 24.8 GB image file using 'dd' but the file is unreadable, probably due to encryption.
I don't see any Developer Settings option to remove encryption or otherwise unencrypt the data partition.
What is the best approach?

jason_hanley said:
First of all, thanks so much to everyone for all the hard work creating and updating utilities for the Pixel C.
I have a rooted (but otherwise stock) Pixel C running the November update of Android 7/N (ryu-nrd91n) and am trying to:
1. Create an image of the userdata partition
2. Read/modify files
3. Re-flash the modified partition back to the Pixel C
So far I've been able to create a 24.8 GB image file using 'dd' but the file is unreadable, probably due to encryption.
I don't see any Developer Settings option to remove encryption or otherwise unencrypt the data partition.
What is the best approach?
Click to expand...
Click to collapse
Do you want to get rid of your encrypted data partition ?
- Format data (inside TWRP - and do not boot into system in between)
- Flash stock boot.img
- Install SuperSU.zip / or flash antother boot.img - with "encryptable" flag.
If you flash stock boot.img and boot into system your are encrypted again - automatically.
yes, "dd" will dump the complete partition including encryption ..
you may can "tar" the complete file system while decrypted .. similar to TWRP backup function.
Why do you want to "backup - modify - restore" at all ?
Why you don´t modify your files directly on data using root-explorer ?
Cheers

followmsi said:
Do you want to get rid of your encrypted data partition ?
- Format data (inside TWRP - and do not boot into system in between)
- Flash stock boot.img
- Install SuperSU.zip / or flash antother boot.img - with "encryptable" flag.
If you flash stock boot.img and boot into system your are encrypted again - automatically.
...
Why do you want to "backup - modify - restore" at all ?
Why you don´t modify your files directly on data using root-explorer ?
Click to expand...
Click to collapse
Thanks so much for the follow-up. The main goal is actually simply to be able to create images, and re-flash them reliably on a different Pixel C (replicating the exact system state). The modification step is simply to verify that it's working.
I was able to unencrypt /data using the method above -- Thanks, that's extremely helpful!
However, I'm still unable to mount/read the file produced by "dd if=/dev/block/mmcblk0p7"
Is there a better way you can recommend to make an image of the userdata partition?

How do you mount the unencrypted dd dump of data ?
mount -o loop dumped.data.img /mnt/data
As well you can untar all twrp backup files into one new data.img file.
Or tar the complete /data partition ... etc.
It's an ext4 partition ..
Of course, you can use the twrp backup files directly to restore on another pixel c device.
There is no modification needed at all.
Good luck!
Sent from my Nexus 10 using Tapatalk

That whole boot.img step.. is that why I always get stuck in a weird misplaced bootloop whenever I modify anything in TWRP? I just flash original boot.img in fastboot?

gnome9er said:
That whole boot.img step.. is that why I always get stuck in a weird misplaced bootloop whenever I modify anything in TWRP? I just flash original boot.img in fastboot?
Click to expand...
Click to collapse
The boot.img step to decrypt data ?
It's a one time task.
Once decrypted you always need to root after new boot.img flash.
If you don't root or don't use encryptable flag boot.img you will get encrypted.
Not really related to bootloops.
What are you changing when you are using TWRP ?

If I use backup then restore it loops. If I flash twrp it just boots to recovery. Now that i think about it i never did root this thing properly. Well i did with a toolkit but that doesnt count. I went beta then hated it lost root on restore then when i tried to do it again i kept running into bootloops. It says in the forums here that its common ans to just fastboot reboot but that dont work. I cant wait for the new nexus 7

gnome9er said:
If I use backup then restore it loops. If I flash twrp it just boots to recovery. Now that i think about it i never did root this thing properly. Well i did with a toolkit but that doesnt count. I went beta then hated it lost root on restore then when i tried to do it again i kept running into bootloops. It says in the forums here that its common ans to just fastboot reboot but that dont work. I cant wait for the new nexus 7
Click to expand...
Click to collapse
Depends what are you restoring
Do you backup / restore other partitions except the data partition ?
Pls don't
In general .. pls backup data partition only !
All other .img files should be taken freshly from image.
In systemless rooting world the su.img is placed on data.
You may have an older version inside your backup.
If you restore boot.img from backup, supersu ramdisk modifications will fail.
Flash what ever system.img, vendor.img and install twrp as recovery.
If you flash a boot.img, it should have stock ramdisk, to be able to let supersu.zip do the work for proper rooting.
At the same time the su.img on data will be refreshed too.
To summarize .. if you change something always install a fresh copy of your boot.img and run SuperSu.zip inside TWRP afterwards!
Hope this helps ..

followmsi said:
Depends what are you restoring
Do you backup / restore other partitions except the data partition ?
Pls don't
In general .. pls backup data partition only !
Click to expand...
Click to collapse
In this case, we're only looking to backup and restore the data partition.
Once we have the raw image created by dd, what would you say is the most reliable way to convert it to something that can be flashed by fastboot?
I've tried several different utilities to convert it to a "sparse image" but flashing it just erases data and doesn't copy anything from the image.

jason_hanley said:
In this case, we're only looking to backup and restore the data partition.
Once we have the raw image created by dd, what would you say is the most reliable way to convert it to something that can be flashed by fastboot?
I've tried several different utilities to convert it to a "sparse image" but flashing it just erases data and doesn't copy anything from the image.
Click to expand...
Click to collapse
Did you test img2simg as well ?
I use simg2img to convert the sparse to raw image, the other way around.
Or something like this below .. "s" defines sparse image.
make_ext4fs -s -l xxxxxxxx -a data data.img
Sorry, can't help you better now ..

followmsi said:
Did you test img2simg as well ?
Click to expand...
Click to collapse
Yes, unfortunately we've tried everything.
First we created the file data.dd using
Code:
dd if=/dev/block/mmcblk0p7
, which can be read as an ext4 image.
Then we tried using SparseConverter, ext2simg and img2simg to convert the raw image to a sparse image.
They all produce files of different lengths, and none of those files seem to flash successfully
They go through the flash process and report being successful, but the data isn't there.
Is there any way to debug this and figure out what's going wrong?

followmsi said:
Do you backup / restore other partitions except the data partition ?
Pls don't
In general .. pls backup data partition only !
Click to expand...
Click to collapse
Why should other partitions not be backed up? Is it not possible to flash the /data partition if you flash other partitions at the same time?

jason_hanley said:
Yes, unfortunately we've tried everything.
First we created the file data.dd using
Code:
dd if=/dev/block/mmcblk0p7
, which can be read as an ext4 image.
Then we tried using SparseConverter, ext2simg and img2simg to convert the raw image to a sparse image.
They all produce files of different lengths, and none of those files seem to flash successfully
They go through the flash process and report being successful, but the data isn't there.
Is there any way to debug this and figure out what's going wrong?
Click to expand...
Click to collapse
If you don´t get an error message from fastboot the sparse image seems to be valid.
How do you verify data partition after flashing ?
Are you able to mount and check ?
Also from recovery .. e.g. before you boot into system ?
For example on Nexus 7 flo there is still a userdata.img inside factory rom (6.0.1), but not for Pixel C anymore.
Maybe some permissions are required (file_contexts) - like on Nexus 7 flo too.
Sorry, no idea whats going wrong for you ..
---------- Post added at 12:49 PM ---------- Previous post was at 12:35 PM ----------
jogrimst said:
Why should other partitions not be backed up? Is it not possible to flash the /data partition if you flash other partitions at the same time?
Click to expand...
Click to collapse
You can backup all partitions you want .. but you should know the possible impact if you restore them.
And Yes - you can flash all partitions at the same time .. but normally you don´t flash data partition via fastboot - not on Pixel C.
In times of systemless rooting and monthly security releases there is no need for system.img backup anymore.
Unless you have special modifications inside your system.img.
If you were rooted before I recommend to use fresh boot.img and reinstall supersu.zip after change.
Just to prevent issues ...

followmsi said:
And Yes - you can flash all partitions at the same time .. but normally you don´t flash data partition via fastboot - not on Pixel C.
Click to expand...
Click to collapse
Why is userdata.img normally not flashed via fastboot on Pixel C?

jogrimst said:
Why is userdata.img normally not flashed via fastboot on Pixel C?
Click to expand...
Click to collapse
Because it will wipe your device.

cam30era said:
Because it will wipe your device.
Click to expand...
Click to collapse
Ok. Do you recommend creating a flashable ZIP file instead?

jogrimst said:
Ok. Do you recommend creating a flashable ZIP file instead?
Click to expand...
Click to collapse
Yep, pls use flashable zip for your purpose.

TWRP Can't Mount /System - OP3 Bricks When Trying to Flash New ROM??

I'm currently running my OP3 on Resurrection Remix 5.7.4 (MM 6.01), the phone has the latest version of TWRP and I've flashed other ROMs in the past without any issue
RR has proven to be quite unstable for me, the system UI keeps crashing, which renders the phone unusable until after a reboot so I wanted to flash Freedom OS as that worked quite well for me previously. When I tried to flash the ROM in TWRP the ROM seemed to flash successfully but when I rebooted, only the boot logo showed and then the screen went black and the phone refused to respond at all for about two hours, after that exactly the same thing happened. I managed to boot into recovery and tried flashing a different ROM but the result was the same every time, the only way to get the phone to work normally was to flash RR again/restore from a Nandroid backup, either way, the result is the same, I'm stuck on an unstable ROM and I don't know why, the last time I flashed these ROMs they worked fine, I followed all of the instructions for flashing said ROMs to the letter, I have the latest version of TWRP, etc.
Also it seems TWRP can't mount /system, in TWRP under "mount", system is unchecked, I checked it and tried again but found that whenever I rebooted the phone or attempted to flash a new ROM, it would always uncheck itself and flashing the ROM would fail, however strangely flashing the same version of RR was always successful. Is this why the flashing keeps failing and does anyone know what the problem could be?
Thanks!

bronderb said:
Go into your bootloader and format system, cache, and userdata (This will wipe EVERYTHING so be sure to backup). Then latest official flash stock rom and reboot. This should fix it. If this helped hit thanks!
Click to expand...
Click to collapse
Bootloader? There is no option in the bootloader to delete everything as far as I can see, do you mean in recovery? And if I wipe everything off the phone, how do I get the stock ROM onto the phone to flash it? When you say wipe userdata do you mean wipe internal storage or am I misunderstanding and you're telling me to wipe the usual dalvik, cache and data as you would for a normal clean flash?

bronderb said:
No, go into fastboot and plug into pc. Open cmd and type "fastboot format userdata" "fastboot format cache" "fastboot format system"
Click to expand...
Click to collapse
Okay, thanks, and then what? Do I flash TWRP then push the stock ROM to the phone and flash that? And exactly how do I do that?

bronderb said:
You just go into twrp (no need to reflash) then adb sideload oxygen os
Click to expand...
Click to collapse
Thank you, I am now running OOS, hopefully everything else will go smoothly now, planning to go back to FOS again
I didn't even have to use the fastboot commands to format, just sideloading the OOS file wiped everything anyway

Turns out the issue isn't actually fixed, "system" still appears unchecked in TWRP. Could the fact that I didn't actually format everything first have made a difference, given everything was wiped in the process anyway?
Looks like I'm going to be living the sideloading life...
Edit: Sideloading freedom OS failed, ended up having to restore from my TWRP backup. I don't know what I'm going to do, my main reason for using custom ROMs is installing a black theme, and OOS doesn't support Layers or Substratum fully so I can't stay on that

Is the thing that says "mount system as read-only" enabled in the mounts menu? /System isn't ever mounted when TWRP starts, you have to manually mount it, but as long as the updater-script tells /system to mount, it will. However, if that option to make it read-only is enabled, then nothing will ever write to /system. Last resort is deleting the .twrps file from /sdcard/TWRP folder to remove current TWRP preferences and then when you boot into TWRP, that first screen has the swiper to allow modifications, and you'll swipe that

frickinjerms said:
Is the thing that says "mount system as read-only" enabled in the mounts menu? /System isn't ever mounted when TWRP starts, you have to manually mount it, but as long as the updater-script tells /system to mount, it will. However, if that option to make it read-only is enabled, then nothing will ever write to /system. Last resort is deleting the .twrps file from /sdcard/TWRP folder to remove current TWRP preferences and then when you boot into TWRP, that first screen has the swiper to allow modifications, and you'll swipe that
Click to expand...
Click to collapse
It isn't set to read only, I have tried updating and resetting TWRP but nothing has helped, in fact it's got worse, I managed to sideload OOS but my attempted install of FOS failed so I tried to restore from a backup and that also failed, something which didn't happen back when I was on RR.

evilkitty69 said:
It isn't set to read only, I have tried updating and resetting TWRP but nothing has helped, in fact it's got worse, I managed to sideload OOS but my attempted install of FOS failed so I tried to restore from a backup and that also failed, something which didn't happen back when I was on RR.
Click to expand...
Click to collapse
Which version of TWRP are you running? The official 3.1.0-0 is good for OOS and OOS based ROMs only. Try eng.stk's version.

Use the latest eng-stk twrp.
https://forum.xda-developers.com/devdb/project/?id=15934#downloads
In recovery go to Wipe > Format data (this will wipe ALL in youre phone).
It's youre Data in F2FS? If yes you must change it to ext4, in recovery Wipe > Change file system. Reboot recovery. Check if it's changed.
Reboot to Bootloader (Fastboot). From PC reflash recovery.
Now copy Oos rom to the phone memory. Flash rom. Reflash recovery (the rom will overwrite the recovery). Reboot to recovery to see if it works. If not do it again.
I sugest to NOT restore old data. This way you'll have all new in youre phone.

tnsmani said:
Which version of TWRP are you running? The official 3.1.0-0 is good for OOS and OOS based ROMs only. Try eng.stk's version.
Click to expand...
Click to collapse
I was running 3.0.2-2 when the issue started, I updated to 3.0.3 and 3.1.0-0 but nothing changed. Thanks for the suggestion, I will give it a try

null0seven said:
Use the latest eng-stk twrp.
https://forum.xda-developers.com/devdb/project/?id=15934#downloads
In recovery go to Wipe > Format data (this will wipe ALL in youre phone).
It's youre Data in F2FS? If yes you must change it to ext4, in recovery Wipe > Change file system. Reboot recovery. Check if it's changed.
Reboot to Bootloader (Fastboot). From PC reflash recovery.
Now copy Oos rom to the phone memory. Flash rom. Reflash recovery (the rom will overwrite the recovery). Reboot to recovery to see if it works. If not do it again.
I sugest to NOT restore old data. This way you'll have all new in youre phone.
Click to expand...
Click to collapse
Help! I attempted to do this and the process failed, I didn't manage to wipe data because "/data could not be mounted as device or resource is busy"
Edit: Device now has nothing but TWRP and fastboot, no OS, no ADB, no data, no internal storage

If you can reflash twrp from PC. If not reflash recovery from twrp recovery : Instal > Image > Select Recovery (you must have TWRP.img in youre phone memory )

I managed to get hold of a few factory images and attempted to flash system.img through fastboot, resulting error was
"target reported max download size of 536,870,912 bytes" (system image is 3080 MB)
"invalid sparse file format at header magi"

null0seven said:
If you can reflash twrp from PC. If not reflash recovery from twrp recovery : Instal > Image > Select Recovery (you must have TWRP.img in youre phone memory )
Click to expand...
Click to collapse
I reflashed the modified TWRP successfully. The second suggestion wouldn't have worked, given my phone has no internal storage

You can't see internal storage because of the recovery does not work write.
If you have a working recovery you can do all you want.

null0seven said:
You can't see internal storage because of the recovery does not work write.
If you have a working recovery you can do all you want.
Click to expand...
Click to collapse
The recovery was working at this point, the issue turned out to be the data partition, system was formatted to ext4 but I didn't realise data was in f2fs which caused the problem in the first place. I formatted to ext4 and everything worked, I was able to install paranoid android and everything worked...
...until I flashed the wrong firmware and now I've hard bricked it! I'm going to try the hard brick toolkit
Edit: After 12 hours of being an unresponsive brick, the phone suddenly booted, system doesn't work but it has recovery. Unfortunately I seem to be back to square one, if I try to flash something it appears to succeed but then the phone doesn't boot up into system

So I am now back to the original issue of system not mounting, in terminal if I type "mount /system" I get following error:
"mounting /dev/block/sde20 on /system failed: Device or resource is busy"
Anyone know how to solve this?

I formatted everything, reflashed recovery and tried to flash OOS, got following error:
Code:
[FONT="Courier New"]"This package is for OnePlus3 devices, this is a oneplus3t" (it isn't)
"Updater process ended with ERROR: 7
Error installing zip file /sdcard/OOS3.2.6.zip" [/FONT]

Issue was "solved" by flashing PA, GApps and the correct firmware
Hopefully that's the end of all this drama, thanks to everyone for their suggestions

Hi, I'm pretty new at rooting and I seem to have a similar error. I'm getting the "can't mount /system" error in TWRP. I am able to boot into OOS 4.1.3 and run the nandroid backup but, I keep getting the same error and am unable to select "system" when choosing the mount option in TWRP.
Do I need to have all partitions in f2fs? My system partition is in ext4. Is this what solved your issue?
It seems my phone is functional even with the error active but, I've been trying to get systemless SuperSU and MagiskHide working and I think that's what is preventing me from doing so.

TPRW Backups does not boot with Nougat 32.3.A.2.33 ROOTED

I have a problem with my TWRP backups created for 32.3.A.2.33 version. When I restore them, the phone restarts once and in the next boot int goes into recovery directly.
I have flashed:
E6853_32.3.A.2.33_1299-1511_R3D.ftf
Boot and recovery from:
https://forum.xda-developers.com/z5...ting-xperia-z5-z5p-variants-drm-keys-t3437902
And works perfectly, backups creates correctly from recovery, but when I try to recover them doesn't work.
The problem seems to be in system partition , data partition recover with no problems.
Any suggestions?

I have the same problem.

srg84 said:
The problem seems to be in system partition , data partition recover with no problems.
Any suggestions?
Click to expand...
Click to collapse
Is dm-verity turned off in kernel? If not this is the reason why. You need to backup system image instead of backup files from system partition like usually in TWRP.
Can you try to make backup and restore with this recovery?
flash with "fastboot flash recovery lineage_twrp_recovery.img"
To boot into recovery power off phone, wait 5-7 sec then press power+vol_down until it vibrate.
Select "System Image" to system partition backup.

Need help creating nandroid backup without screen.

Hi there,
Last friday I broke my lcd screen, it simply does not turn on anymore. Tomorrow an UPS employe will pick up my device and ship it to OP for repair.
My device: Oneplus 3 running TWRP recovery and rooted LineageOS. Adb is enabled in dev settings.
So now I am trying to create a nandroid backup to easily restore my system and data when I get my phone back since OP will reset it ...
I had this genious idea to boot into recovery and create a backup using adb but I ran into a problem.
I have to unlock my device in order to create the backup :/
adb console:
> D:\Desktop\Fastboot+ADB\Win32>adb backup -f backupbeforerepair --twrp system data boot
> Now unlock your device and confirm the backup operation.
So now I am hoping any of you will come up with another genious idea or a workaround so I can create this damn backup
Thanks alot in advance!

If anyone else would ever find himself in the same situation as me I was able to create the backup by doing the following:
You can use open recovery script in TWRP. See http://wiki.rootzwiki.com/OpenRecoveryScript
If you have the file
/cache/recovery/openrecoveryscript
then TWRP will process it when it starts.
If the file contains
backup BSDO myBackup
then the boot (kernel), system, and data partitions will be backed up in compressed format and named myBackup.
So, on your PC, create the openrecoveryscript file, boot into recovery, use ADB to push the file to /cache/recovery/openrecoveryscript then reboot back into recovery. That should trigger the backup process.
Note, the backup will be written to the default location TWRP for backups.
Click to expand...
Click to collapse

Pie, TWRP and vendor partition help

So I've been running Oreo Omnirom and I'm looking at moving to Android Pie. I have never installed any of the OnePlus Beta's or any other Pie based ROM.
I am currently running TWRP version twrp-3.2.3-0-20180822-codeworkx-dumpling and from the date code, it looks like to could be a little old. When I jump into TWRP, I get errors that no vendor partition is found.. I have always ignored this as I was on Oreo and didn't need/care about a vendor partition. This also led me to believe that the version of TWRP I am using support the vendor partition.
Is there an easy way to create and format the vendor partition? Is my TWRP version too old? I tried running twrp-3.1.1-x_blu_spark_v8.55-op5_op5t as recommended in another thread (even tho it's older) and I had to re-flash TWRP via fastboot because that IMG file no worky...
I am trying to avoid formatting my data since my phone has a larger drive than my computer. If someone could point me in the right direction, it would be very much appreciated.

Flashing twrp-3.1.1-x_blu_spark_v8.55-op5_op5t or twrp-3.2.3-0-20190311-codeworkx-dumpling.img (both TWRPs for PIE) wont boot to recovery. Both boot to a blank screen with the notification LED turned blue. I assume that this is because I am still on Oreo firmwares... Seems like I have a chicken and egg sort of situation going on here....

Flashed OxygenOS_5.1.4 full OTA. /vendor partition was not created, but I only booted until I reached the lockscreen at which point told me that my password was incorrect. Do I need to remove my lockscreen password in Omni and try again?

crakerjac said:
Flashed OxygenOS_5.1.4 full OTA. /vendor partition was not created, but I only booted until I reached the lockscreen at which point told me that my password was incorrect. Do I need to remove my lockscreen password in Omni and try again?
Click to expand...
Click to collapse
Remove your security before flashing anything and then try 5.1.4 full OOS zip. If that works, great. OTA up to latest, or as far as it will allow and then use the full zip files available in the forum to get onto latest Pie. You'll need to fastboot flash latest Pie TWRP (codeworkx) when you get up to date.
If you run into security problems you might have to try a clean flash, or even format. Make sure you are in front of your PC before beginning and backup everything.

Still having no luck here.
Here is what I have done to try and get TWRP to recognize a /vendor partition:
Formatted Data
Side loaded OnePlus5TOxygen_43_OTA_037 in TWRP
Flashed OTA_037
Booted OxygenOS and completed setup
Rebooted to bootloader
Flashed twrp-3.2.3-0-20180822-codeworkx-dumpling
Rebooted to TWRP recovery
At this point I am STILL getting 'Failed to mount '/vendor' (invalid argument)'.
WTF am I doing wrong? How to I create a /vendor partition?

crakerjac said:
Still having no luck here.
Here is what I have done to try and get TWRP to recognize a /vendor partition:
Formatted Data
Side loaded OnePlus5TOxygen_43_OTA_037 in TWRP
Flashed OTA_037
Booted OxygenOS and completed setup
Rebooted to bootloader
Flashed twrp-3.2.3-0-20180822-codeworkx-dumpling
Rebooted to TWRP recovery
At this point I am STILL getting 'Failed to mount '/vendor' (invalid argument)'.
WTF am I doing wrong? How to I create a /vendor partition?
Click to expand...
Click to collapse
hi, may be this will help you, see post #2 from this thread:
"..... (because 5.1.4 creates the vendor partition and 5.1.5 fills ot)"
https://forum.xda-developers.com/oneplus-5t/help/solved-twrp-error-7-vendor-partition-t3846181

My thought process was that if 5.1.4 creates the /vendor partition, then TWRP should still be able to recognize it and mount that partition... I'm not sure if I care if it's full or empty at this point... I fear that I will download another 1.6GB of data just to find that the update wont happen because /vendor cannot be found... Unless I am completely wrong...

Well dang... I was wrong. Flashing 5.1.5 patched the /vendor partition and removed the error in TWRP.
Thanks for the help guys.
Mods please close (solved).