Develop Bugs

Internal storage: Errors when bulk-writing small files

Hello fellow N5X owners,
I've just registered at XDA Developers (also I did read here a long time ago) because I experience some strange problems with internal storage of my device.
I've first noticed when I wanted to use an app which on first startup unpacks a large OBB file: Initial setup fails all the time stateing internal storage may be low. I checked storage, there is enough space left. The curious thing is that it always failed at different percentages and after countless attempts it even finished successfully once.
Further investigating it, I tried to install the app on other Android devices (including another Nexus 5X, both phones are on build OPM2.171019.029), which worked flawlessly. Also I did a a "factory reset" by unrooting and re-flashing the latest stock ROM on my device, which did not help.
I then took a closer look at the OBB (the app is open source, so this was easily possible) and tried to manually extract the OBB ZIP file on my device. This did not work either: Same behavior, Total Commander randomly fails unpacking it with message "Error unpacking: Error writing target file" (perhaps not in this wording, my Android is on German).
Since I suspected the ZIP file (although it's working fine on other devices), I created some ZIP files myself.
Unpacking them on the device, I was able to observe the following:
ZIPs containing large files (for example MP3 files, even if this is pretty pointless) work OK
ZIPs containg a lot of small files (like the original OBB, which contains several hundred files between 100 KiB and 1 MiB) will show the same error
Apart from that, I don't experience any issues with the device (no reboots, force closed apps etc).
Does anybody have an idea on what's going on here? I would RMA it (one month of guarantee left), but I'm afraid that this error will not be reproduced at the service center.
EDIT: I just noticed, that it does not seem to be related only to ZIP files. I connect the device to my PC and copied a large amount of MP3s to it, which finished successfully, everything OK. Then I copied a large amount of small files (the ones from the ZIP archive) and after some time Windows Explorer did not show progress anymore (even though the files are only a few 100 KiB large, after some time name of the file currently copied did not change anymore).
EDIT2: It did the experiment from last edit again, this time at the exact moment Windows Explorer "freezed" a background app on the phone force closed.

Some new observations:
I pushed some TAR files to the device (as before two types: one containg large files, one containg small files) and connected to the device by using ADB shell.
Then, with the following command I extracted the archives:
Code:
tar -xf archive.tar /sdcard/Download/
'Large files' archive: no problems observed.
'Small files' archive: sporadic errors.
If I put some load on the device (starting apps, incoming phone call, ...) suddenly extracting the 'small files' archive throws a lot of errors:
Code:
tar: dummy0015.txt: can't open: No such file or directory
tar: chown 0:0 'dummy0015': No such file or directoy
tar: dummy0063.txt: can't open: No such file or directory
tar: chown 0:0 'dummy0063': No such file or directoy
tar: dummy0065.txt: can't open: No such file or directory
tar: chown 0:0 'dummy0065': No such file or directoy
[...]
Also after rebooting, for a long time extracting the 'small files' archive will throw errors. At this point certainly a lot of background processes are executed, so there is also a lot of CPU load.
So somehow all seems to be traceable back to CPU load. But why?
Heat? But it won't get any worse if I let the device heat up under a pillow and also occurs right after boot.
Electromagnetic interference? If this is the case, one would expect it to be designed badly and therefore other devices affected by it, too.
Instability related to CPU clock? Probably the device clocks up, when there is load. On the other hand: The device runs with standard clock speed and was never overclocked.

Hello,
I did some flashing and was able to narrow the problem down to Android Oreo:
First, I flashed the latest Oreo factory image (8.1.0, OPM4.171019.016.A1, May 2018). Problem persists.
Then, I went back to last Marshmallow build (6.0.1, MTC20K). No problems extracting the files.
Next, I upgraded to last Nougat build (7.1.2, N2G48C, Aug 2017). Again, it works flawlessly.
Finally, I flashed first Oreo build (8.0.0, OPR6.170623.013, Aug 2017). Problem appears again.
All flashing was done using fastboot and the flash-all batch script supplied with the factory images, I did not install or configure anything after flashing, nor did I restore a backup.
So, clearly something in Oreo is 'killing' my device. And nobody else seems to undergo this issues.
Do any of you have any idea what might be causing this or what else I could do to further narrow down the problem?

[GUIDE][MAGISK]Create manual backup of stock boot.img

As it seems impossible to hotboot TWRP recovery for Magisk installation and installing TWRP only for this sole purpose is a little overkill, many people are using rooting procedure with flashing patched boot.img directly. However this does not automatically create a backup of stock boot image, which is used later for OTA updates.
It is fairly easy to create the backup manually (and hopefully Magisk developers will add this feature into Magisk Manager..).
Option 1 (ADB shell on the PC or terminal emulator on the phone):
Code:
copy boot.img into the root of internal sdcard
adb shell
su
cd /data/adb/magisk
./magiskboot sha1 /mnt/sdcard/boot.img
(copy generated SHA1 checksum)
./magiskboot compress /mnt/sdcard/boot.img /mnt/sdcard/stock_boot_[I]putSHA1here[/I].img.gz
cp /mnt/sdcard/stock_boot_[I]putSHA1here[/I].img.gz /data/stock_boot_[I]putSHA1here[/I].img.gz
Example:
Code:
tissot_sprout:/data/adb/magisk # ./magiskboot sha1 /mnt/sdcard/boot.img
cb925c4fe36ace17b2ff94b34ddcde1e564acaaf
tissot_sprout:/data/adb/magisk # ./magiskboot compress /mnt/sdcard/boot.img /mnt/sdcard/stock_boot_cb925c4fe36ace17b2ff94b34ddcde1e564acaaf.img.gz
tissot_sprout:/data/adb/magisk # cp /mnt/sdcard/stock_boot_cb925c4fe36ace17b2ff94b34ddcde1e564acaaf.img.gz /data/stock_boot_cb925c4fe36ace17b2ff94b34ddcde1e564acaaf.img.gz
Option 2 (Windows PC with Total Commander):
Code:
1. get stock boot.img
2. calculate SHA1 of it (file, create CRC, SHA1)
3. copy calculated SHA1 to clipboard
4. rename boot.img to [B]stock_boot_[I]putSHA1here[/I].img[/B]
5. zip to file, GZ
6. copy resulting file [B]stock_boot_[I]putSHA1here[/I].img.gz [/B]to /data on the phone
Option 2 will generate file with slightly different size than option 1, but it works just as fine for Magisk restore function.
Option 3 (rooted phone):
Code:
1. boot phone with Magisk patched boot.img
2. get stock boot.img
3. flash stock boot.img from Franco Kernel manager app, do NOT reboot
4. Magisk Manager - install, direct install
I tested this on Mi A1, but there is probably no reason why it shouldn't work on other phones too.
Option 4 (any phone)
After patching a stock image you can find a backup image in (assuming non-hidden Manager) /data/user_de/0/com.topjohnwu.magisk/install
Source

Important note - it seems that Magisk 20.2 changed the backup structure. Backups of stock boot.img are located in /data/magisk_backup_SHA1/boot.img.gz now. Each backup has its own folder.
v20.1 and below -> /data/stock_boot_SHA1.img.gz
v20.2 -> /data/magisk_backup_SHA1/boot.img.gz

Just a heads up that if you want to change the backup image to a different one you have to run magisk --path to get the path, edit the magiskpath/.magisk/config file to the new SHA1, force stop Magisk, and then restart Magisk

Note: the sbin folder does not always exist on Android 11 and up (see here).
Instead, look a folder under /dev with a random short name. In my case it was /dev/XFmlBk/.magisk

Armand Bernard said:
Note: the sbin folder does not always exist on Android 11 and up (see here).
Instead, look a folder under /dev with a random short name. In my case it was /dev/XFmlBk/.magisk
Click to expand...
Click to collapse
Holy hell, do you have any idea how long I've been searching for this very specific explanation on why I can't locate my sbin folder? Thank you!

LG G7 G710EM sof. ver V30f-EUR-XX root

This morning I deleted my system camera app and that made my phone bootloop. After some trickery I managed to install a new version but lost all my photos and files along with root. Can I get a tutorial how to create my own boot image with magisk so I can hopefully recover some of the photos. Thank you

Hey, there might be more ways, but this is the one I follow everytime I root my g7.
Download the firmware (.kdz file). You can get that here.
In the meantime (the download will take some time) you can search and download a tool called "LG Firmware Extract", mine is v1.2.6.1. If you struggle finding it, you can download the one I attached (for Windows).
Once both are downloaded, open the firmware extractor and there click "Open" next to "KDZ File" label and locate the .kdz file you just downloaded. You can also type in the file path instead of locating.
In the table of files below, check the largest file starting with G71030f and click "Extract KDZ"
In the working directory (marked as "Working folder ..." above the table) you should now find a file named after the file you checked. Locate that one on "DZ File" row by clicking "Open" next to it.
Now in the second table, find a file called boot.img_65286, check, Extract DZ.
The file should be in the same directory previous one (G71030f...) was. Rename it to boot.img.
Send the boot.img to your phone
Install Magisk Manager, open, click "Install magisk"
On method selection choose "select and patch a file" and pick the boot.img file and proceed.
That's it, you should be told where your patched image sits.
In case you want to skip steps 1 to 7, I attached a boot.img file. I also attached a patched boot.img, since I did it for mine, too.
BTW if you choose to download the firmware extractor from the attached files, you can find a short tutorial on how to extract boot image in a text file inside.
Hope it still helps you.

Repackage old Stock Rom? Remove Stock Rom Sig?

Hi all
I'm trying to flash and old kernel (boot.img) and system (super.img) to a S20 z1s exynos - SM-G980F/DS with a SW REV. higher than I would like...
I flashed TWRP sucessfully, this tripped knox, so I don't care at all about unsigned code running.
So I tried to:
1. disable AVB using a prebuilt vbmeta.img from here: https://forum.xda-developers.com/t/...root-s20-series-and-upgrade-firmware.4079353/
2. unpacking and then re-packing the stock firmware using "superr"s kitchen, but this produced a zip with which twrp was not happy with, even fixing a lot of updater-script errors... then again I think it does not help that my TWRP thinks the device is a z3s (no other twrp build available)...
3. flashing via ODIN obviously failed due to the device vs binary SW REV. difference.
4. flashing boot and super "by hand" in twrp -> error about SW REV. mismatch (DEVICE: X BINARY: X-1)
I have so many Questions to which I am unable to find answers, just suspicions/opinions....
Qs:
1. Can I simply disable all boot verification somehow?
2. how are vbmeta images created? do I need to fakesign my images? (vbmeta.img and vbmeta_samsung.img)
Thanks a lot for some clarifications

Defekkt said:
Hi all
I'm trying to flash and old kernel (boot.img) and system (super.img) to a S20 z1s exynos - SM-G980F/DS with a SW REV. higher than I would like...
I flashed TWRP sucessfully, this tripped knox, so I don't care at all about unsigned code running.
So I tried to:
1. disable AVB using a prebuilt vbmeta.img from here: https://forum.xda-developers.com/t/...root-s20-series-and-upgrade-firmware.4079353/
2. unpacking and then re-packing the stock firmware using "superr"s kitchen, but this produced a zip with which twrp was not happy with, even fixing a lot of updater-script errors... then again I think it does not help that my TWRP thinks the device is a z3s (no other twrp build available)...
3. flashing via ODIN obviously failed due to the device vs binary SW REV. difference.
4. flashing boot and super "by hand" in twrp -> error about SW REV. mismatch (DEVICE: X BINARY: X-1)
I have so many Questions to which I am unable to find answers, just suspicions/opinions....
Qs:
1. Can I simply disable all boot verification somehow?
2. how are vbmeta images created? do I need to fakesign my images? (vbmeta.img and vbmeta_samsung.img)
Thanks a lot for some clarifications
Click to expand...
Click to collapse
For these and other reasons, I quickly grew frustrated with SuperR's Kitchen and ultimately ended up rolling my own purpose-built solution that peels the layers of the onion of the Samsung package, gets to the point of mountable filesystem images, performs the desired transforms, and then packs everything back up into Odin-flashable .tar.md5 files. It's rough and minimalistic, but it works.
I'm not familiar with the error you mention, but usually those sorts of errors come from flashing a bootloader in Odin that is below the level of the one residing on the device. I haven't done much flashing via TWRP since I'm producing Odin-flashable output and I don't know why you'd have issues on boot.img or super.img. For my own sanity I don't regularly flash bootloader images. boot.img doesn't contain the booatloader though: it's the kernel and initramfs.
vbmeta.img files are generated with avbtool. I created one with a null signature which is identical to the ones floating around on the forum. I also cleaned up my fstab and manifests using the techniques of ianmacd's multidisabler.

sjevtic said:
For these and other reasons, I quickly grew frustrated with SuperR's Kitchen and ultimately ended up rolling my own purpose-built solution that peels the layers of the onion of the Samsung package, gets to the point of mountable filesystem images, performs the desired transforms, and then packs everything back up into Odin-flashable .tar.md5 files. It's rough and minimalistic, but it works.
I'm not familiar with the error you mention, but usually those sorts of errors come from flashing a bootloader in Odin that is below the level of the one residing on the device. I haven't done much flashing via TWRP since I'm producing Odin-flashable output and I don't know why you'd have issues on boot.img or super.img. For my own sanity I don't regularly flash bootloader images. boot.img doesn't contain the booatloader though: it's the kernel and initramfs.
vbmeta.img files are generated with avbtool. I created one with a null signature which is identical to the ones floating around on the forum. I also cleaned up my fstab and manifests using the techniques of ianmacd's multidisabler.
Click to expand...
Click to collapse
Thank you for your reply!
1. any chance you could list the steps you used to re-package a samsung rom?
2. is there a guide on how to create null-sigged vbmetas? can you use avbtool or do I need to manually edit the vbmeta file(s) ?
I did not try to downgrade the bootloader, just boot.img (kernel) and super (system, etc.).
Thanks in advance and best regards!

Sorry for the delay. I am on vacation and have not been regularly checking the forum.
To give you a conceptual overview of the workflow, here is the debug output my script generates for each step when trivially unpacking/repacking a Samsung firmware archive:
Extracting source zip.
Extracting .tar.md5 files.
Unlz4ing, unsparsing, and copying images.
Unpacking super.img.
Unpacking boot.img and up_param.bin.
Mounting filesystem images.
Applying main transforms.
Unmounting and checking filesystem images.
Repacking up_param.bin and boot.img.
Repacking super.img.
Sparsing, lz4ing, and copying images.
Archiving .tar.md5 files.
The real magic, of course, is in the transforms that are performed along the way, of which there are many. Most (notably those that are operations against the filesystem images) are performed during step 7, though some are performed before or after other steps by necessity. At some point I'll probably document this process a bit more and release the scripts since it is nontrivial.
Here is how I created my null vbmeta image:
Code:
avbtool make_vbmeta_image --flags 2 --padding_size 256 --output "${STEP_03_DIR}/BL/vbmeta.img"

Question Magisk OverlayFs /Product for replace carriersettings files ?

Hello dears.
I have been working for 5g volte vowifi since i bought the phone.
Also You know we have carrier settings files in /product/etc/carriersettings I edited one of them and now I need to replace with original one to try.
but this /product fs cannot be touched cannot be mounted R/W. But We have a option from Magisk. it is overlayfs for all R/O filesystem acting like real one.
Question is How we do this /product fs as overlayfs?
Here one guide we have but I did not excaly get it because I didnot work about building magisk models before.
https://topjohnwu.github.io/Magisk/guides.html
Also I find a module but When I try this I am getting Unzip error. (I already tried to zip again without upper folder )
GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
Emulate read-write partition for read-only system partitions - GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
github.com
I worked with for ex vodafone_tr.pb file I get parameters from att5g_us.pb( It has lots of parameters for 5g volte vowifi) And I add these information to my carrier file. But I have to push this file to product.
Thank you

You need to create a Magisk module. Best is to download an existing module, unzip it, then modify it, the re-zip it.
Under the directory of the Magisk module, create a subdir called 'system', create a subdir 'product' underneath and 'etc/carriersettings´ subdirs as well. Put your file in there. Re-zip the module. Use adb to push to phone and install the module with the Magisk app. Reboot phone.
Magisk will mount your (modified) file in /product/etc/carriersettings/...
Check if the file is there:
Code:
> adb shell
$ su
# cd /product/etc/carriersettings/...
# ls

Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post

tauio111 said:
Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post
Click to expand...
Click to collapse
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.

furkanosman said:
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Click to expand...
Click to collapse
I just copy and paste the xml values every time, its persistent until you enter a different sim or update android.
I think it would be more viable to make a magisk script that adds the part you want to the front of the xml instead of replacing the bp files.

furkanosman said:
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.
Click to expand...
Click to collapse
Hi
Did u manage to work on the pb files in /product
I tried the cfg method but my ims registration isn't changing