Develop Bugs

[Q] Cant build kernel modules for Android Desire CM6

Hi All
While I am a relative noob when it comes to Android development (I have been an embedded developer for over 25 years), I am pretty good at creating custom embedded kernels for MIPs targets. However, this issue seems to be getting the best of me.
Environment:
Ubuntu 10.04 + any automatic upgrades
HTC desire phone, rooted, running cyanogenmod 6.0.1 (and 6.1.0 from the daily builds)
Issue: I am trying to add batman-adv to the phone, but I cannot create a batman-adv.ko that insmod will load. When I put my ko file on the phone and try to load it I get "insmod: init_module 'batman-adv.ko' failed (Exec format error)" (Yes I am in su mode). I know that this generally indicates that I have built for the incorrect architecture and I suppose that is the real question. How do I GET the correct architecture.
I extracted a ko from the phone that works on the phone and ran 'file' on it:
ah6.ko: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
and this is what I get on my ko
batman-adv.ko: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
Based on this, the architecture is correct (or appears to be correct).
EDIT1:
I am getting a bit closer. I get this in dmesg
batman_adv: version magic '2.6.35.4-cyanogenmod-gb7da4b8-dirty preempt mod_unload ARMv7 ' should be '2.6.35.4-cyanogenmod preempt mod_unload ARMv7
Looks like I need a way to remove the "-gb7da4b8-dirty" portion of the string (actually avoid having it there in the first place).
End EDIT1
Build environment:
I have downloaded (via repo / git) and built the cyanogenmod SDK (daily build from 9/11/2010 kern: 2.6.35.4). The output of a build of this repo is an update-foo.zip file which I loaded on the phone and verified that it works properly. The repo includes a kernel-msm directory with (an apparently) fully functional kernel build tree (as well as the sdk and ndk trees).
I have pulled the config.gz file from the (working nicely) phone (update-foo.zip version), unzipped it and fed it into the build system. Answered the annoying questions you get when you merge in a new config file and added batman-adv to the selected loadable modules list to be built (also had to play with the file to fix a few other issues).
Build script looks like this (CCOMPILER points to the pre-built compiler in the sdk):
cd $SYSTEM_PATH || exit
. build/envsetup.sh
lunch cyanogen_bravo-eng
cd $SYSTEM_PATH/kernel-msm || exit
make ARCH=arm CROSS_COMPILE=$CCOMPILER -j$CPU_COUNT clean || exit
make ARCH=arm CROSS_COMPILE=$CCOMPILER -j$CPU_COUNT | tee build.log ||
The build completes just fine and I get all of the expected ko's. I have not tested the kernel itself since I am not really interested in it. It should be the same as the kernel built by the SDK and included in the update-foo.zip file. Worst case the KO would load and crash if the ko and the kernel on the phone were incompatible (I would be happy to get that far).

Issue is resolved.
While the insmod function returned a bad format error, it was really complaining about the magic number mismatch. Once I took care of matching the ARMv5 vs ARMv7 issue and then did an over ride on the KERNELVERSION string, I was able to load and execute the ko without any additional issues.

Wow! You do like to write don't you mate?
I'm glad it got resolved....should I close this thread or you want to get some more input?
Cheers,

We can close the thread. I write so much because I see so many post with insufficiant information to make a good diagnosis. In addition, those threads also tend to wander off the subject, making them a bit less than usefull.
Martin

I believe I am trying to do the same thing here, but for a different phone.
You say you're adding your module to the selected loadable modules to be built. Isn't there a way to build a loadable module by itself without rebuilding or changing kernel settings?
I have been able to compile/install modules for ubuntu linux but I haven't been able to figure out how to compile them for my android device. I have the source tree for the kernel build I'm using but I'm not quite sure how to go about cross compiling my module for it...

Linux cross-compile or ready binaries

Hi, I would put some linux binaries (arm architecture ?) on my phone to support the ADB SHELL activity, to enlarge the busybox capabilities.
For instance, I mean stuff like a bash shell, a complete gtar for backups (the busybox one isn't adeguate), md5deep, and so on.
Could you please give any hint to do that ?
Thanks

Val3r10 said:
Hi, I would put some linux binaries (arm architecture ?) on my phone to support the ADB SHELL activity, to enlarge the busybox capabilities.
For instance, I mean stuff like a bash shell, a complete gtar for backups (the busybox one isn't adeguate), md5deep, and so on.
Could you please give any hint to do that ?
Thanks
Click to expand...
Click to collapse
The first post here: http://forum.xda-developers.com/showthread.php?t=777380 contains a link that actually contains a complete binary arm cross compiler. If you use linux (works with most 32 bit distros, needs multilib on 64 bit distros) you can use that to cross compile your code to arm.

Many thanks for your answer.
I'd static link libraries, I guess...
Btw could I also use normal binaries for 32bit arm architecture ? (i.e. debian distro...)

You can also use some of the binaries of the CyanogenMod for Desire (or any other phone) and use these (bash, nano) http://androidforums.com/evo-4g-all...shell-nano-tab-completion-color-over-ssh.html
Bash 4.1 : http://android.modaco.com/content/software/301932/bash-4-1-native-app-for-android/ (not sure its the one i got but you can give it a try)

Jb boin said:
Bash 4.1 : http://android.modaco.com/content/software/301932/bash-4-1-native-app-for-android/ (not sure its the one i got but you can give it a try)
Click to expand...
Click to collapse
Many thanks.
The bash 4.1 binary could also be found in the Better Terminal Emulator package from market...
Actually I'm looking for a way to get other working stuff for low level backups, as said gtar, zip, ...

[KITCHEN] Minimal kernel building VM with scripts

I added some convenience scripts to the config. Read the new tutorial here
The old one still works:
I put together a small virtualbox vm with a very minimal debian config with which I was able to compile a working kernel. This is for all the people who are running Windows, and want to try modifying kernels/initramfs's. As this is a minimal config there are some drawbacks (like it doesn't have a graphical interface), but you can always install additional packages, if you want to.
The VM includes: debian, build essentials, git, vb guest additions, mc, vim and codesourcery 2010q1 gnu-eabi. (it's larger than the simple eabi version, but can be used to compile non-kernel applications too)
First of all I hate both SunOracle VirtualBox and debian, but VB is free, and debian is lean, so they'll do the job.
To get the image running do the following:
First download and install VirtualBox
Next download the VM image: http://android.sztupy.hu/dl/KernelCompilerVM-1.1.7z and extract it. (it's a large download. If you can please put it up a mirror)
Mirrors: (thanks to the people mirroring it)
- http://www.multiupload.com/THJV19BJ9X
- http://bote.ro/sztupy/KernelCompilerVM-1.1.7z
After this run virtualbox, and import this VM.
Run the VM. The username/passwords are: root/root and kernel/kernel. Login with kernel. (you can always switch to root using sudo)
Next choose what kernel/initramfs you want to compile.
Here are some links to kernels:
The original sources can be found at supercurio's git:
Code:
git://github.com/project-voodoo/linux_gt-i9000.git
froyo-samsung branch
The kernel of the voodoo project can be found at the same place:
Code:
git://github.com/project-voodoo/linux_gt-i9000.git
froyo-voodoo branch
The ULTK kernel can be found at my account:
Code:
git://github.com/sztupy/universal_lagfix_kernel.git
Here are some links to initramfs files:
The original froyo initramfs can be found at supercurio's git page:
Code:
http://github.com/project-voodoo/samsung_ramdisks.git
afaik the voodoo initramfs is build using scripts from these images, so you have to apply them.
The ULTK initramfs can be found here:
Code:
git://github.com/sztupy/universal_lagfix_kernel_initramfs.git
For the rest of the tutorial I'll be showing how to compile ULTK:
First get the kernel sources using git:
Code:
git clone git://github.com/sztupy/universal_lagfix_kernel.git kernel
Unfortunately no copy-paste function is available, so you have to write this manually...
Next get the initramfs using git:
Code:
git clone git://github.com/sztupy/universal_lagfix_kernel_initramfs.git initramfs
If everything goes well you'll have two directoryes, called kernel and initramfs. Next we have to modify some values in the kernel configs.
Nano, mcedit and vim are installed, use the one that suits you best. (mcedit is the most user friendly)
Switch to the kernel directory and edit the Makefile there:
Code:
cd kernel
nano Makefile
Find the row that says
Code:
CROSS_COMPILE ?= some value
Replace it to
Code:
CROSS_COMPILE ?= /home/kernel/arm-2010q1/bin/arm-none-linux-gnueabi-
(if using nano make sure it won't add a line break into the row)
Next load the default config:
Code:
make aries_eur_defconfig
And edit it:
Code:
nano .config
We have to supply the directory of the initramfs. Search for the line:
Code:
CONFIG_INITRAMFS_SOURCE=some value
and replace it to
Code:
CONFIG_INITRAMFS_SOURCE=/home/kernel/initramfs/out
(this is for the ULTK. For voodoo the initramfs locations are "froyo-xxjp6" and "froyo-xxjpm" instead of "out")
if you've managed to do that too, let's compile the kernel:
Code:
make
If everything goes fine after a while you will have your shiny new kernel. Now let's get it to your computer, so it can be flashed:
First, you have to create a shared folder in VirtualBox. Simply create a directory somewhere on your computer, and add it as a shared folder in virtualbox. The name of the shared folder should be simple, for example "shr".
Next, you have to mount that directory inside the VM. To do this enter:
Code:
sudo mount -t vboxsf shr ~/share
After it has been mounted you can copy the fresh kernel to your host OS:
Code:
cp arch/arm/boot/zImage ~/share
If everything goes well you have a zImage ready at the folder you've just shared. TAR it, fire up odin, and flash.
---------------------------------
Now that you've succesfully compiled a working kernel try to modify it. You can use "make menuconfig" in the kernel directory to switch some kernel functions (like filesystem supports) on and off. You can edit .config by hand to add or remove some configuration values. And you can edit the files in the initramfs directory. After modifications you only have to enter "make" in the kernel directory to get your kernel inside the "arch/arm/boot/zImage" dir ready.
You can also copy files from your host OS, to the guest OS, by putting the file inside the shared folder and copying it:
Code:
cp ~/share/thefile ~/initramfs/copyithere
Hope this guide was useful.

Impressive, thanks. Even a noob like me can try this.

Oh god, not only a talented developer, but a community helper!
STICKY!!
Gonna try it laters <3

oh yea, nice!

DocRambone said:
Impressive, thanks. Even a noob like me can try this.
Click to expand...
Click to collapse
If you're a noob, then I'm a baby

Darkyy said:
If you're a noob, then I'm a baby
Click to expand...
Click to collapse
high time you start compiling kernels

tnx
its very usefull

Magnificent!
I just came to take a quick look, but the thread deserves a closer one.
Thank you for sharing knowledge.

Worthy of a sticky.
Hopefully this will solve all those "I want this in a kernel but not that" scenarios.

This is VERY nice for the devs.. but.. i think this will make this forum spammed with 69 diff kernels with just minor changes.. as with the roms

Its better to let other people brick their phone than you brick your own
Just kidding! I for one will try and compile different versions with different lag-schemes to find the fastest combination. Including /system with fastest reading. I think I will add a benchmark to the recovery menu directly. Benchmarking each mount for read and for write speeds. Maybe I can use backup/restore code and just time it without writing (cp to null). Lets get to work
Thanks sztupy!
Sent from my GT-I9000 using XDA App

_JKay_ said:
Its better to let other people brick their phone than you brick your own
Just kidding! I for one will try and compile different versions with different lag-schemes to find the fastest combination. Including /system with fastest reading. I think I will add a benchmark to the recovery menu directly. Benchmarking each mount for read and for write speeds. Maybe I can use backup/restore code and just time it without writing (cp to null). Lets get to work
Thanks sztupy!
Sent from my GT-I9000 using XDA App
Click to expand...
Click to collapse
You cannot brick your phone with a bad kernel

@sztupy
I am impressed how productive master mind and kind person you are sharing all your knowledge and developments with us.
Congratulation, with you contributions you push Android Development forward.
Thank you

this is really cool, if i need minimal GUI, what would you suggest? xfce ? or there is something smaller and lighter? thx again.

avary said:
this is really cool, if i need minimal GUI, what would you suggest? xfce ? or there is something smaller and lighter? thx again.
Click to expand...
Click to collapse
Fluxbox/Openbox. However I don't see the need for one, if you're running the machine just for compiling.
Also, shouldn't gpm+guest additions solve the copy-paste problem?
E: Apparently not. Meh, stupid virtualbox.. sshd+putty then!

Awesome! Thanks for this sztupy! The more people we can get involved in this stuff, the better the end result will be. Always!

aziztcf said:
Fluxbox/Openbox. However I don't see the need for one, if you're running the machine just for compiling.
Also, shouldn't gpm+guest additions solve the copy-paste problem?
E: Apparently not. Meh, stupid virtualbox.. sshd+putty then!
Click to expand...
Click to collapse
actullay i was looking for something light and minimal (with GUI, or at least something like PCman FM and gedit or kedit, im not good with emac, vi ) to play and compile AOSP, for now i do this with Kubuntu dual boot on my laptop. if i put a litghweight desktop on top of szytup's image and use it with VB, that would be great.
thank you !

avary said:
actullay i was looking for something light and minimal (with GUI, or at least something like PCman FM and gedit or kedit, im not good with emac, vi ) to play and compile AOSP, for now i do this with Kubuntu dual boot on my laptop. if i put a litghweight desktop on top of szytup's image and use it with VB, that would be great.
thank you !
Click to expand...
Click to collapse
I'd use a shared folder+IDE/editor/whatever on windows. But that's just because I have so awful experiences about X in VMs, things might be better nowadays

Thanks a lot sztupy, will try it now. Sunday is enough time for testing
Each morning i have a look at XDA and you have brought out a new very helpfully posting, kernel, rom or else!
So let me ask you one question: when do YOU sleep?

avary said:
actullay i was looking for something light and minimal (with GUI, or at least something like PCman FM and gedit or kedit, im not good with emac, vi ) to play and compile AOSP, for now i do this with Kubuntu dual boot on my laptop. if i put a litghweight desktop on top of szytup's image and use it with VB, that would be great.
thank you !
Click to expand...
Click to collapse
The advantage of using X is that afaik guest additions has copy-paste support, that would make modifying easier.
I also plan on making some simple scripts inside the default home directory one can run to communicate with the shared folder. After that we could provide simple scripts that you only have to put inside the shared folder, which will download, modify and run the compilation.

FTDI kernel module unknown relocation

Hey guys,
I've spent a few days now researching and attempting to build the FTDI kernel module for my Archos 70b. After following this guide I managed to get a successful compilation. I uploaded it to my device (which was rooted via Paul's root and rebooted into sde). When I attempt to use insmod, I get the following error: insmod: cannot insert '/sdcard/ftdi_sio.ko': Invalid module format (-1): Exec format error. I ran 'dmesg -c' and got the following output: [ 684.472290] ftdi_sio: unknown relocation: 27. I have the usbserial module installed already as well. What is the problem here? I'm assuming by relocation it means it's loading the module into the wrong part of the address space? I could be very off I'm new to this process. I just want to thank everyone in advance for their help, this is my first post and I've used these forums extensively over the last few days. You have a great community here.
A little extra information:
I'm using the toolkit from the android-ndk-r8. My makefile looks like this:
obj-m := ftdi_sio.o
KDIR := ~/bin/gen8/archos-gpl-gen9-kernel/
PWD := $(shell pwd)
CCPATH := ~/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin
default:
$(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-linux-androideabi- -C $(KDIR) M=$(PWD) modules

whistlinwilly said:
Hey guys,
I've spent a few days now researching and attempting to build the FTDI kernel module for my Archos 70b. After following this guide I managed to get a successful compilation. I uploaded it to my device (which was rooted via Paul's root and rebooted into sde). When I attempt to use insmod, I get the following error: insmod: cannot insert '/sdcard/ftdi_sio.ko': Invalid module format (-1): Exec format error. I ran 'dmesg -c' and got the following output: [ 684.472290] ftdi_sio: unknown relocation: 27. I have the usbserial module installed already as well. What is the problem here? I'm assuming by relocation it means it's loading the module into the wrong part of the address space? I could be very off I'm new to this process. I just want to thank everyone in advance for their help, this is my first post and I've used these forums extensively over the last few days. You have a great community here.
A little extra information:
I'm using the toolkit from the android-ndk-r8. My makefile looks like this:
obj-m := ftdi_sio.o
KDIR := ~/bin/gen8/archos-gpl-gen9-kernel/
PWD := $(shell pwd)
CCPATH := ~/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin
default:
$(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-linux-androideabi- -C $(KDIR) M=$(PWD) modules
Click to expand...
Click to collapse
Hi !
First be sure that you have the same version kernel->module , run modinfo module.ko and vermagic need to be the same with kernel version !
- if it's ok you can try strings module.ko and than insmod module.ko !
( but can be many things wrong in your process of compiling module !!! ) Good luck ...

I can confirm that the versions are the same, I was using an incorrect version before and dmesg was complaining. I've since compiled the module for the right kernel version and now dmesg only gives me the "unknown relocation: 27" error.
I ran the strings command, it outputted over 4000 lines. I'm not quite sure what you could gain from the output but I can attached the text file I piped it to if that will be helpful.

whistlinwilly said:
I can confirm that the versions are the same, I was using an incorrect version before and dmesg was complaining. I've since compiled the module for the right kernel version and now dmesg only gives me the "unknown relocation: 27" error.
I ran the strings command, it outputted over 4000 lines. I'm not quite sure what you could gain from the output but I can attached the text file I piped it to if that will be helpful.
Click to expand...
Click to collapse
Ok!
May be you have a problem when your module "export symbols" -> take a look into Module.symvers ... and see if something export !

I looked around and couldn't find Module.symvers either in the source code or on the actual device. Do you know where I might be able to find it or how I can create the file myself? When the module is compiled, make throws a warning about not being able to find it. I think this is the problem.

whistlinwilly said:
I looked around and couldn't find Module.symvers either in the source code or on the actual device. Do you know where I might be able to find it or how I can create the file myself? When the module is compiled, make throws a warning about not being able to find it. I think this is the problem.
Click to expand...
Click to collapse
Hi !
...maybe it help you => http://tldp.org/LDP/lkmpg/2.6/html/lkmpg.html
( module.symvers is created after you run make ( for make module) in directory where you compile module )

I'm not quite sure I understand. I get a warning
WARNING: Symbol fakepath/kernel/Module.symvers
is missing; modules will have no dependencies and modversions.
when I run make in the modules folder. There IS a file created called Module.symvers after I run make, but it is empty; I'm assuming its just a copy of whatever Module.symvers file it is actually looking for.
Has anyone compiled the archos gen9 kernel that might have created this file and can pass it along to me?

whistlinwilly said:
A little extra information:
I'm using the toolkit from the android-ndk-r8. My makefile looks like this:
obj-m := ftdi_sio.o
KDIR := ~/bin/gen8/archos-gpl-gen9-kernel/
PWD := $(shell pwd)
CCPATH := ~/android-ndk-r8/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin
default:
$(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-linux-androideabi- -C $(KDIR) M=$(PWD) modules
Click to expand...
Click to collapse
Hi whistlinwilly
Are you sure you've grabbed the right kernel source - your KDIR says gen9, if your compiling for the 70b you probably need the gen8 kernel source code as I think it's on the Gen8 product line
you can check which kernel version you have by running the following from a command prompt.
Code:
adb shell uname -a
Hope that helps

Hey whistlinwilly,
i don't know if you were successful in the meantime, but here's what i suggest for simple kernel module compilation...
You might refer to my custom kernel thread and check my toolchain, which is ready to rock (only 32bit linux right now):
http://forum.xda-developers.com/showthread.php?t=1328027
To compile custom kernel/modules only, look here as well:
http://forum.xda-developers.com/showpost.php?p=19134490&postcount=8
The archos specific toolchain uses uclibc and is used for kernel and userland.
So if you use stock kernel, but choose a different toolchain (e.g. libc based) to compile your modules, you might run into trouble at some point.
BTW, is it correct that you talk about this device:
http://www.archos.com/products/ta/archos_70b/index.html?country=de&lang=de
Not sure which family it exactly belongs to
If you're talking about this one:
http://www.archos.com/products/ta/archos_70it2/index.html?country=de&lang=de
...the kernel should be 2.6.35 then.
http://gitorious.org/archos/archos-gpl-gen9-kernel
I'm not aware of any config for the A70IT2 though.
Please refer to trevd's suggestion and find out your kernel release first...
Good luck!
scholbert

I had the same problem (Relocation error: 27) when trying to compile a custom kernel module for my Moto xoom running Honeycomb.
I was able to compile successfully by rolling back to Android NDK 5b. I also had to force the vermagic string to read ARMv7 so I know this isn't the right way to solve the problem, but when I compiled with the 5b toolchain, the relocation error goes away and I am able to insmod without error on the tablet.
I would love to know why this works and the right way to get NDK 7 to work with this, if anyone can shed some light??
EDIT:
On further digging and testing- I realize it is not necessarily NDK 5b, but rather the toolchain "arm-eabi-4.4.0" as opposed to using the "arm-linux-androideabi-4.4.3" toolchain, that makes the difference with the "unknown relocation 27" error. It just so happens that NDK 5b is the last NDK that shipped with both toolchains, AFAIK.
It still for some reason likes to compile under ARMv5 unless I force it to use ARMv7 but this issue is probably unrelated- I didn't mean to confuse things with the ARM v5/v7 versus NDKr5 /NDKr7 which don't correlate...
Anyway I hope this helps, or maybe someone else who knows more can help us both!

Well, I solved my toolchain issue- I just needed to add an EXTRA_CFLAGS=-fno-pic to get the arm-linux-androideabi-4.4.3 toolchain to compile without seeing an unknown relocation error on the tablet.
Hope this helps someone!

the_zuck said:
Well, I solved my toolchain issue- I just needed to add an EXTRA_CFLAGS=-fno-pic to get the arm-linux-androideabi-4.4.3 toolchain to compile without seeing an unknown relocation error on the tablet.
Hope this helps someone!
Click to expand...
Click to collapse
Thanks a million, finally got past the relocation 27 error now. Insmod doesn't complain anymore and the drivers load successfully, now to just figure out why I'm getting a backtrace on device insertion.
Thanks for solving my major headache tho

Exec format error
the_zuck said:
Well, I solved my toolchain issue- I just needed to add an EXTRA_CFLAGS=-fno-pic to get the arm-linux-androideabi-4.4.3 toolchain to compile without seeing an unknown relocation error on the tablet.
Hope this helps someone!
Click to expand...
Click to collapse
Ok, thanks for the answer. My first post here.
After this got solved. I still have "Exec format error". And! and no errors are display in dmesg. Are there any other places where the error might be displayed? I just created a simple printk("hello world") project.
If i remove the printk it doesn't show the error.
I don't know whether to laugh or cry. I dabbled my self to the brink of madness to solve this. I couldn't find a solution anywhere. Finally solved it after 6 weeks of my time. Probably my boss at work will not be happy if he finds I was working on this instead of working.
And I find the solution here 2 days after I solved it. WOW!!!:crying::crying:
But my saga still continues...

[TOOL] A QUICK Android OTA payload dumper

Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf
Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)
Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin
Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.

ssssut said:
Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf
Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)
Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin
Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.
Click to expand...
Click to collapse
Thanks for creating this, I wanted to give it a try on Windows but it came out this error: liblzma-5.dll not found. Do I need to install any per-requisite? Thanks
EDIT: managed to get the dll from here https://tukaani.org/xz/ and it's all working nicely.

Works great. As a Mac user this is extremely helpful!

Another very good option! cheers

zellleonhart said:
Thanks for creating this, I wanted to give it a try on Windows but it came out this error: liblzma-5.dll not found. Do I need to install any per-requisite? Thanks
EDIT: managed to get the dll from here https://tukaani.org/xz/ and it's all working nicely.
Click to expand...
Click to collapse
can i ask how to install liblzma-5 please? in system? in the program?

mixlex said:
can i ask how to install liblzma-5 please? in system? in the program?
Click to expand...
Click to collapse
You just put the .dll in the same directory as the payload-dumper-go .exe; the issue could be pretty easily avoided if it were compiled static.
In fact, I spent some time today figuring out how to static cross-compile payload-dumper-go from my Ubuntu VM to Win32, Linux x86 and armhf, since it's usually better to go for lowest common denominator, and of course having arm since on-device is where payload-dumper-go might be most useful!
After digging into the recent Docker commit for some hints, then adding stripping and disabling DWARF debugging info generation to have the smallest binary possible, here are my notes for Linux x86:
Bash:
# install latest Go (currently 1.16.2) to /usr/local/go per the Linux instructions at https://golang.org/doc/install
export PATH=$PATH:/usr/local/go/bin
git clone https://github.com/ssut/payload-dumper-go
cd payload-dumper-go
apt-get install liblzma-dev
GOOS=linux GOARCH=386 CGO_ENABLED=1 CC=i686-linux-gnu-gcc go build -a -ldflags '-extldflags "-static -s -w"'
Then, I found that payload-dumper-go's go-xz dependency also in turn being dependent on the toolchain hopefully containing liblzma is extremely problematic/frustrating for Go cross-compiling, but was able to hack the MSYS2 mingw-w64-i686-xz liblzma into the Ubuntu mingw-w64 toolchain to make a static Win32 build:
Bash:
apt-get install mingw-w64
# install include and lib from https://packages.msys2.org/package/mingw-w64-i686-xz to /usr/i686-w64-mingw32
GOOS=windows GOARCH=386 CGO_ENABLED=1 CC=i686-w64-mingw32-gcc go build -a -ldflags '-extldflags "-static -s -w"'
And finally, for Android, NDK gcc wasn't cooperating with `go build` but, since we're building static, Linux armhf will still work fine, but we still need a similar trick to get Ubuntu's own armhf liblzma into the armhf toolchain:
Bash:
apt-get install gcc-arm-linux-gnueabihf
# install include and lib from https://launchpad.net/ubuntu/bionic/armhf/liblzma-dev/5.2.2-1.3 to /usr/arm-linux-gnueabihf
GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build -a -ldflags '-extldflags "-static -s -w"'
I also noticed it doesn't print instructions even though there are some in the code, and have added a PR to fix that: https://github.com/ssut/payload-dumper-go/pull/5
Hopefully @ssssut will still see about adding Incremental OTA support at some point, maybe do something about go-xz to make cross-compiling easier, and ideally add a feature to only dump specific partitions, since extracting the entire payload.bin can be time-consuming (and RAM-consuming!) when all you want is boot.img.
So, without further ado, here are my builds:
[ Attachments removed since they're now superseded by CI releases on GitHub in all major architectures! ]

Made a Magisk module with a wrapper to get the arm build working smoothly on-device: https://forum.xda-developers.com/t/...ices-platforms.2239421/page-149#post-84753275

osm0sis said:
ideally add a feature to only dump specific partitions
Click to expand...
Click to collapse
There is python variant of dumper with this feature (if anyone interested).
GitHub - sabpprook/payload_dumper
Contribute to sabpprook/payload_dumper development by creating an account on GitHub.
github.com

Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)

hayvan96 said:
Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)
Click to expand...
Click to collapse
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."

OK I'll try to extract it on my phone directly then

osm0sis said:
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Click to expand...
Click to collapse
Ok fixed by dumping the xz5.2.5 libs in the same directory (as instructed above) -- but I had to rename libzlma.dll to libzlma-5.dll, maybe this should be added to have a working fix.

hayvan96 said:
Ok fixed by dumping the xz5.2.5 libs in the same directory (as instructed above) -- but I had to rename libzlma.dll to libzlma-5.dll, maybe this should be added to have a working fix.
Click to expand...
Click to collapse
Even with my static compiles? Weird..

osm0sis said:
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Click to expand...
Click to collapse
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract!
Hopefully @ssssut can make some new official binary release builds (static this time ), and I'll be happy to post some for any architectures not covered and update my Magisk module.

osm0sis said:
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract!
Hopefully @ssssut can make some new official binary release builds (static this time ), and I'll be happy to post some for any architectures not covered and update my Magisk module.
Click to expand...
Click to collapse
Man it was using 7GB of RAM here, I had to fix it.

v1.1.0 is up thanks to some more solid work from @LuK1337! Now it automatically builds in all major architectures.
Releases · ssut/payload-dumper-go
an android OTA payload dumper written in Go. Contribute to ssut/payload-dumper-go development by creating an account on GitHub.
github.com
Updated Magisk module to v1.1.0 as well: https://forum.xda-developers.com/t/...ices-platforms.2239421/page-149#post-84753275
Probably solid now until Incremental OTA support can be looked into.

@ssssut - I tried a few different versions of windows_386 payload-dumper-go up to 1.1.1 on Windows XP Professional with Service Pack 3. Unfortunately, the payload-dumper-go software does not work. When trying to execute payload-dumper-go.exe, I receive the following error message:
[Path the executable]\payload-dumper-go.exe is not a valid Win32 application.
Click to expand...
Click to collapse
Please update the software so that it may work on Windows XP Professional with Service Pack 3.

Hmm I think it worked fine for me on Windows 10 x86 last I checked, so I guess it just doesn't support XP.. Not sure if there's anything to be done for that.

Very useful tool! Updated my firmware on [email protected]
Thank you!

thanks, bro...
very simple but pretty useful tool