Hi All
While I am a relative noob when it comes to Android development (I have been an embedded developer for over 25 years), I am pretty good at creating custom embedded kernels for MIPs targets. However, this issue seems to be getting the best of me.
Environment:
Ubuntu 10.04 + any automatic upgrades
HTC desire phone, rooted, running cyanogenmod 6.0.1 (and 6.1.0 from the daily builds)
Issue: I am trying to add batman-adv to the phone, but I cannot create a batman-adv.ko that insmod will load. When I put my ko file on the phone and try to load it I get "insmod: init_module 'batman-adv.ko' failed (Exec format error)" (Yes I am in su mode). I know that this generally indicates that I have built for the incorrect architecture and I suppose that is the real question. How do I GET the correct architecture.
I extracted a ko from the phone that works on the phone and ran 'file' on it:
ah6.ko: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
and this is what I get on my ko
batman-adv.ko: ELF 32-bit LSB relocatable, ARM, version 1 (SYSV), not stripped
Based on this, the architecture is correct (or appears to be correct).
EDIT1:
I am getting a bit closer. I get this in dmesg
batman_adv: version magic '2.6.35.4-cyanogenmod-gb7da4b8-dirty preempt mod_unload ARMv7 ' should be '2.6.35.4-cyanogenmod preempt mod_unload ARMv7
Looks like I need a way to remove the "-gb7da4b8-dirty" portion of the string (actually avoid having it there in the first place).
End EDIT1
Build environment:
I have downloaded (via repo / git) and built the cyanogenmod SDK (daily build from 9/11/2010 kern: 2.6.35.4). The output of a build of this repo is an update-foo.zip file which I loaded on the phone and verified that it works properly. The repo includes a kernel-msm directory with (an apparently) fully functional kernel build tree (as well as the sdk and ndk trees).
I have pulled the config.gz file from the (working nicely) phone (update-foo.zip version), unzipped it and fed it into the build system. Answered the annoying questions you get when you merge in a new config file and added batman-adv to the selected loadable modules list to be built (also had to play with the file to fix a few other issues).
Build script looks like this (CCOMPILER points to the pre-built compiler in the sdk):
cd $SYSTEM_PATH || exit
. build/envsetup.sh
lunch cyanogen_bravo-eng
cd $SYSTEM_PATH/kernel-msm || exit
make ARCH=arm CROSS_COMPILE=$CCOMPILER -j$CPU_COUNT clean || exit
make ARCH=arm CROSS_COMPILE=$CCOMPILER -j$CPU_COUNT | tee build.log ||
The build completes just fine and I get all of the expected ko's. I have not tested the kernel itself since I am not really interested in it. It should be the same as the kernel built by the SDK and included in the update-foo.zip file. Worst case the KO would load and crash if the ko and the kernel on the phone were incompatible (I would be happy to get that far).
Issue is resolved.
While the insmod function returned a bad format error, it was really complaining about the magic number mismatch. Once I took care of matching the ARMv5 vs ARMv7 issue and then did an over ride on the KERNELVERSION string, I was able to load and execute the ko without any additional issues.
Wow! You do like to write don't you mate?
I'm glad it got resolved....should I close this thread or you want to get some more input?
Cheers,
We can close the thread. I write so much because I see so many post with insufficiant information to make a good diagnosis. In addition, those threads also tend to wander off the subject, making them a bit less than usefull.
Martin
I believe I am trying to do the same thing here, but for a different phone.
You say you're adding your module to the selected loadable modules to be built. Isn't there a way to build a loadable module by itself without rebuilding or changing kernel settings?
I have been able to compile/install modules for ubuntu linux but I haven't been able to figure out how to compile them for my android device. I have the source tree for the kernel build I'm using but I'm not quite sure how to go about cross compiling my module for it...
So I must be missing something here. On armel devices, a very simple (unrar compiled from source) binary would run as long it was properly built for armel.. This doesn't seem to be the case for x86_64. What am I missing? I even tried a x86-32 build (despite uname -a saying it is indeed running on x86_64). Any pointers? I usually bring aria2, 7z, and unrar at the very least.. 7z is a bit more complicated than unrar due to it not being a static binary. Despite the awesome X86ness, this phone is irritating; over-willingly catering to stupidity it seems.. Lollipop is awful.
At least it should give you some error message...
data/local/unrar <
tmp-mksh: /data/local/unrar: not found
data/local/bin/unrar <
tmp-mksh: /data/local/bin/unrar: No such file or directory
This is all the error I get. The second output is where the correct location of the file, the first input is just to show the difference between an incorrect location and the binary failing to run. I'm a bit rusty but I believe this is similar behavior to trying to run an armel compiled program that relies on external libraries.
So I just tried aria2 pre-built for android to see if it had an x86 variant. To my surprise, it ran.. To my further surprise:
Compiler: gcc 4.9 20140827 (prerelease)
built by x86_64-pc-linux-gnu
targetting arm-unknown-linux-androideabi
on Feb 23 2015 23:55:54
System: Linux 3.10.20-x86_64_moor-g9f2abf6 #1 SMP PREEMPT Fri May 15 16:07:23 CST 2015 armv7l
Yeah, you see that target? Or perhaps the bit at the end? ARM, yet my device is clearly x86_64. Should I be pissed or impressed? Perhaps this is a Term em thing? Couldn't mount loop device either.. But that's a tangent. I really wish this phone ran Ubuntu. I'm going to try compiling unrar for armel, it'll probably run.. While x86 won't.. That'll piss me off. I knew I recognized that "No such file or directory" error..
Ugh, I forgot about PIE requirements for lolli. (Un)fortunately, unrar works (armel)... But both (armel) 7z and par2 throw an error about not being PIE. Again, unrar for _compiled for armel_ works, while x86_64 does not. Also, there's an entire folder for doppelganger armel libs in /system/lib/ . I though I bought an Intel-based device, not an Intel based armel emulator. It's so irritating how many layers of junk are added to formerly decent OS/UIs just to cater to windowlickers..
Like anyone cares.. heh. I managed to bork up my linker binary by trying to remove PIE check.. Whoops, kind of essential.. Result is same as when I try to run x86_64 compiled programs. Adorable. I fixed it by booting into the fugazi-CWM for this phone (that does not have root access?!).. creating a flashable zip that restored previous linker binary.. but not before having to find a proper x86 binary-updater binary that wasn't a script (SuperSU you sneaky).
Blades said:
So I must be missing something here.
I even tried a x86-32 build (despite uname -a saying it is indeed running on x86_64). Any pointers?
Click to expand...
Click to collapse
Any further discovery ?
I'd need some binaries working inside ZE551ML terminal (ssh sessions)... for instance:
zip/unzip (full), rar/unrar, md5deep, dcfldd, nano, ...
Is there any way statically compiled ones from a std linux x86_64 distro will work ?
Any hint/guide to cross compile for Intel and not for ARM ?
Thanks
I also tried to remove PIE security check and phone would boot into recovery. I was trying to disable cellular radiomodem and bins were not executing.
You only need to replace linker file with oem to boot again.
EDIT: This issue was caused by a change to the extract-files.sh. Please disregard
Hi guys,
I'm trying to make my first build (V410 with minor source changes).
I'm going through the guide (http://wiki.lineageos.org/v410_build.html) but came across an issue I haven't been able to solve. When extracting proprietary blobs I get the following (in terminal):
"./../../lge/v410/extract-files.sh: 7: ./../../lge/v410/extract-files.sh: Cannot fork "
Since I am using a remote server I am extracting from the CM14.1 ROM which I've extracted into
/home/Downloads/lineage
my terminal looks as follows:
[email protected]:~/android/system/device/lge/v410# ./extract-files.sh /home/Downloads/lineage
./../../lge/v410/extract-files.sh: 7: ./../../lge/v410/extract-files.sh: Cannot fork
I cannot find extracted blobs in ~/android/system/vendor/lge
Any thoughts?
hello guys,
i hope you can give me a tip to solve this issue. i try to build lineageos 13 from source. I have to say: it's an unoffical mtk device.
-> ubuntu 16.04 64bit
-> i used Java 7 JDK
-> installed all listed apt's (had to install maven also)
so i used a guide for official devices: http://mikestechlife.blogspot.de/2016/11/setting-up-ubuntu-to-compile.html
but my vendor and device trees are pasted in vendor/vendorname/codename and device/vendorname/codename
Vendor blobs: https://github.com/pedropereira22/android_vendor_leagoo_z5
Device Tree: https://github.com/pedropereira22/android_device_leagoo_z5
if i start brunch codename, it begins and finished after ~40-50 minutes.
#### make completed successfully (46:26 (mm:ss)) ####
when i check the zip-file in the out-folder, there are some files missing. so in framework are only 4 files and some app / priv-apps are missing. what is my problem if the compiiler shows no error? i can upload the zip-file as well if you want to take a look. maybe you have an idea
(im sorry for 2x the thread in different forums here but did not see lineage forum before)
Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf
Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)
Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin
Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.
ssssut said:
Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf
Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)
Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin
Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.
Click to expand...
Click to collapse
Thanks for creating this, I wanted to give it a try on Windows but it came out this error: liblzma-5.dll not found. Do I need to install any per-requisite? Thanks
EDIT: managed to get the dll from here https://tukaani.org/xz/ and it's all working nicely.
Works great. As a Mac user this is extremely helpful!
Another very good option! cheers
zellleonhart said:
Thanks for creating this, I wanted to give it a try on Windows but it came out this error: liblzma-5.dll not found. Do I need to install any per-requisite? Thanks
EDIT: managed to get the dll from here https://tukaani.org/xz/ and it's all working nicely.
Click to expand...
Click to collapse
can i ask how to install liblzma-5 please? in system? in the program?
mixlex said:
can i ask how to install liblzma-5 please? in system? in the program?
Click to expand...
Click to collapse
You just put the .dll in the same directory as the payload-dumper-go .exe; the issue could be pretty easily avoided if it were compiled static.
In fact, I spent some time today figuring out how to static cross-compile payload-dumper-go from my Ubuntu VM to Win32, Linux x86 and armhf, since it's usually better to go for lowest common denominator, and of course having arm since on-device is where payload-dumper-go might be most useful!
After digging into the recent Docker commit for some hints, then adding stripping and disabling DWARF debugging info generation to have the smallest binary possible, here are my notes for Linux x86:
Bash:
# install latest Go (currently 1.16.2) to /usr/local/go per the Linux instructions at https://golang.org/doc/install
export PATH=$PATH:/usr/local/go/bin
git clone https://github.com/ssut/payload-dumper-go
cd payload-dumper-go
apt-get install liblzma-dev
GOOS=linux GOARCH=386 CGO_ENABLED=1 CC=i686-linux-gnu-gcc go build -a -ldflags '-extldflags "-static -s -w"'
Then, I found that payload-dumper-go's go-xz dependency also in turn being dependent on the toolchain hopefully containing liblzma is extremely problematic/frustrating for Go cross-compiling, but was able to hack the MSYS2 mingw-w64-i686-xz liblzma into the Ubuntu mingw-w64 toolchain to make a static Win32 build:
Bash:
apt-get install mingw-w64
# install include and lib from https://packages.msys2.org/package/mingw-w64-i686-xz to /usr/i686-w64-mingw32
GOOS=windows GOARCH=386 CGO_ENABLED=1 CC=i686-w64-mingw32-gcc go build -a -ldflags '-extldflags "-static -s -w"'
And finally, for Android, NDK gcc wasn't cooperating with `go build` but, since we're building static, Linux armhf will still work fine, but we still need a similar trick to get Ubuntu's own armhf liblzma into the armhf toolchain:
Bash:
apt-get install gcc-arm-linux-gnueabihf
# install include and lib from https://launchpad.net/ubuntu/bionic/armhf/liblzma-dev/5.2.2-1.3 to /usr/arm-linux-gnueabihf
GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build -a -ldflags '-extldflags "-static -s -w"'
I also noticed it doesn't print instructions even though there are some in the code, and have added a PR to fix that: https://github.com/ssut/payload-dumper-go/pull/5
Hopefully @ssssut will still see about adding Incremental OTA support at some point, maybe do something about go-xz to make cross-compiling easier, and ideally add a feature to only dump specific partitions, since extracting the entire payload.bin can be time-consuming (and RAM-consuming!) when all you want is boot.img.
So, without further ado, here are my builds:
[ Attachments removed since they're now superseded by CI releases on GitHub in all major architectures! ]
Made a Magisk module with a wrapper to get the arm build working smoothly on-device: https://forum.xda-developers.com/t/...ices-platforms.2239421/page-149#post-84753275
osm0sis said:
ideally add a feature to only dump specific partitions
Click to expand...
Click to collapse
There is python variant of dumper with this feature (if anyone interested).
GitHub - sabpprook/payload_dumper
Contribute to sabpprook/payload_dumper development by creating an account on GitHub.
github.com
Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)
hayvan96 said:
Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)
Click to expand...
Click to collapse
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
OK I'll try to extract it on my phone directly then
osm0sis said:
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Click to expand...
Click to collapse
Ok fixed by dumping the xz5.2.5 libs in the same directory (as instructed above) -- but I had to rename libzlma.dll to libzlma-5.dll, maybe this should be added to have a working fix.
hayvan96 said:
Ok fixed by dumping the xz5.2.5 libs in the same directory (as instructed above) -- but I had to rename libzlma.dll to libzlma-5.dll, maybe this should be added to have a working fix.
Click to expand...
Click to collapse
Even with my static compiles? Weird..
osm0sis said:
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Click to expand...
Click to collapse
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract!
Hopefully @ssssut can make some new official binary release builds (static this time ), and I'll be happy to post some for any architectures not covered and update my Magisk module.
osm0sis said:
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract!
Hopefully @ssssut can make some new official binary release builds (static this time ), and I'll be happy to post some for any architectures not covered and update my Magisk module.
Click to expand...
Click to collapse
Man it was using 7GB of RAM here, I had to fix it.
v1.1.0 is up thanks to some more solid work from @LuK1337! Now it automatically builds in all major architectures.
Releases · ssut/payload-dumper-go
an android OTA payload dumper written in Go. Contribute to ssut/payload-dumper-go development by creating an account on GitHub.
github.com
Updated Magisk module to v1.1.0 as well: https://forum.xda-developers.com/t/...ices-platforms.2239421/page-149#post-84753275
Probably solid now until Incremental OTA support can be looked into.
@ssssut - I tried a few different versions of windows_386 payload-dumper-go up to 1.1.1 on Windows XP Professional with Service Pack 3. Unfortunately, the payload-dumper-go software does not work. When trying to execute payload-dumper-go.exe, I receive the following error message:
[Path the executable]\payload-dumper-go.exe is not a valid Win32 application.
Click to expand...
Click to collapse
Please update the software so that it may work on Windows XP Professional with Service Pack 3.
Hmm I think it worked fine for me on Windows 10 x86 last I checked, so I guess it just doesn't support XP.. Not sure if there's anything to be done for that.
Very useful tool! Updated my firmware on [email protected]
Thank you!
thanks, bro...
very simple but pretty useful tool