***THIS POST IS NOT COMPLETE, I WILL UPDATE MORE LATER***
First, an introduction:
The Image Update system allows the OEM (us! ) to issue updates to a "Live" filesystem - without disrupting user data. This allows, for example, a buggy driver to be updated after the phone has been shipped, or a software package to be updated to the latest version, with minimal knowledge on the user's part. The system validates all updates against an internal list of certificates, and refuses the update if a match is not obtained. This system can also be used to deploy entirely new software to the device (such as support for another locale, input method editor, application support for a new feature the carrier is rolling out, etc.)
Potential usage scenarios for this system
A central server could be maintained for all SYS/OEM updates - each ROM Chef would need to maintain a list of original packages, any updated package(s), and download URL's for each updated package. The user would then receive these updates through the built-in AutoUpdate facility in Windows Mobile, which can check periodically, or on-demand. Each Chef could maintain seperate download servers from the update server to minimize server load.
Alternatively, a chef could provide .cab.pkg updates in his or her ROM thread, on their own web site, etc., and the user could download these and install them at will. These packages can optionally be authenticated to be coming from the Chef, if the Chef wants to ensure updates are coming from him only. A public certificate could also be used to allow users to issue updates as well.
The more technical Summary
Image Update allows an OEM to issue updates to the OEM's, XIP, SYS, (possibly) Radio, or any combination of these. The update can be pushed to the user via a specially formatted SMS or by manual execution. There are at least 2 levels of certificate checking involved in the process, I believe against \SYS\Metadata\DefaultCerts.dat. The system reboots into the ULDR to apply the update, because the filesystem cannot be modified while actively mounted. The ULDR provides a minimal operating enviornment to facilitate this.
How does a Chef need to prepare a ROM for Image Updating?
The Chef would need to use a ROM Kitchen that leaves the .dsm and .rgu file structure intact (i.e. an "unprotected" ROM) - All .dsm's in this ROM would need to be properly formatted with Package Name, versioning info, etc. during the cooking process, in order to facilitate version checking, etc. Each .dsm would need to be signed with a certificate the Chef would use to validate the update as coming from him or her. (Alternatively a public certificate could be used like SDKCerts if the Chef chooses not to maintain direct control over updates, and allow other users to create updates as well)
The Chef also needs to ensure not to Reduce the ULDR partition or remove it; this would cripple the Update Loader and prevent the Image Update system from functioning.
The .cab.pkg format
At the root of a package, the .dsm defines the Package structure (all files, registry entries, etc.) It contains version info, certificates, and other data. A ROM consists of a number of these packages, in an area of flash memory that is not user-writable. When someone wants to issue an update using the ImageUpdate system, they create a matching .dsm, same guid, with a newer version number, same internal package name, same processor ID, os version, etc., there is also a flag that can be marked as an update package or a new package - in this .dsm they define the files that will make up the new, updated package. Here they can add or remove files. One of the files defined by the package is optionally an .rgu, and this defines the registry entries associated with the package. Also optionally included is a provxml to facilitate file operations (copy/replace/delete/rename/etc.) and further registry or metabase operations. This collection of files is rolled up into a ".cab.pkg" archive by a program like cabarc. Once in a pkg.cab format, the package is signed.
A .cab.pkg is referred to as a "Canonical Package" and multiple Canonical Packages can be rolled up into a single "Super Package" to facilitate updating multiple Packages at the same time.
Inside this .cab.pkg is where the "MNGE" file format comes in to play. Essentially, this format is Microsoft's way of storing an XIP Module in the filesystem. (Their equivalent of our imageinfo.bin + s000, s001, etc.) - The "MNGE" format is simply a 4-byte MNGE header, followed by the imageinfo.bin, followed by the s00x sections attached to the end. When the Image Update system processes a file in this MNGE format, it is added to the IMGFS as an XIP Module.
Deploying a .cab.pkg to a device
Now there are several ways to deploy this package to the device, the primary way is an OMA-DM SMS message, which triggers the Image Update system to initiate a download from a server defined in the SMS message. The server can be either a normal HTTP server or a secure HTTPS server. The Image Update system will prompt the user to plug in the usb cable to download over the users home connection if the user has not already authorized the Auto Update system to utilize their GPRS data connection.
Another way for a .cab.pkg update to be pushed to the system is simply through a file copy operation, be it ActiveSync, SD Card, Bluetooth, or otherwise. Once on the device, the .cab.pkg is executed by the user the same way a .cab would be.
The Update Agent
Initiated by either a completed download from push SMS, or user-executed, the "Update Agent" program (which is part of the \SYS\FWUPDATE Package) attempts to validate the Certificates, Package dependancies, and other info contained in the .cab.pkg and .dsm. Once validated, the "Update Agent" sets a flag that the bootloader reads, the flag is a boolean, off = boot into normal OS, on = boot into ULDR - so then the system reboots, the flag is read, and you load into...
The Update Loader
The "Update Loader" or "ULDR" which is a minimal kernel configuration, that provides just enough driver support to display info on screen, respond to user input, and read/write from the internal flash (NAND or NOR)
From here the ULDR does further validation on the .cab.pkg, and applies it to the filesystem. If there are any modules in the package it dynamically relocates the memory map to make sure there are no overlaps. This is why it's important that reloc's not be removed from your ROM - ULDR will fail in this case.
The End Result
Once the ULDR has completed updating, the device is again rebooted, back into the full system, where the now-updated packages are now a part of the IMGFS, any updated files are processed (.rgu, .provxml, etc.) - The package is now a full part of the ROM.
The new .dsm replaces the old .dsm (along with the other files in the package) and now a future update will be checked against this new package.
If the update was pushed via OMA-DM SMS, or AutoUpdate, the device Pushes a notification to the OMA-DM server notifying it of the update status.
What's missing right now to implent the ImageUpdate system?
We need a Kitchen that's properly configured to allow us to create versioning info, proper package names, and insert this along with a certificate (or multiple certificates) into the .dsm's.
We also need the Kitchen to be able to modify \SYS\Metadata\DefaultCerts.dat with the certificates used, so that it passes authentication. Alternatively the authentication checking could be patched out. (this one is easily doable at build-time)
We need a program that can convert from a standard file to an MNGE format, so we can implement modules in our .cab.pkg's. (done it seems, thanks ervius!)
We (optionally) need a properly configured web server that supports HTTP/HTTPS, can communicate the proper xml configuration data, and can be updated with new packages by Chefs. (this one's a ways off)
We (optionally) need a program to convert from MNGE format to a standard file to facilitate extracting modules from .cab.pkg's. (working hard on that)
I've attached a .cab.pkg for NetCF2. Open up mscoree.dll in a hex editor, and check out the MNGE header. This file becomes a module once processed by the ImageUpdate system. Note that all the executables (.exe/.dll/.mui) that become modules contain this MNGE header. All executables that are inserted into ROM as files keep their normal MZ file header. The first major step here will be in being able to convert between MZ<-->MNGE freely NetCF2 is a well known package that can be found in any stock ROM, so with this we have a good baseline to work with.
http://rapidshare.com/files/238295848/netcf.cab.pkg
--Reserved--
More Technical Specifications
The basic ImageUpdate Layout consists of:
[IPL] -- [MBR] -- [ULDR] -- [NK] -- [IMGFS] -- [TFAT]
[IPL] is the "Initial Program Loader" that handles basic init functions and determines if control should be handed over to ULDR, or NK through a flag set by UpdateBin.exe - the IPL is not contained within a partition. The IPL is copied entirely to RAM and executed from there. IPL loads NK into RAM, and also handles any decompression of NK if it's required - some SmartPhone's ive seen use SRPX compression for the NK partition. Once NK is copied to RAM it is then executed. The IPL is handled seperately from the other parts of the operating system, and is not flashed during a normal update.
[MBR] is the "Master Boot Record" and contains partition tables for the below components - it points to NK so when IPL loads the MBR, control is handed over to NK. The MBR contains information on where each partition is located on the flash (memory address), the size of the partition(s), and the type of each partition. The MBR is referenced from many components on the device such as IPL and ULDR in order to facilitate handoff of control between ULDR and NK. The MBR also serves as a boundry between the IPL which is not part of the regular partition structure, and the rest of the flash, which is part of the partitioning structure.
[ULDR] is the "Update Loader" and provides a basic WinMo system so that file operations can be done on the IMGFS partition while it's unmounted. The Update Loader is even able to update itself - during operation it is loaded entirely to RAM. On development workstations the ULDR supports a KITL connection, that can be used to load updates directly from the "Release" folder. It seems it may be possible through this method to flash a new image to the device, possibly opening up the ability to flash to devices that have not yet been flashed with "HardSPL"
[NK] is the "Kernel Partition" - or what we know as xip.bin - This component is updatable by ImageUpdate, and has a pre-defined "free space buffer" with room to grow, which defaults to 512KB. This partition holds only the kernel and drivers necessary to bring up the rest of the filesystem, from which the rest of WinMo is loaded. The Kernel Partition uses the same "Package" format as the IMGFS and is updatable in the same manner.
[IMGFS] is the "System Partition" - running the Image Update filesystem. This component is updatable by ImageUpdate, and has a pre-defined "free space buffer" with room to grow, which defaults to 9.5MB. The IMGFS uses the "Package" format to further split its components.
[TFAT] is the "Transaction Safe FAT File System" which is where all user-writable data goes.
In most Device Designs, there's a single NOR or NAND chip used for flash. This is important as due to the typical layout above, both NK and IMGFS must have a pre-defined amount of free space - because TFAT is the last partition on the drive, and cannot be shifted once flashed to the device. It's possible for the partition layout to be setup differently (Partitions in different order) to help alleviate that problem. The ImageUpdate system would really shine on a device with 2 flash chips, a NOR chip dedicated to the ImageUpdate partition and a NAND chip dedicated to the TFAT, but no OEM has created such a design yet.
Packages
Package Types
There are 3 different types of packages, Canonical, Update, and Super.
Canonical contains the entire contents of the package. It is used for a first-time package install, and if there are any major updates to be issued that would require the complete package. The file extension is .cab.pkg
Update contains a binary delta between a package already on the device, and the updated version of that package. In this manner the limited space is conserved (i.e. if a package change was a simple registry entry - no need to replace the 5mb of .dll and .exe in that package, just alter the .rgu with the new data. These packages are also referred to as "Delta" packages. The concept is similar to the unix implementation of Diff/Patch. The file extension is .cab.pku
Super contains a collection of update and/or canonical packages. This is very useful when you are attempting to bring in a new package that has dependencies on other packages - rather than reboot into ULDR for each individual package in the proper dependency order, they can all be introduced at once. Every package contained inside a super package is validated, and if one fails, the remaining valid updates may still be applied, as dependencies allow. A super package is simply an un-compressed .cab containing other packages, renamed to .cab.pks
The package layout itself is quite basic, it consists of a .dsm which contains all versioning info, association info, and dependency info. It also contains a list of all modules and files inside the package, and a certificate store of all approved certificates that will be allowed to update that package. Alongside the .dsm is an optional .rgu, which defines the registry settings associated with that package. Also optional is a .provxml file, which can be: mxip_[packagename]_[version].provxml, mxipcold_[packagename]_[version].provxml, or mxipupdate_[packagename]_[version].provxml. mxip and mxipcold are effectively treated the same, executed only on a cold boot. mxipupdate_ provxml's will be executed any time that package is updated, in addition to a cold boot - so if you are adding new .cab.pkgs and wish the .provxml to be executed immediately, it would need to be mxipupdate. This may not be desirable in some cases, such as when the provxml might override a user preference - in that case you would only want it to run on a cold boot, in order to avoid "strange" behavior on the user's side of things.
There is a "shadow order" defined in the .dsm as well - this controls what "priority" .rgu's are compiled together into the device registry hive - a package that shadows another package will override any .rgu entries that shadowed package may contain. This is important to consider when utilizing .cab.pkgs in order to obtain your desired end registry. This shadow order also applies to provxmls inside the package - a package that shadows another package will override its provxml settings as well.
The user registry hive is always top-most in the shadow order (except in the case of an mxipupdate_ provxml) - so any changes to .rgu registry settings will not override a user-changed registry setting. (example: You had foobar set to 5 in your initial deployment. At some point after flashing to his/her device, the user modified the registry, changing the value of foobar to 6. Your new .cab.pkg contains an .rgu changing foobar to 7 - on device, foobar will remain set to 6, as the user registry is higher in the shadow order than the .rgu) - in the case of an mxipupdate_ provxml these will override user settings.
So theoretically if we get this working we can apply updates to ANY portion of a rom via the .cab.pkg system. To XIP,SYS,or OEM without flashing a new rom. Essentially with this system working we would never need to flash again unless a major corruption occured?
Ive always been confused as to why autoupdate was included in roms still. I guess this sheds some light on it. I know that several individuals have toyed with OTA updates in the past. This could make that and a whole lot more a reality.
It seems to me silly that we aren't exploiting the MS autoupdate feature already...
I have no idea how to get it to work, but I'd love to have it working!
Yes, the .cab.pkg is the key to this whole thing, I already know how to redirect the Windows Mobile Auto Update client to look at another server, and i've studied the connection, it's a simple HTTPS connection, WinMo sends the server a manifest of all the .dsm's contained within your ROM, with version info, then the server checks against it's internal list of packages, if it finds an update, it pushes a URL to the device, which then triggers a download (it requests you to plug in the activesync cable if you've not checked the box to 'use my data connection for updates') - once the .cab.pkg is downloaded, it's checked against the signatures on the system, once verified the system reboots into the ULDR, and the update is applied.
I've attached a NetCF2 package to my first post, I can't get it to deploy on my ROM (fails during validation step) but it contains the modules in MNGE format, if we can decipher that format there's a whole bunch of goodies that will become available...
Also these packages can even be used to update the radio rom, it seems. So essentially everything but the Bootloader/ULDR can be updated with .cab.pkgs. It even looks like we can resize existing packages (i.e. remove files or modules from the ROM entirely) - this is something we can't do at all right now without a flash!
Da_G said:
Yes, the .cab.pkg is the key to this whole thing, I already know how to redirect the Windows Mobile Auto Update client to look at another server, and i've studied the connection, it's a simple HTTPS connection, WinMo sends the server a manifest of all the .dsm's contained within your ROM, with version info, then the server checks against it's internal list of packages, if it finds an update, it pushes a URL to the device, which then triggers a download (it requests you to plug in the activesync cable if you've not checked the box to 'use my data connection for updates') - once the .cab.pkg is downloaded, it's checked against the signatures on the system, once verified the system reboots into the ULDR, and the update is applied.
I've attached a NetCF2 package to my first post, I can't get it to deploy on my ROM (fails during validation step) but it contains the modules in MNGE format, if we can decipher that format there's a whole bunch of goodies that will become available...
Click to expand...
Click to collapse
So in theory, there could be a central place for SYS/XIP packages, where as and when new XIP/SYS updates come out, they can be uploaded and pushed to every device?
Have you worked out how to create the cab.pkg files, or is the one you've attached one taken from platform builder?
Just a thought:
Could the MNGE headered files not be replaced by files from a converted module, thus getting around the problem of what the hell the MNGE format does? Sure, we lose the ability to have modules instead of files, but it does bring more immediate benefits to your findings...
EDIT:
The MNGE headered files are smaller than the MZ equivalents... Are they simply a compressed version?
Yep, not pushed though as that needs to be triggered via an OMA-DM SMS message, and it's not practical for someone to maintain a database of all our numbers for such a purpose.. but easily though settings - autoupdate
I am able to extract files from .cab.pkg with winrar and 7zip, not able to create them just yet.. working on that. This one came from a blue birdy.
The MNGE headered files could indeed be replaced by a converted module, but in this case, there's a different reason for needing to convert from MNGE -> MZ,
It appears to me as though the file size difference had to do with the PE executable headers that are missing..
Da_G said:
Yep, not pushed though as that needs to be triggered via an OMA-DM SMS message, and it's not practical for someone to maintain a database of all our numbers for such a purpose.. but easily though settings - autoupdate
I am able to extract files from .cab.pkg with winrar and 7zip, not able to create them just yet.. working on that. This one came from a blue birdy.
The MNGE headered files could indeed be replaced by a converted module, but in this case, there's a different reason for needing to convert from MNGE -> MZ,
It appears to me as though the file size difference had to do with the PE executable headers that are missing..
Click to expand...
Click to collapse
Oh right. It's not hard to just check for updates every so often.
I just ran Cab2OEM on the cab.pkg files, and it extracts fine. So cab.pkg files are just cab files in terms of compression.
Is that because there are more up to date MNGE file versions than the MZ equivalents?
Is it just a case of replacing the file headers? *opens up hex edit*
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Da_G said:
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Click to expand...
Click to collapse
Where are you getting your MNGE files from? Or does your blue birdy wish to remain anonymous?
EDIT:
Looking at the hex, apart from the file header, the main difference seems to be that the whitespace has been removed in the MNGE version...
l3v5y said:
Where are you getting your MNGE files from? Or does your blue birdy wish to remain anonymous?
Click to expand...
Click to collapse
Do you know what a "hint" is ? I'd say quit asking... If he wanted to say who the birdy was, he would. Thank you!
Da_G said:
Yep. the compression is your typical cab compression. That's why winrar and 7zip can open 'em and extract, but they don't support adding (i imagine cabarc would...
Yes, there are more up to date MNGE file versions than the MZ equivalents.
Unfortunately it doesn't look quite as simple as a simple hex copypasta, not terribly much more difficult though.
Click to expand...
Click to collapse
put cabarc.exe intoa folder, then create a new subfolder called "package", put inside all files you need for package (dsm and rgu also!)
open in cabarc.exe root folder a dos prompt and write:
cabarc.exe N new_pack.cab.
pkg package\*.*
some seconds and you'll have the .pkg file ready, but nothing I know on how to install by the rom!
Thanks for the input ervius!
Installing into the ROM is simple, .cab.pkg is treated similar to a .cab by Windows Mobile, simply copying to device and clicking on it in file explorer allows you to install - device will authenticate signature, then do some further checking (i think checking on current packages in device by .dsm) - then once validated reboot into ULDR to apply update.
ervius said:
put cabarc.exe intoa folder, then create a new subfolder called "package", put inside all files you need for package (dsm and rgu also!)
open in cabarc.exe root folder a dos prompt and write:
cabarc.exe N new_pack.cab.
pkg package\*.*
some seconds and you'll have the .pkg file ready, but nothing I know on how to install by the rom!
Click to expand...
Click to collapse
for posted netcf example, the optimum is:
cabarc.exe - 20 N new_pack.cab.
pkg package\*.*
so, header (-s 20 reserve space for sign!), is same size, but how and with , sign it, I don't know!
Da_G said:
Thanks for the input ervius!
Installing into the ROM is simple, .cab.pkg is treated similar to a .cab by Windows Mobile, simply copying to device and clicking on it in file explorer allows you to install - device will authenticate signature, then do some further checking (i think checking on current packages in device by .dsm) - then once validated reboot into ULDR to apply update.
Click to expand...
Click to collapse
shure, but now maybe we have to fight against right sign code!?!?
bye!
Yes, I think the ROM will need be cooked with additional certs, in \SYS\Metadata\DefaultCerts.dat - these appear to be the certs that are being checked against. So we can replace with SDKCerts, sign .cab.pkgs with that, should be good!
Da_G said:
Yes, I think the ROM will need be cooked with additional certs, in \SYS\Metadata\DefaultCerts.dat - these appear to be the certs that are being checked against. So we can replace with SDKCerts, sign .cab.pkgs with that, should be good!
Click to expand...
Click to collapse
ok, go to work then, I ', yet thiniing about oldstyle buildos with this all new features of visualkitchen without platformrebuilder if someone wants use oldstyle (maybe I'm at my goal!)
ervius said:
ok, go to work then, I ', yet thiniing about oldstyle buildos with this all new features of visualkitchen without platformrebuilder if someone wants use oldstyle (maybe I'm at my goal!)
Click to expand...
Click to collapse
Can we not look at removing the signing check the same way cmonex did for the kernel? Or is that the same signing check?
Just so I am getting this right? I could cook a ROM with a custom DaveShaw https update server IP and then provide automatic updates to my ROM, bug fixes, new build release, normal CABs, etc. all using Windows Update from my website??
That would be damn useful, no more re-flashes
Dave
probably to create MNGE from module folder we just need
copy /b imageinfo.bin + S000 + S001 + ... module.dll
and add MNGE header to the beginning of the file.
but maybe i am wrong and this will not work in all cases.
at least when i converted in this way dll module from 21725 to MNGE and compared it to the same file which was originally in MNGE format, there were 0 differences.
Related
Today Samsung released a patch for their WP7 devices to resolve the update problems. WP7 version 7.0.7392.0 has been withdrawn for Samsung devices because of error 801812C1 during the update. After the release of this patch Microsoft has now resumed the release of the OS update for Samsung devices.
The update for Samsung Omnia7 can be downloaded from here.
I did a quick analysis of the file. It is an executable that should be run on a Windows PC. It installs the latest USB drivers for Samsung devices on the PC. Then a flash utility starts. A ROM-file is embedded as a resource in the flashtool. It's called OS.nb and it has a familiar Ra000FF header. So I tried to open it with the latest WP7 ROM Tools. It opens and it shows the "OS partition". But, unfortunately, as soon as I expand that node the WP7 ROM Tools crash. I hope that AnDim, the author of the WP7 ROM Tools, can work on that. The embedded ROM image is about 12 MB. It should contain only the files that are updated.
So this flash tool is able to replace some specific files on the device. That cool and we may be able to use that for other purposes.
I did a quick look at the ROM file in a hex-editor. The file itself does not seem to be signed at all. Maybe the individual files in the image are signed. But so far I haven't seen anything that looks like it.
So if the WP7 ROM Tools can be adapted for this image, be might be able to exploit it and deploy our own patched ROM Images to the phone, just like custom kernels are now deployed to the Samsung Galaxy S. That would be extremely cool.
I attached the extracted OS.nb to this message. Maybe someone can have a look at it. This needs more research.
Ciao,
Heathcliff74
Heathcliff74 said:
Today Samsung released a patch for their WP7 devices to resolve the update problems. WP7 version 7.0.7392.0 has been withdrawn for Samsung devices because of error 801812C1 during the update. After the release of this patch Microsoft has now resumed the release of the OS update for Samsung devices.
The update for Samsung Omnia7 can be downloaded from here.
I did a quick analysis of the file. It is an executable that should be run on a Windows PC. It installs the latest USB drivers for Samsung devices on the PC. Then a flash utility starts. A ROM-file is embedded as a resource in the flashtool. It's called OS.nb and it has a familiar Ra000FF header. So I tried to open it with the latest WP7 ROM Tools. It opens and it shows the "OS partition". But, unfortunately, as soon as I expand that node the WP7 ROM Tools crash. I hope that AnDim, the author of the WP7 ROM Tools, can work on that. The embedded ROM image is about 12 MB. It should contain only the files that are updated.
So this flash tool is able to replace some specific files on the device. That cool and we may be able to use that for other purposes.
I did a quick look at the ROM file in a hex-editor. The file itself does not seem to be signed at all. Maybe the individual files in the image are signed. But so far I haven't seen anything that looks like it.
So if the WP7 ROM Tools can be adapted for this image, be might be able to exploit it and deploy our own patched ROM Images to the phone, just like custom kernels are now deployed to the Samsung Galaxy S. That would be extremely cool.
I attached the extracted OS.nb to this message. Maybe someone can have a look at it. This needs more research.
Ciao,
Heathcliff74
Click to expand...
Click to collapse
I'm highly curious to see what is inside of this file-as you said, it's likely the individual files that need to be updated to the phone. In this case, this could be huge for bringing custom ROMs to Samsung devices. I would experiment some more with this, but I only have one Samsung phone and it's my main, and something on a level this low could easily prove hazardous.
Heathcliff74 said:
I hope that AnDim, the author of the WP7 ROM Tools, can work on that. The embedded ROM image is about 12 MB. It should contain only the files that are updated.
Heathcliff74
Click to expand...
Click to collapse
There are two WP7 partitions in it: SLDR1, SLDR2.
To use with htcrie - cut with hexeditor from 0x8000 and replace in any Full Samsung WP7 rom.
AnDim
AnDim said:
There are two WP7 partitions in it: SLDR1, SLDR2.
To use with htcrie - cut with hexeditor from 0x8000 and replace in any Full Samsung WP7 rom.
AnDim
Click to expand...
Click to collapse
Hi AnDim
Thank you very much for your help. But I can't get this to work. I must have understood your instructions. I used a copy of the stock ROM "I8700XENJK1". I took the file "signed_pda_Cetus_Europe.nb0". It's about 209 MB. I did two attempts:
First attempt:
I took the new file "OS.nb". Copied bytes 0x8000 - C9493F. Opened the file "signed_pda_Cetus_Europe.nb0" and pasted the bytes at position 0x8000, overwriting the existing bytes and writing the file. The resulting file is the same size of 209 MB. Opening this file with htcRIE.exe crashes.
Second attempt:
I took the first 0x8000 bytes from "signed_pda_Cetus_Europe.nb0" and copied them over the first 0x8000 bytes from the new "OS.nb". The resulting file is the same size as the original "OS.nb", around 12 MB. Opening the OS node results in a crash.
Could you please tell me what I'm doing wrong? Thank you!
Heathcliff74
Hi folks,
FIRST RELEASE:
Wow, lots of bugs that needed fixing. There are still some things I'd like to improve, but they can wait. This is officially the first release of XapHandler, for stock ROMs with WP7 Root Tools, and optionally for custom ROMs as well.
INSTRUCTIONS:
1) Install XapHandler.
2) Run WP7 Root Tools.
3) Pivot to Policies, and set XapHandler as trusted (stock ROMs only).
4) Run XapHandler.
5) When prompted to set the file association, hit OK.
6) Exit XapHandler.
7) Open a XAP file from email, IE, or an app.
8) Check the data about the app given in XapHandler.
9) If the app is already installed and you want to do a clean install, exit XapHanderl and delete the app from your phone normally, then open the XAP file again.
10) Hit the Clean Install or Update button.
11) Wait for the Success message box.
12) Close XapHandler.
SOURCE CODE:
I don't have permission to release the actual XAP installation source code, which is from @ultrashot. Therefore, the source release just contains a pre-compiled static library (it's the same code as is used in the native XAP installer on custom ROMs, and the Bazaar app). You're able to build the full app, though.
The app has two parts: the native ARM DLL that has the COM library and does the actual installation, and the C#/Silverlight WP7 app that provides the UI and calls the COM library. You should unpack both parts to the same folder. Then, build the native part first (requires Visual Studio 2008 with the Windows Mobile 6.5 compiler). This will copy the native DLL into the managed app. Make sure the managed app includes the COM DLL, and then build it to produce a XAP file.
PROBLEMS:
If you have any issues, please report the following info:
A) Your phone model.
B) Your ROM (including version and firmware)
C) The problems or exception messageboxes you encountered.
RESTORING THE REGISTRY:
For stock ROMs, if you want to put the registry back the way it was so XAP files open in the built-in XAP handler of your custom ROM:
1) Open a registry editor
2) Go to HKEY_CLASSES_ROOT (HKCR)\.xap
3) Change the default value back to what it used to be
3.1) The original value on most custom ROMs should be "xapfile" which is the same value the app uses; don't panic if it's unchanged!
3.2) The old default value *should* be backed up in a registry value simply called "0"
4) If there's a value called 0, delete it
5) Go to HKCR\<originalClassName>
5.1) For most people, that's HKCR\xapfile
6) Change the default value back to what it used to be
6.1) The original value on most custom ROMs should be empty
6.2) This step is not important; the value is cosmetic and intended for the user
6.3) The old default value *should* be backed up in a registry value simply called "0"
7) If there's a value called 0, delete it
8) If you have a registry key called BrowseInPlace (eg. HKCR\xapfile\BrowseInPlace) you can delete it
8.1) I don't *think* it'll hurt anything if left, though
9) Set the registry value EditFlags to DWORD 65536 (0x00010000)
9.1) This value is set by release versions of XapHandler
9.2) If the value is missing, create it with the data listed above
10) Go into the registry keys "shell", "open", and "command"
10.1) For example, HKCR\xapfile\shell\open\command
11) Change the default value back to what it used to be
11.1) The default value on most custom ROMs should be "XAPDeployer.exe "%1"" (remove first and last quotes)
11.3) The old default value *should* be backed up in a registry value simply called "0"
12) If there's a value called 0, delete it
13) If you had to change anything in step 3, go to HKCR
13.1) Don't do this step if you didn't have to change anything in step 3!
14) Delete HKCR\xapfile and all its contents
14.1) This step isn't needed, just cleanup
14.2) Don't do this step unless you had a different original progID than "xapfile"!
Have fun!
Please feel free to include this app in WP7 homebrew collections or custom ROMs, so long as the Author attribution is not removed.
Total downloads of the test versions: 330.
Thank you my friend
big business
Big Thanks to useful reports, of course!
I'm also looking at releasing an app that allows people to edit their file associations on the phone - things like add new filetypes that will open in Word (works for any plain-text file), or do complex things like "open" a Mobi file by putting it in the Kindle app's isostore, then launching the Kindle app. That will build on some of the work done in this app. It's also related to some stuff for the next version of Root Webserver. I really want to figure out what's going on here, though...
a.) Samsung Focus v1.3
b.) My ROM (see sig) firmware 2124.11.9.4
c.) xapfile
d.) no change, errors: system.io.filenotfoundexception the system can not find the specified at xaphandler.xapdeployerinterop.setfileassociation() and so on and on then crashes to home screen
e.) reboot changes nothing, no other steps apply.
I installed with my hd2 program
program did not work
19.Marc HD2 Pdaimatejam Rom Wp7.5 Tango 7.10.8773.98 v7.8 >FULLY UNLOCKED>CABS Update
Mozart
5.13-based custom rom (fully unlocked of course)
FileNotFound exception when trying to associate, IndexOutOfRangeException when trying to launch xap (after manual association)
ultrashot said:
Mozart
5.13-based custom rom (fully unlocked of course)
FileNotFound exception when trying to associate, IndexOutOfRangeException when trying to launch xap (after manual association)
Click to expand...
Click to collapse
yes
I get the same error
Thanks, guys. I know where the error is for you - I took a shortcut testing for the existence of certain registry entries and didn't check for FileNotFound, which I should have. Expect a new version in a day or so!
do you mean use xapdeploy.exe to install a xap?
XapDeploy.exe can't be used on stock ROMs until we break the policy system to allow homebrew executables. This app runs under TskHost.exe, like any other executable. Using WP7 Root Tools v0.9, it should be possible to run this app on stock ROMs.
Additionally, it's an alternative to XapDeploy.exe for users of custom ROMs. It displays more info about the XAP before installing it - not enough in any way to prevent Trojan malware, but enough to stop you from accidentally installing v1.15 when you wanted 1.16.
Please do post test results for the second test version. I still can't get it working on my phone, but I want to know if it works on a full-unlock - if not, there's a bug in the app, but if so then the issue is in the way I elevate. Please give it a try!
gave the same error
version 2
GoodDayToDie said:
Please do post test results for the second test version. I still can't get it working on my phone, but I want to know if it works on a full-unlock - if not, there's a bug in the app, but if so then the issue is in the way I elevate. Please give it a try!
Click to expand...
Click to collapse
OOM exception while checking association, on minimizing-resuming it asks for associating (and does it correctly), but then SUCCESS on installation
except update
application tries to install
thanks
Now *that* is bloody weird; even without elevation the app gets further than that. This happens when you open a XAP file after associating the app with XAP files, right? (Was it really too much to ask that people actually provide a proper report?) There's only two places that the app uses GetResourceStream - one to open the XAP file, and one to open the manifest from within the XAP file.
I've made some changes to the app, which should make it more resilient to unexpected parameters and will hopefully help make the GetResourceStream calls less likely to fail. Please remove any manual file association you created for the app (set it back to the defaults) and run the test procedure again with the new test3 build (will be uploaded momentarily). The app was able to associate itself correctly on my phone starting with test2.
I'd also like to see this tried on some different custom ROMs. It *should* work but I still can't get the actual install to succeed on my phone (though it does get that far...).
@ultrashot: Wow, the OOM is weird and I haven't seen that in my testing, but hey, if it worked the second time... OK then! Thanks for the success report! Time to look into what level of unlock is required and figure out how to get it on a stock ROM, because elevating the app doesn't seem to be enough...
Hi GoodDayToDie, I am using HTC Gold with Dyanamic7 Rom. Firmware 51201.401. Have installed the XapHandlerTest2.xap and I am able to install Xap files which are stored in the device. Please let me know if you require any more details.
Can you do updates? I'm *sometimes* able to install an app that is not yet installed, but if the app is installed, either an update or a clean install always fails. Clean install seems more reliable immediately after rebooting the phone, but it still appears non-deterministic to me.
I hate Schroedenbugs... although HtcRoot seems to have a lot of them. I need to work on that some more.
Still, it's progress. Hopefully, by the time WP7 Root Tools 0.9 is out and the app can be used on more stock ROMs, it'll be fully working...
v3 test
except update
application tries to install
thanks
Where Internet explorer save the XAP? (I have a custom ROM (Omnia7))
Not sure if I have anything worth while, but ASUS left a "Tools" folder on my desktop...attached a screenshot of what it included.
phxtravis said:
Not sure if I have anything worth while, but ASUS left a "Tools" folder on my desktop...attached a screenshot of what it included.
Click to expand...
Click to collapse
Can you please zip that and upload them?
Yeah please zip it up and upload them so we can see what it's doing.
Yes, ZIP them please - the auditmode files are for sysprep audit mode, securebootdebug - are probably from microsoft HCK and add debug key to key storage (so you can run testsigned apps), FWVar - probably allows editing UEFI firmware variables (I've already made the same tool myself), everything is interesting of cause.
Here you go
Thank you for the tools.
SetAuditMode/ClearAuditMode - clears the sysprep audit mode (useless)
setup.cmd, SecureBootDebug* - installs "secure boot debug" policy. I.e. allows running of testsigned (or unsigned) apps. More info: http://msdn.microsoft.com/en-us/library/windows/hardware/hh998740.aspx
Securebootdebug needs the signed policy file. It is probably left on your device too, as "tools" directory is not erased. To obtain it - run CMD or powershell as administrator, then type there: "mountvol S: /s" without quotes. This would assign S: to your BCD partition. There should be SecureBootDebugPolicy.p7b file. Please share it too
To dismount disk S: after copying that file - type "mountvol s: /d"
reset.cmd - this file would delete the tools directory and all other files reverting things back.
hsc.vbs, pdq.vbs - tiny support scripts for reset.cmd
FWVar.exe - writes firmware variables. Not UEFI vars that are documented, but it plays with some other asus-specific vars like sensors calibration and platform IDs. Would be interesting to decompile it
mamaich said:
Thank you for the tools.
SetAuditMode/ClearAuditMode - clears the sysprep audit mode (useless)
setup.cmd, SecureBootDebug* - installs "secure boot debug" policy. I.e. allows running of testsigned (or unsigned) apps. More info: http://msdn.microsoft.com/en-us/library/windows/hardware/hh998740.aspx
Securebootdebug needs the signed policy file. It is probably left on your device too, as "tools" directory is not erased. To obtain it - run CMD or powershell as administrator, then type there: "mountvol S: /s" without quotes. This would assign S: to your BCD partition. There should be SecureBootDebugPolicy.p7b file. Please share it too
To dismount disk S: after copying that file - type "mountvol s: /d"
reset.cmd - this file would delete the tools directory and all other files reverting things back.
hsc.vbs, pdq.vbs - tiny support scripts for reset.cmd
FWVar.exe - writes firmware variables. Not UEFI vars that are documented, but it plays with some other asus-specific vars like sensors calibration and platform IDs. Would be interesting to decompile it
Click to expand...
Click to collapse
I was unsuccessful at finding the file, I wiped the tablet yesterday as it was running like crap(freezing, and wouldn't download updates), not sure if that wiped what you are looking for.
OK, I see what that setup.cmd file is doing. It generates an unsigned SecureBootDebug.p7b file authorizing full debug unlocking for the particular serial number of your device - the -u switch to createsecurebootpolicy.exe locks it to your device. It then sends that file to a server aptly named "\\secureboot" on ASUS's internal network and waits for a reply. Sometime later, a signed version of that file appears on that server's share, and the Vivo RT copies it to the EFI system partition to use with SecureBootDebug.efi.
After the service center does what it needs to do, they run reset.cmd, which deletes the Secure Boot policy from EFI NVRAM at next reboot and deletes SecureBootDebugPolicy.p7b from the EFI system partition.
phxtravis said:
I was unsuccessful at finding the file, I wiped the tablet yesterday as it was running like crap(freezing, and wouldn't download updates), not sure if that wiped what you are looking for.
Click to expand...
Click to collapse
If you had not done that, we could have made an image of your EFI system partition and recovered the deleted SecureBootDebugPolicy.p7b file, which would have permanently jailbroken your Vivo RT forever. Sadly, it was locked to your device. What method did you use to wipe it? Did you ask it to repartition your system?
Myriachan said:
OK, I see what that setup.cmd file is doing. It generates an unsigned SecureBootDebug.p7b file authorizing full debug unlocking for the particular serial number of your device - the -u switch to createsecurebootpolicy.exe locks it to your device. It then sends that file to a server aptly named "\\secureboot" on ASUS's internal network and waits for a reply. Sometime later, a signed version of that file appears on that server's share, and the Vivo RT copies it to the EFI system partition to use with SecureBootDebug.efi.
After the service center does what it needs to do, they run reset.cmd, which deletes the Secure Boot policy from EFI NVRAM at next reboot and deletes SecureBootDebugPolicy.p7b from the EFI system partition.
If you had not done that, we could have made an image of your EFI system partition and recovered the deleted SecureBootDebugPolicy.p7b file, which would have permanently jailbroken your Vivo RT forever. Sadly, it was locked to your device. What method did you use to wipe it? Did you ask it to repartition your system?
Click to expand...
Click to collapse
I did the "remove everything and reinstall windows" option in settings.
phxtravis said:
I did the "remove everything and reinstall windows" option in settings.
Click to expand...
Click to collapse
It *might* still be there then if we take an image of your EFI System Partition and search it manually with a hex editor. There are definitely no guarantees, though. Also, this would likely reveal your device's serial number to whoever you give the image to. It probably would *not* have any other information, though, like personal documents or your Windows RT CD key--those're on the main partition.
If this file were found, I think it would permanently jailbreak your device - Windows RT would let you run whatever unsigned code.
The SecureBootDebug.efi tool needed to use this .p7b file is also part of the publicly-available final 8.1 Windows Driver Kit. The one in your .zip file looks like the 8.0 version.
Myriachan said:
It *might* still be there then if we take an image of your EFI System Partition and search it manually with a hex editor. There are definitely no guarantees, though. Also, this would likely reveal your device's serial number to whoever you give the image to. It probably would *not* have any other information, though, like personal documents or your Windows RT CD key--those're on the main partition.
If this file were found, I think it would permanently jailbreak your device - Windows RT would let you run whatever unsigned code.
The SecureBootDebug.efi tool needed to use this .p7b file is also part of the publicly-available final 8.1 Windows Driver Kit. The one in your .zip file looks like the 8.0 version.
Click to expand...
Click to collapse
Not sure if I am sold on jailbreaking, what's the advantages other than being able to run "hacked" exe's? Aslo, what would you need me to do?
The EXEs are not "hacked" in any proper sense of the word, just recompiled. Sometimes some changes are needed, but they're usually basic. It lets you run (normal) Windows programs. .NET programs run as-is, native ones need to be ported (usually a pretty simple recompile, if they built under Visual Studio in the first place, but we need the source code), and it unlocks full Powershell scripting power. Programs written in other languages, like Python and Java, can be run using ported runtimes. In fact, it's even possible to run some x86 programs (unmodified Win32 native EXEs) via a dynamic recompilation layer written by @mamaich here on XDA; I can play some old games and run some nice old programs that I like that way.
phxtravis said:
Not sure if I am sold on jailbreaking, what's the advantages other than being able to run "hacked" exe's? Aslo, what would you need me to do?
Click to expand...
Click to collapse
Hacked EXEs?
No hacking. We legally take the source code for a program and compile it for win32 in THUMB_2 instead of win32 on x86. You can then run these desktop applications on your lovely ARM tablet as you could on a normal windows PC. That is an absolutely huge advantage which should have been a default option.
Quake alone makes it worth it.
I didn't mean to use "hacked" in a negative context, merely meant it as a general term of modifying original EXEs to run on ARM, I haven't been following the RT jailbreaking at all.
Can't modify an existing EXE. Totally incompatible (unless its a .NET application in which case no mods needed). Need to rebuild the EXE and any supporting libraries from source.
phxtravis said:
Here you go:
https://docs.google.com/file/d/0BzebTu1H3-aIbXlTV09BMjZsLVk/edit?usp=sharing
Click to expand...
Click to collapse
This requires approval. Does anyone still have a copy?
jordanmills said:
This requires approval. Does anyone still have a copy?
Click to expand...
Click to collapse
It doesn't require approval
Curiously, there is a Secure Boot debug policy creator ,signed by Microsoft for ARM but sadly not generating signed policies...
It may be used for jailbreak as the other tools are unsigned(most of them) if there is a bug allowing to load a unsigned policy somewhere(there should be one)
Will try downloading the HCK to see if there is something useful there
black_blob said:
It doesn't require approval
Click to expand...
Click to collapse
Hmm, it doesn't now. But there doesn't seem to be any way to download the whole thing. It only shows individual files.
jordanmills said:
Hmm, it doesn't now. But there doesn't seem to be any way to download the whole thing. It only shows individual files.
Click to expand...
Click to collapse
There is the download button at the top
Current Version: Extraction Tool v5.92, Analytics Tool v5.60, iOS Extraction Tool v0.2, Fit Sync Tool v0.2 (Updated 6/20/2015)
Here is a tool I wrote for Xiaomi Mi band which provides following functionality:
1. Exports data from Xiaomi Mi band on both rooted and non-rooted Android phones and jailbroken (and possibly non-jailbroken) iPhones under Windows and Linux, generating file in CSV format which can be reviewed in Excel.
Bonoboo maintains a script to perform extraction directly on your smartphone (see link in post 3).
2. Extracts detailed (per minute) walking and running information from Mi Band database (thanks to the input provided by mritsch and Osid).
3. Produces files that can be uploaded to fitnesssyncer.com/ to syncronize data with various fitness services: FitBit/HealthVault/RunKeeper/JawBone and soon Google Fit! See 3rd post for details on data mapping for fitnesssyncer.
4. Provides multiple analytics reports to review extracted data. These reports can be shared on cloud drives to be accessed from anywhere (see post 3 for details)
5. Supports localized UI (currently supported languages are English, Russian, French, Spanish and Italian)
6. Support for synchronization data with Google Fit Service - see post #2 for details
How to use:
Preparation steps:
1. If you plan to use multiple packages, unpack them all to same directory.
1a. If you want to extract data from smartphone, download miband_extract package. To extract data from iOS devices, see to "To extract data from iOS devices" step.
1b. If you want to display local charts of your extracted data on your computer, download miband_analytics package.
1c. If you want to synchronize your extracted data with Google Fit, download miband_fit package.
2. Make sure you have USB drivers for your device properly installed and that your device is accessible by ADB when you connect it through USB
3. If you use it on windows, extract package has all binaries included, for Linux/OSX see comments below.
Checking configuration settings:
1. Review SDPath parameter value in run.bat/run.sh. The program will copy files from Mi app location to folder specified in SDPath before pulling them to desktop. In most cases default value (/sdcard) shoud work fine, however if your phone does not have this directory, find the path where your Internal/External SD is mounted and put that path string into SDPath value. Second most common value might be /storage/sdcard0
2. Review config.js and make any changes to your liking (set Goals for sleep hours and daily steps, force override UI language to specific value)
3. If you do not want main report being open every time you run extract, change OpenHTML=Y in run.bat/run.sh to OpenHTML=N
4. If your device is not rooted or have any issues with first (root) method that application uses and prefer to skip straight to the second (backup) method, set ForceBackupMode value to Y in run.bat.
5. If you want to generate Raw data extract in addition to summary data, set ExtractRaw parameter to Y, update Height parameter to your height in centimeters and update Weight parameter to your weight in kilograms. Please note that data extracted from raw blobs may be slightly different from total numbers in summary data.
If you are planning to use ADB over Wifi:
0. I would not be able to provide much support for ADB over Wifi - so many things could go wrong. Check you firewall settings, check TCPIP ports on both ends, but you are basically on your own.
1. If you are planning to use ADB over WiFi, edit run.sh and set up IPAddr value to IP address of your phone, if you use USB cable, leave IPAddr value blank. If you using non-default port, you may need to change TCPPort value.
2.You may also think of a great idea of running syncronization automatically and unattended using ADB over Wifi - at least I liked that idea initially. I gave that idea more thought and as of now I strongly recommend not to do it - having ADB running over Wifi is a security risk, if you have to use it over Wifi, enable it manually, run the sync and disable ADB over Wifi right away.
For Linux/OSX users:
1. You would need to to manually install android-sdk for (adb binary), sqlite3 version 3.8.3 or higher and openssl with zlib support (version 1.0.2 or higher with zlib support is recommended) to uncompress zlib data. Please note that versions of sqlite3 and openssl that are preinstalled on your machine might be too old to be used with this package, so you might need to obtain newer versions. For example, I was told that OSX has sqlite3 version 3.7.13 preinstalled and it is incompatible with some of functions used in script, so you would need version 3.8.3 at least. If you cannot get openssl with zlib support, you can use similar functionality from python instead - you will need to comment out a line with openssl zlib in run.sh and uncomment next line instead which invokes python script.
2. You'll need to grant execute permissions to run.sh by using chmod +x run.sh and you will need to execute run.sh instead of run.bat in steps listed below. You'll also need to make configuration changes in run.sh instead of run.bat
3. Check that your sqlite3 is properly configured for your time zone. Run following command and see if it returns correct timestamp:
bin\sqlite3 dbfile "select datetime('now','localtime');"
Running application:
1. Connect phone through USB and make sure USB debugging setting is enabled on your phone. Unlock phone screen.
2. Execute run.bat - if your phone is rooted, the data would be pulled automatically. If your phone is not rooted you would see backup screen and you need to press "Back up my data" button in the bottom left corner.
3. Data from your mi band will be saved to extract.csv file and extract.js. After extraction is complete, if OpenHTML is set to Y, mi_data.html will be opened automatically to show charts for your Mi usage.
4. HTML reports are using Google Charts framework and Google TOS does not allow storing their scripts offline along with the application, therefore you will need to have working internet access for reports to work. Your data is not being sent to Google, the internet connection is only used to download latest version of Google Charts javascripts.
Troubleshooting connectivity issues:
1. If you phone is rooted:
Review that your phone has /sdcard directory and it is writable. After you run extract, check if you have origin_db file in that directory. If you do not, ADB may have issues writing files to that directory. Check configuration settings section, item 1
2. If your phone is not rooted or if you use ForceBackupMode=Y:
Check if you get full backup screen, which looks like this:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
If you do not get this screen, then Android backup functionality on your device is broken. If your phone is rooted, consider setting ForceBackupMode=N and using rooted method; if your phone is not rooted, try to run Helium Backup https://play.google.com/store/apps/details?id=com.koushikdutta.backup and see if it can backup MI app to sdcard.
To extract data from iOS devices:
Thanks to joshmosh for detailed instructions.
This is reported to work on jailbroken iPhone, but may work on regular iPhones as well - if you try it on non-jailbroken iPhone please let me know if it worked for you.
1. Get a copy of itools. Google it to find a download location (cnet might have it)
2. Plug in your iPhone to your PC
3. Open itools and click on Applications in top row
4. On the left you will find name of your iPhone and below "Applications". Click on Applications
5. Locate your application Mi Fit and click it. On the right you will find a button labelled "browse". Click it.
6. A window will open. Doubleclick on "Documents" then click on user.sqlite, then on "Export" above.
7. A window will open asking you where to export this file. Export it to a db subdirectory of miband extractor main directory and click OK. You should have a file db\user.sqlite now.
8. Once file is extracted to db\ directory, run runIOS.bat from main miband extractor directory. It should generate extract.js, extract.csv and minmaxtime.csv files and will open local charts in your browser if you have analytics package installed.
Screenshots:
Report descriptions:
mi_data.html:
This report shows overview of the data being extracted. It consists of 3 elements:
a. The chart on the left shows sleep data over time. Total bar height represents overall sleep time and colored segments represent light and deep sleep parts of it.
b. The chart on the right shows daily activity metrics over time. It shows total distance, number of steps and calories burned.
c. The chart on the bottom shows all data inside text table. You can sort the data by any column by clicking on the title of that column. If you select any specific row in the table, same day will be hightlighted in two other charts.
Both graphic charts also show target goals in green line, so you could see how well your results are matching target values. You can dynamically change target values by pressing + and - buttons. The table below has arrows next to Total Sleep and Total Steps columns which would be green for days when goal was met and red for days when goal was not achieved.
mi_data_byDoW.html:
This report allows you to see the scatter of your activities grouped by the day of the week.
This would be a good chart to see if you are performing better on any specific day of the week.
There are 4 charts on that report:
a. The left chart shows various metrics in scatter chart. You can select a specific metric from the dropdown located above the chart. You can also see a daily goal trendline in green and an approximation line in black. If the approximation line is not flat and you have statistics over many weeks, this might mean that your performance is consistently different depending on the day of the week.
b. The right chart shows line chart data for the metric selected in the left chart. It has time zoom controls at the top to only show the data over last day, last 5 days, 1 week, 1 month, 3 months, 6 months and 1 year. You can also select time interval manually by dragging time sliders at the bottom of the chart.
c. The chart at the bottom is a calendar view for your activity-related metrics. It shows a calendar and each days recorded activity is colored based on whether the value for that activity is above or below the goal. The days that have metric value below daily goal will be colored red, the days with metric value above the goal will be colored blue. The higher the deviation from the goal, the more intense color would be. In other words, the days where results are significantly above the goal would be colored in saturated blue and the days where results are significantly below the goal would be colored in saturated red.
This chart could help searching for more complicated patterns in your performance, patterns that are not just days of the week.
You can also change goal values dynamically by using + and - buttons at the top and see green trendline moving on the left chart and days changing colors on the bottom chart.
d. The last chart shows a timeline of your sleep on a daily level. This chart will only be visible if you select "Sleep Timeline" metric from the dropbox. This chart shows the time you went to sleep and the time you woke up. It shows this information for each day the history data is available and weekends are colored in a different color. This chart could help you to see how consistent your sleep/wake hours across different days and also if you have same sleeping patterns on weekends.
Google Fit Syncronization:
Limitations:
1. Daily summary statistics is synced, so charts by Week and Month will show correct data; chart by day will show correct totals, but not correct hourly breakdown.
Prerequisites:
1. You need to have at least version 5.71 of miband_extract package.
2. You will need to have Internet access and your browser should be configured to enable pop-ups at least from a specific domain below.
3. You need to open following URL: Get Fit Token and allow it to access your Google Fit data. Latest version of Chrome worked for me, while IE 11 had issues with popups.
Installation:
1. Download latest version of meband_fit package and unpack it to the directory where you have miband_extract v5.71 or above.
2. If you upgrade from previous version of fit sync package, you will need to clean up old data populated by old version. To do so run following command:
fit clean
Running Sync:
1. Synchronize your data from smartphone to desktop.
2. Obtain a new OATH key for syncing data with Fit - open Get Fit Token and copy resulting string to your clipboard.
3. Run fit.bat and when asked for new token paste the string from previous step and press enter. If fit.bat did not ask you for token and failed with Authentication error, just run it again.
Troubleshooting:
1. The process is logging into fit\*.log files. If you have errors, review these logs first.
2. The token from step 2 is valid for 1 hour - you will need to refresh the token after that. After 1 hour subsequent fit syncs will start failing until you obtain a new token.
3. Current token is cached in GKey environment variable. Once batch determines that old token is expired, it would clean the variable and will ask you for a new key on a subsequent run.
Related links:
Latest version of Bonoboo port for smartphone extraction.
Known Issues:
1. Vertical axis is not always in sync for two charts in mi_data_byDoW.html
2. Timeline axis is only shown at the bottom of timeline
Version History:
Code:
[B]v5.92 06/20/2015[/B]
Added support for Mi Scale - extracting weight history into weight.csv
[B]v5.90 03/14/2015[/B]
Walking and Running statistics with minute granularity are now extracted from Raw data thanks to the input provided by mritsch and Osid.
[B]v5.80 03/07/2015[/B]
Google Fit sync package 0.3:
Fixed error in 1324438.json file which prevented Fit Sync for some users.
Added FitTools.bat which allows pulling various information from Google Fit account.
[B]v5.80 03/01/2015[/B]
Extraction package:
Added experimental support for raw data extraction. To enable raw data extraction, set ExtractRaw=Y in run.bat
Analytics package:
German localization added to analytics package (thanks to joshmosh)
default app_locale.js file is added to analytics package to support data extracted from iOS devices.
Data Extraction Tool v0.1 for iOS is added along with instructions on how to manually extract Mi Band data from iPhone
[B]v5.71 02/16/2015[/B]
Extraction package:
Extra fields added to extract.csv to support for Google Fit sync v0.2.
extract.json is not created anymore, this data is now calculated on the fly from extract.csv
Google Fit sync package 0.2:
Added sync for activity durations (walk duration, run duration, sleep duration)
Added sync for distance walked. Since distance metric is considered location, Token application will request new permission:
https://www.googleapis.com/auth/fitness.location.write
Fixed activity timestamps - Google Fit expects time to be in UTC time zone.
[B]v5.70 02/16/2015[/B]
Extraction package:
Initial support for Google Fit sync.
Google Fit package 0.1:
Initial release
[B]v5.50 02/05/2015[/B]
Extraction package:
Added new columns to CSV file for better FitnessSyncer compatibility.
Added support for time correction. See db\offset.sql for sample entries.
Updated run.sh from [URL="http://forum.xda-developers.com/member.php?u=6091617"]jlaunay[/URL] to support new run.bat parameters and localization.
Analytics package:
Changed mi_data_byDoW.html to use AnnotationChart for right graph - support for time interval zooming.
Added new metric to mi_data_byDoW.html called "SleepTimeline" to show Sleep Timeline chart.
Various small fixes.
[B]v5.35 02/03/2015[/B]
Extraction package:
Fixed logging issue - adb would sometimes lock the log files and subsequent extraction invocations would fail completely on locked log files.
Added support for ADB over Wifi - see instructions in the first post.
Analytics package:
Fixed sorting order on main table
Second fix for incorrect sleep/wake hours calculation
Updated colors across all reports to be consistent
Converted total/deep/light sleep from minutes to hours and minutes in the main table. Show percentages of deep/light sleep compared to total sleep.
Add fall asleep and wake up times as selecrable metrics to mi_data_byDoW.html
mi_data_byDoW.html absorbed all functionality from mi_data_Sleep.html - the sleep calendar functionality and sleep timeline is available when you select Total Sleep metric. You will need to scroll down to see sleep timeline chart.
[B]v5.3 02/02/2015[/B]
Extraction package:
Add new column with number of walk time + run time in seconds (as per request)
Added new parameter ForceBackupMode to skip attempt at rooted extraction and go straight for backup mode
[B]v5.2 02/1/2015[/B]
Splitted program into data extract package and analytics packages. Data extract should be pretty stable now and reports could be updated independently.
Spanish translation (thanks to Viriatox)
[B]v5.1 01/31/2015 [/B]
Fix for incorrect sleep/wake hours calculation
[B]v5.0 01/31/2015[/B]
Localization setting is pulled from Mi database - no need to set it manually anymore (in most cases)
Two completely new reports - Sleep Report and Statistics by day of the week.
Fixes to unix shell script encoding.
[B]v4.0 01/29/2015[/B]
Month number calculation fixed.
Removed extra rows with zeroes that we incorrectly extracted
Localization moved to separate file, all other components are unified for all languages
Scripts for rooted and non-rooted phones combined into one.
Daily goals and data grouping added on main report.
Unix shell script for data extraction (thanks to jlaunay)
French translation (thanks to jlaunay)
[B]v3.0 01/26/2015[/B]
New script to support data extraction for non-rooted phones
Reverse sort order in table part of report, so newest dates are on top.
Week number and day of the week added to report.
[B]v2.0 01/25/2015 [/B]
All needed Windows binaries included in the package.
Batch file to automatically run all binaries to extract the data.
First report showing overall statistics.
[B]v1 01/24/2015[/B]
Initial release.
Syncronizing with fitnesssyncer.com
FitnessSyncer FAQ: https://www.fitnesssyncer.com/frequently-asked-questions
How to set up import and export.
1. Open account at fitnesssyncer.com/
2. On Dashboard, go to Sources and Destinations and select Add New Source
3. Select Type: Activity, source Amazon S3 (CSB), Dropbox (CSV) or OneDrive (CSV) depending on where your files are stored. Enter path and file name in Path field. Select highlighted columns mappings:
Activity:
Include Date (US) - Column 18
Include Activity - Column 19
Include Distance in Meters - Column 8
Include Duration in Seconds - Column 17
Include Calories Burned - Column 10
Include Total Steps - Column 9
4. Click on Save and authenticate into your cloud storage account.
5. Back on Sources and Destinations account, click on Sync Now link next to your newly added connection to confirm that it loads without errors. Click on Dashboard button and confirm that you see some data in charts there.
6. Go back to Sources and Destinations and select Add New Source again
7. This time, select Type: Sleep, same Source entry, same Path value and set up highlighted column mappings:
Sleep:
Include Date (US) - Column 18
Include Bed Time Hour - Column 20
Include Bed Time minute - Column 21
Include Awake Hour - Column 22
Include Awake Minute - Column 23
Include Sleeping Minutes - Column 2
8. Click Save, go to Sources and Destinations and click Sync Now on Sleep source now to see that it imports without issues.
9. Click on Stream option and see that you have both Daily Activity and Sleep data loaded.
10. Go back to Sources and Destinations and select Add New Destination Task
11. Select SyncType Activity and select Destination from the dropdown list
12. Repeat same steps for Sleep syncronization
Accessing your charts directly from Dropbox:
If you have a Public folder on your dropbox, you can copy all the .js and .html files there and get the shared url by selecting one of htmk files and selecting "Copy public link..." button. It would give you a URL that you can use from anywhere. You need to have Dropbox Pro account or free account created before October 4, 2012 for Public folder functionality to work on your dropbox.
Accessing your charts directly from Google Drive:
1. It is recommended that you create new folder in your Drive for this purpose.
2. Select the folder and Choose Share... option
3. On Share with others page dialog, click on Advanced button in bottom right corner.
4. On Sharing Settings dialog, click on Change... link at the top line of Who has access list.
5. On Link sharing dialog, select On - Public on the web option and click Save button.
6. Copy the URL that is shown on Sharing Settings page and click Done to close it
7. The URL that you copied will look something like the sample below. Copy just the alphanumeric sequence that is located between ?id= and &usp=sharing
Code:
https://drive.google.com/folderview?id=[B][COLOR="Red"]123-abcdcdefghijklmnopqrstUV[/COLOR][/B]&usp=sharing
8. Put www.googledrive.com/host/ before this string and /mi_data.html (or any other report) after it, you should get something like www.googledrive.com/host/123-abcdcdefghijklmnopqrstUV/mi_data.html - this should be your sharing link
Here is Google support page on sharing files/directories: https://support.google.com/drive/answer/2881970?hl=en
Using Helium backup instead of Android backup:
Helium produces .ab files that are ALMOST correct tar file, however besides extra characters on file header level they also have some differences in header entries for each file. I've looked through different methods and utilities and the best option so far is to use Helium_ab2tar.zip package (credit goes to xaos.cz for writing it and to johnny1176 for discovering it). It used to be available on xaos.ic.cz/Helium_ab2tar.zip, but that site is down now. You can still google for Helium_ab2tar.zip to find other locations that have copy of this file.
Plans for future versions:
Code:
0. Get more sleep :)
1. Titles for each chart
2. Implement dashboard comparing current week with previous statistical data. (For same days of week)
3. Use timeline component for right chart in data_byDow.htm
4. Show percentage of good days based on selected goals. Show min/max values for each metric.
5. Modify scatter chart to allow using any other series for X axis (Search/display correlations between sleep time and total number of steps).
6. Make a help page showing all the charts and how to use them.
7. Add ratio of deep/light sleep as part of selectable charts.
8. Merge all reports into single multi-tab document.
9. Parse detailed daily binary data - this is going to be a big one.
How to translate application to your language:
To add support for a new language:
For HTML pages
1. Open locale.js in editor supporting UTF-8 encoding.
2. Create a new section starting with
if (lang == "ru_RU") { }
and replace ru_RU with a locale string for your language.
3. Copy all strings from english version to inside { } brackets and translate the text.
4. Save file in UTF-8 encoding.
5. Add reference to your new language in comments section of config.js
For unix shell script:
For all subsequent entries, replace XX with 2-character locale name of your language
1. Create new po file using following command:
bash --dump-po-strings run.sh > po/XX.po
2. Edit newly generated file and translate msgstr entries using msgid entries as references
3. Save the file and generate new directory for the compiled langage file (.mo) using following command:
mkdir -p i18n/XX/LC_MESSAGES/
4. Compile .po file to .mo file by running following command:
msgfmt -o i18n/XX/LC_MESSAGES/mibandextract.mo po/XX.po
If you do not want to use the i18n folder and want the langage file to be system wide, you just have to comment (adding a #) the line "export TEXTDOMAINDIR=./i18n/" in the script.
Then you will have to move your po file to /usr/share/locale/XX/LC_MESSAGES/ (where XX is your langage).
Changelog:
v5.2 released on 2/1/2015.
v5.3 released on 2/2/2015.
v5.32 released on 2/3/2015
v5.7 released on 2/16/2015
v5.71 + fit v0.2 released on 2/21/2015
v5.8 + fit v0.3 released on 3/7/2015
v5.9 released on 3/14/2015
v5.92 released on 6/20/2015
Reserved
Hi.
Working perfect. Nexus 5 non-rooted.
Thank you very much for your work :good:
Grretings from Spain.
Hi,
It does not work for me at all ... All the time the same thing.
C:\Users\plmaciejewk\Desktop\miband_extract_v53>run
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Press "Backup My Data" button on device...
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
bin\tail: mi.ab: No such file or directory
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Could Not Find C:\Users\plmaciejewk\Desktop\miband_extract_v53\mi.ab
Could Not Find C:\Users\plmaciejewk\Desktop\miband_extract_v53\mi.tar
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Extraction failed
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Can I do anything with this?
I face the same issue as @macia10. I have rooted Nexus 5.
Dell112 said:
I face the same issue as @macia10. I have rooted Nexus 5.
Click to expand...
Click to collapse
I found a temp workaround. Edit rub.bat and remove all >> log from end of the lined. This causes the issue.
macia10 said:
Hi,
It does not work for me at all ... All the time the same thing.
C:\Users\plmaciejewk\Desktop\miband_extract_v53>run
The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
Can I do anything with this?
Click to expand...
Click to collapse
This might be caused by adb process still running in memory and locking log file. I've modified script not to redirect output from adb commands into log file and also added few commands to kill previous adb service once extraction is complete. Could you please download v5.33 and see if it works?
xmxm said:
This might be caused by adb process still running in memory and locking log file. I've modified script not to redirect output from adb commands into log file and also added few commands to kill previous adb service once extraction is complete. Could you please download v5.33 and see if it works?
Click to expand...
Click to collapse
works perfectly
Thanx!!
Thanks, very useful!!
FYI, I needed to make some modifications to make it work in my case:
- In 'run.sh', I had to modify the SDPath variable to the correct path (in my version of Cyanogenmod):
SDPath=/storage/sdcard0
- run.sh log indicated a "permission denied" error. It was caused because adb did not have root privileges in my android. To correct it:
In Android: Settings > Administrative access > Select "Applications and ADB"
- sqlite3 in OSX is an old version, so run.sh complains with:
Error: unknown command or invalid arguments: "once". Enter ".help" for help
Error: unknown command or invalid arguments: "print". Enter ".help" for help
To solve it, install a new version of sqlite3. For example using brew:
> brew install sqlite3
and replace sqlite3 in run.sh for the brew version: /usr/local/Cellar/sqlite/3.8.6/bin/sqlite3
- Files from: miband_analytics need to be in the same folder than miband_extract
With these modifications it is working properly in my mac.
Keep up the good work!
Hi.
I'm extracting data with Bonoboo Mi Band Export Stats, which exports to sd card the files:
app_locale.js; config.js; extract.js; locale.js; Mi_Band_Table.csv; Mi_Band_Table.html and Mi_Band_Graphs.html.
In order to used that exported data with your analytics tools, which files should i move to your folder which contains multiple files like mi_data_byDoW.html and mi_data.html ?
Thanks
herpez said:
Hi.
I'm extracting data with Bonoboo Mi Band Export Stats, which exports to sd card the files:
app_locale.js; config.js; extract.js; locale.js; Mi_Band_Table.csv; Mi_Band_Table.html and Mi_Band_Graphs.html.
In order to used that exported data with your analytics tools, which files should i move to your folder which contains multiple files like mi_data_byDoW.html and mi_data.html ?
Thanks
Click to expand...
Click to collapse
You'll just need at app_locale.js and extract.js. The folder with analytic tools already has config.js and locale.js and CSV file is not required for analytic tools, you only need it if you want to see your data in Excel.
SynerG said:
Thanks, very useful!!
FYI, I needed to make some modifications to make it work in my case:
- In 'run.sh', I had to modify the SDPath variable to the correct path (in my version of Cyanogenmod):
SDPath=/storage/sdcard0
- run.sh log indicated a "permission denied" error. It was caused because adb did not have root privileges in my android. To correct it:
In Android: Settings > Administrative access > Select "Applications and ADB"
- sqlite3 in OSX is an old version, so run.sh complains with:
Error: unknown command or invalid arguments: "once". Enter ".help" for help
Error: unknown command or invalid arguments: "print". Enter ".help" for help
To solve it, install a new version of sqlite3. For example using brew:
> brew install sqlite3
and replace sqlite3 in run.sh for the brew version: /usr/local/Cellar/sqlite/3.8.6/bin/sqlite3
- Files from: miband_analytics need to be in the same folder than miband_extract
With these modifications it is working properly in my mac.
Keep up the good work!
Click to expand...
Click to collapse
Thank you for your feedback.
Regarding SDPath, I'll put a a more clear explanation that SDPath may need to be changed depending on device.
For ADB privileges, the way you had to enable it might be specific to your Android version (I've granted permissions in a different way on my phone), but generally speaking the application should work through Backup logic even when adb has no root permissions.
For sqlite3 issue, could you please tell me which version of sqlite3 OSX has by default, so can add a note in instructions about version compatibility?
Regarding same folder location, I think there is already a mention about the need to have both packages exptracted to the same folder.
xmxm said:
Thank you for your feedback.
For ADB privileges, the way you had to enable it might be specific to your Android version (I've granted permissions in a different way on my phone), but generally speaking the application should work through Backup logic even when adb has no root permissions.
Click to expand...
Click to collapse
I got a message in my Android asking for permission to do the backup, but at the same time I saw the "permission denied" message in the console. Maybe I got both because the script tries to get data from two different mechanisms. Even if I clicked yes to do the backup, the script was not able to get the data until adb was granted permissions.
For sqlite3 issue, could you please tell me which version of sqlite3 OSX has by default, so can add a note in instructions about version compatibility?
Click to expand...
Click to collapse
My Mac with OSX 10.9 (Mavericks) had installed sqlite3 version 3.7.13. I think that the required commands were added in 3.7.15, but I could be wrong. Brew has installed 3.8.6 and is working properly.
SynerG said:
I got a message in my Android asking for permission to do the backup, but at the same time I saw the "permission denied" message in the console. Maybe I got both because the script tries to get data from two different mechanisms. Even if I clicked yes to do the backup, the script was not able to get the data until adb was granted permissions.
Click to expand...
Click to collapse
Windows version supports ForceBackupMode=Y setting to skip root-specific method and go straight for backup method, but I guess that run.sh is outdated by now - I'll need to find a way to copy newest logic changes to unix script. I still think hope that giving ADB root is not mandatory for extract to work.
My Mac with OSX 10.9 (Mavericks) had installed sqlite3 version 3.7.13. I think that the required commands were added in 3.7.15, but I could be wrong. Brew has installed 3.8.6 and is working properly.
Click to expand...
Click to collapse
Thank you for this information - I'll put version numbers in the header.
xmxm said:
Windows version supports ForceBackupMode=Y setting to skip root-specific method and go straight for backup method, but I guess that run.sh is outdated by now - I'll need to find a way to copy newest logic changes to unix script.
Click to expand...
Click to collapse
You will find attached a zip with a new version of run.sh.
run.sh now support ForceBackupMode and internalization.
Thats why I add 2 folders (po and i18n).
If you want to add support for a new langage, let's say spanish for example, you will first have to create a new po file
Code:
bash --dump-po-strings run.sh > po/es.po
Then edit the file newly created with any text editor and translate the "msgid" with the "msgstr"
Once done, you have to create a new directory for the compiled langage file (.mo)
Code:
mkdir mkdir -p i18n/es/LC_MESSAGES/
Then compile the file with
Code:
msgfmt -o i18n/es/LC_MESSAGES/mibandextract.mo po/es.po
For another langage replace "es" with the one you want.
If you do not want to use the i18n folder and want the langage file to be system wide, you just have to comment (adding a #) the line "export TEXTDOMAINDIR=./i18n/" in the script.
Then you will have to move your po file to /usr/share/locale/XX/LC_MESSAGES/ (where XX is your langage).
Version updated to 5.5 with better support for FitnessSyncer and some chart updates
Good Work
xmxm said:
Version updated to 5.5 with better support for FitnessSyncer and some chart updates
Click to expand...
Click to collapse
Thanks for your work!!!
Hello dears.
I have been working for 5g volte vowifi since i bought the phone.
Also You know we have carrier settings files in /product/etc/carriersettings I edited one of them and now I need to replace with original one to try.
but this /product fs cannot be touched cannot be mounted R/W. But We have a option from Magisk. it is overlayfs for all R/O filesystem acting like real one.
Question is How we do this /product fs as overlayfs?
Here one guide we have but I did not excaly get it because I didnot work about building magisk models before.
https://topjohnwu.github.io/Magisk/guides.html
Also I find a module but When I try this I am getting Unzip error. (I already tried to zip again without upper folder )
GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
Emulate read-write partition for read-only system partitions - GitHub - Magisk-Modules-Alt-Repo/magisk_overlayfs: Emulate read-write partition for read-only system partitions
github.com
I worked with for ex vodafone_tr.pb file I get parameters from att5g_us.pb( It has lots of parameters for 5g volte vowifi) And I add these information to my carrier file. But I have to push this file to product.
Thank you
You need to create a Magisk module. Best is to download an existing module, unzip it, then modify it, the re-zip it.
Under the directory of the Magisk module, create a subdir called 'system', create a subdir 'product' underneath and 'etc/carriersettings´ subdirs as well. Put your file in there. Re-zip the module. Use adb to push to phone and install the module with the Magisk app. Reboot phone.
Magisk will mount your (modified) file in /product/etc/carriersettings/...
Check if the file is there:
Code:
> adb shell
$ su
# cd /product/etc/carriersettings/...
# ls
Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post
tauio111 said:
Did you edit the confseq files? What exactly did you edit, they are binary blobs. I also believe they are signed in one way or another - however I do not know what happens if the phone fails to accept one. Does it fall back to a hardware level carrier policy? Does it only leave out the specific confseq that was tampered with? Does it stop the modem from booting up at all?
Theres the cfg.db file along with its cfg.sha2 signature file - that would be my first point of try for remapping and editing via magisk... no idea wether the sha2 signature matters and what happens if it doesnt match.
It is a simple sqlite database:
1) you can look at the confnames table to identify the sequences per carrier
2) refer to the confmap table to see which carrier_id from the previous table responds to the confman hash
3) there are a load of confman_* tables, each of which includes the list of confseq files from the confseq folder - these confseq files have their corresponding names in their file headers - LTE CA carrier policies, NRDC carrier policies, mobile network based carrier policies, they all get added together in a sequence and all of them share most of the confseq files (the carrier specific ones are the ones that differ)
so what you can try, is, assuming you're on a "unsupported network", refer to the wildcard profile - this has the carrier_id of 0, which uses confman_43f507494f63c42cbf1aba626685b29710cd90eb as its table - the 10th one in order corresponds to the wildcard.sim1 confseq file which you can try an replace with one from another carrier (I've made a list of them https://paste.ee/p/NVju0)
the hashes also change by every modem release
Here is the sequence for wildcard and Orange Spain as an example:
default.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
wildcard.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
wildcard.sim2
default.multislotdefault.common
suclr_big_data_cc_num.common
lte_ca_common
lte_ca_0.common
lte_ca_1.common
endc_nr_ca_common.common
eu_common.common
default.sim1
endc_nr_ca_common_manual.sim1
endc_nr_ca_common.sim1
eu_common.sim1
eu_nr_common.sim1
es_orange.sim1
default.sim2
endc_nr_ca_common_manual.sim2
endc_nr_ca_common.sim2
eu_common.sim2
eu_nr_common.sim2
es_orange.sim2
default.multislot
eu_common.multislot
es_orange.multislot
Edit: nvm, didnt read the footer of your post
Click to expand...
Click to collapse
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.
furkanosman said:
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Click to expand...
Click to collapse
I just copy and paste the xml values every time, its persistent until you enter a different sim or update android.
I think it would be more viable to make a magisk script that adds the part you want to the front of the xml instead of replacing the bp files.
furkanosman said:
Woow you started middle of the book.
These carrierconfig part another story in the /vendor.
I didn't touch this confseq files because as you say these look like certificated. Also yes they are binary.
First /product/carriersettings section seems easy to touch for me.
Why I started this part you know some files coming in /data/user_de/0/com.android.phone/files when insert a sim card it's name like carrierconfig-com.google.android.carrier-899002....xml this is simple xml file that contains carrier parameters.
I started to try edit this file for a while. And Really ıf I change some thing wrong It directly affects the modem operation.
Seeing the same parameters in pb files in carriersettings, I thought I could edit them.
This is the story.
The next first I will try build a magisk module I will try to mount in hex base edited pb file
After that I will start to dig into these confseq binaries. Thank you for respond.
Click to expand...
Click to collapse
Hi
Did u manage to work on the pb files in /product
I tried the cfg method but my ims registration isn't changing