Related
I had an idea as I was finishing wrapping the last of my Christmas gifts .
I went straight scientific method with this one too
An app needs to be created that will put a constant, equal load on our phones CPU's and with this will be able to accurately determine which kernels are going to give us the best battery life.
Here's where I'm going with this:
You change the setting on your phone so that the screen stays on and put your phone into airplane mode after a fresh flash of a ROM and the desired testing kernel. You make sure that nothing else is running on the phone asides from this app. Run the app until your phone dies and compare with other kernels. Granted, this wont be the same for every kernel-rom combination but I think that it is a good starting point as for some kind of "battery benchmarking app" that we could use to try to determine what is going to work best for the ROM of our choosing.
99 Reviews....No Reply. Friendly bump
i think its a beast idea.
Battery/Kernel stress testing...hmm.... love concept but too many other variables to base results solely on kernel. Maybe possible if those many variables can be taken into effect or results from many different phones, batteries from different manufacturers, different size batteries, etc are averaged. Another issue is that not all kernels are built for battery life..ie..concept of BFS and CFS based kernels.
Great idea and I think this is possible but again variables need to be taken into mind.
devnulldroid said:
Battery/Kernel stress testing...hmm.... love concept but too many other variables to base results solely on kernel. Maybe possible if those many variables can be taken into effect or results from many different phones, batteries from different manufacturers, different size batteries, etc are averaged. Another issue is that not all kernels are built for battery life..ie..concept of BFS and CFS based kernels.
Great idea and I think this is possible but again variables need to be taken into mind.
Click to expand...
Click to collapse
Maybe a stripped down battery testing rom. No radios so that signal strength won't affect the results. Maybe you just run it and see what's best for your specific setup and forget about averaging.
Something very similar has already been done. If I can find the http I will post it. To do what your saying sounds like it would be extremely time consuming and possibly harmful to the phone, depending on the amount of stress you put it under for the time it takes to drain the battery.
Sent from my PC36100 using XDA App
We just need a Prime 95 for android now!
I think you'll run into a problem since different phones handle kernels differently
Just install System Monitor and Battery Info Widget. You can see from System Monitor accurate CPU usage (a little over ~1% idle should be typical) and from Battery Info Widget how much mA's that usage is taking (between -40mA and -120mA is typical for 1%).
definitley seems like a great idea
As we all know quadrant is no reliable measure for speed. At least I knew this for a while now and it was repeated and quoted many times.
This article tells anybody with a functioning brain (that is used of course) that quadrant means pretty much nothing.
I can't help to run it from time to time anyway
So I sat on the to... in my room in front of my computer with my phone. I9000 with supersonic ROM and the remount script from adrenaline shot 7. I sat there and said to myself "how hight can you score in quadrant LOL"
I started quadrant up and ran the benchmark: 2309
Then I opened the task manager-> Exit all & Clear memory
Then via long press homebutton back to quadrant to run the benchmark again score: 2453
But since I am a programmer and can imagine all kinds of optimizations and caching I pressed the back button and just ran it again just after it finished
Score: 2675
How the hell could anyone call that a benchmark?^^
just to be sure could anyone confirm that behavior? And does anyone know of a mor reliable alternative? I'd like to collect that knowledge in this thread.
TL;DR: quadrant sucks, you know anything better or want to flame away: do it here
Those are not the actual numbers from my first experiment, I repeated the scenario just now and took the numbers from those runs.
Additional runs scored 2775, 2907 and 2820, that's just silly
I think this behaviour is well known and has to do with JIT optimizations or something like that
allotrios said:
I think this behaviour is well known and has to do with JIT optimizations or something like that
Click to expand...
Click to collapse
The reason is irrelevant. The fact it doesn't provide a reliable benchmark is.
no benchmark is precise if you don't use it as intended. Quadrant produces a reliable comparative benchmark when used as designed: run it five times, remove the lowest and highest scores and average the remaining 3 -- that is your benchmark. You may not like it, but that is how it is designed to be used.
Now if you want to be pedantic, you could reasonably test again, by running quadrant 5 times, removing the outliers and average your 3 remaining scores. Repeat 10 times and then tell me how your average scores do or do not vary: they will in fact be within a narrow range, your actual benchmark.
Alternatively, tell us which benchmark produces the same score each run, as that appears to be the sum total of your objection to quadrant.
There are other benchmarks, such as Caffiene Mark, AnTuTu and NenaMark, but they are all apps just as Quadrant is and all require several runs and averaging to produce a comparable benchmark.
Moreover, the primary use of any benchmark is to compare firmware (kernel and rom) builds on the same phone to see relative performance gain and drop.
A benchmark is supposed to give way of comparing the capabilities of a given device. This means that a device with a high average score implies a better device than a lower score.
But the Quadrant score does nothing of this sort! In a competition with a friend I achieved an average Quadrant score of about 4300, with a peak of 4462. According to Quadrant my device is a lot better than the OP! Which is just not true.
Quadrant is unreliable as a benchmark, no matter how it is "designed to be used".
Sent from my GT-I9000 using XDA App
whaave said:
But the Quadrant score does nothing of this sort! In a competition with a friend I achieved an average Quadrant score of about 4300, with a peak of 4462. According to Quadrant my device is a lot better than the OP! Which is just not true.
Click to expand...
Click to collapse
You're doing it wrong.
lgsshedden said:
Moreover, the primary use of any benchmark is to compare firmware (kernel and rom) builds on the same phone to see relative performance gain and drop.
Click to expand...
Click to collapse
Quadrant scores are useless. I've used custom roms with scores of 2500+ but they aren't as smooth as stock roms, which only have scores of 1600-1800.
Antutu is indeed quite reliable imho. My results never fluctuate more than +-5% on the same config. That's an acceptable range, considering I don't set cpu governor to performance before running my tests.
Sent from my GT-I9000 using XDA App
upichie said:
You're doing it wrong.
Click to expand...
Click to collapse
w00t?
Quadrant does not reflect performance, and therefore can not be used as a comparison parameter.
It can't be much worse than I thought.
My phone with 2.1 and 'lag fix' scored 2200 and lagged so bad I wanted to throw it against a wall multiple times a day.
With stock 2.3 quadrant can be ~1000 but the phone runs much smoother.
Other than the obvious file systems I/O 'cheats' that resulted in the above, there is also the frame rate cap that makes the GPU tests useless as well.
if your trying to measure height with a scale , u wont get your answer .
The only benchmark tool that ever reflected how the phone felt in my hands , in real life usage is linpack .
changing OC / kernel is mainly the only thing that will affect linpack if your trying to use it to compare roms ill efer you to my first statement .
In order to have a good feel of a rom / set up on the phone , use some apps that will use lots of ressources , for example TW4 launcher , go in there scroll a lot open gallery (if you have many pics) scroll thru them and repeat ... Any benchmark tools will basically tell you the 'ability of your device ' ( comparing 2 different models like an inspire and an sgs2 for example will be accurate )
ZioGTS said:
Antutu is indeed quite reliable imho. My results never fluctuate more than +-5% on the same config. That's an acceptable range, considering I don't set cpu governor to performance before running my tests.
Click to expand...
Click to collapse
I recently tried Passmark Mobile. Still a beta version, but I like it. Test results reflect real performance improvement and degradation pretty closely, particularly for what concerns I/O and memory speed.
Quick question:
Occasionally and sometimes after a reboot, my stock ATT GS3 pulls a Quadrant score of 5300-5400. But, I can randomly run the benchmark and get about 3300. Now, I know the community doesn't put a lot into these scores, but I wanted to ensure there isn't an application sapping power.
Does anyone know which applications I should be wary of or what could be causing this? Is there a better benchmark app to gauge performance (besides real-world use) ?
Benchmarks are pointless. They can be faked all too simply. Never put faith in them
Wayne Tech S-III
when I install a google edition or cyanogenmod rom or any rom thats not touchwiz I am getting lower scores just wondering why?
Lower's better. And all the TW junk in the background is taken out, leaving room for more processes, so of course it scores better.
Sent from my GT-I9505G using XDA Premium 4 mobile app
Tak23 said:
when I install a google edition or cyanogenmod rom or any rom thats not touchwiz I am getting lower scores just wondering why?
Click to expand...
Click to collapse
But does it feel faster? Because that's what really matters...
lordcheeto03 said:
But does it feel faster? Because that's what really matters...
Click to expand...
Click to collapse
This plus eleventy bagillion.
Benchmarks are totally arbitrary and meaningless.
TouchWiz ROMs scores great on benchmark tests because TW is specifically designed to perform well on such tests by putting the whole device into overdrive when the test is run. It doesn't mean that your day to day usage is better.
Tak23 said:
when I install a google edition or cyanogenmod rom or any rom thats not touchwiz I am getting lower scores just wondering why?
Click to expand...
Click to collapse
That's a very subjective statement. "Lower is better" really depends on the apo being used. Some, eg. Quadrant, a higher score is better. Linpack is the same way.
As said before, benchmark apps don't tell you the real performance of a device. Though they can give you a baseline or at least a general idea of different aspects. I wouldn't jump on the "benchmarks are useless" bandwagon as they do serve a purpose.
Sent from my SGH-M919 using xda app-developers app
KCRic said:
That's a very subjective statement. "Lower is better" really depends on the apo being used. Some, eg. Quadrant, a higher score is better. Linpack is the same way.
As said before, benchmark apps don't tell you the real performance of a device. Though they can give you a baseline or at least a general idea of different aspects. I wouldn't jump on the "benchmarks are useless" bandwagon as they do serve a purpose.
Sent from my SGH-M919 using xda app-developers app
Click to expand...
Click to collapse
Well they are useless when ROMs are set up specifically to perform well on benchmark tests. That gives aboslutely no real world value.
The only time there is any validity to a banchmark comparison is when you are trying to compare 2 things that are set up almost exactly the same with only a single differing variable.
Like if you want to see if a phone is bad, you'd need to run a benchmark on the target phone and another on a different phone of the same model with the exact same settings. Or if you want to see in a mod if the cause of reduced performance, you would need to run a benchmark without the mod, then immediately after the mod, and repeat it several times to ensure you get repeatable results. Etc etc etc...
You can't just abandoned the scientific method and compare a benchmark of a TouchWiz rom with a benchmark of an AOSP rom with god only knows how many differences in the set up (kernel, clock speed, gpu settings, voltage, etc etc etc) and have it be a valid means of determining which is the better rom in any way. You can't even set it up properly so that all the settings match and the only difference is the ROM because there is so much incompatibility betweent the two that factors into it. Especially when the TouchWiz rom is set up with methods of inflating it's scores on benchmark tests and the AOSP ROM isn't.
I agree. Scientific method comes first in determining the performance of a ROM. Actually I would say real world and use experience is the best method. Given today's Hugh end phones, our for example, getting the most out of a benchmark or anything else serves little purpose. I can disable 2, even 3 core and still have a smooth experience with most daily a activities. Even underclocking the CPU & GPU still retains all the speed and usability.
I think most tech literate people use benchmarks just to see what the phone can do. Also, as a quick though fairly inaccurate method of determining if you're ROM and it's mods/scripts are efficient.
To answer the OP's question though - since AOSP is HW accelerated, has been for a while now, it should be "faster" and yield better benchmarks. Less running in the background. The most things are integrated as opposed to added on to the base ROM and code. That's a very generic and butchered explanation.
Sent from my SGH-M919 using xda app-developers app
I started one for g4 plus now for g5 plus .
Cosmic os 2.1 unofficial
Elemental x kernel over clocked
What benchmark program are you using?
username8611 said:
What benchmark program are you using?
Click to expand...
Click to collapse
Antutu
PureNexus using ElementalX stock CPU speeds and GPU governor, CFQ, custom CPU governor settings
Lineage OMS with ElementalX kernel stock CPU speed and governor. ZEN with custom readahead.
This is kind of useless, benchmark comparison means nothing if it is not on the same device with same set of apps installed.
Sent from my LG G5 using XDA Labs
suhridkhan said:
This is kind of useless, benchmark comparison means nothing if it is not on the same device with same set of apps installed.
Sent from my LG G5 using XDA Labs
Click to expand...
Click to collapse
That doesn't make any sense. Devices are manufactured to a certain tolerance and winning the "silicon lottery" doesn't make a device faster, it makes it more overclockable. Device to device, stock for stock, the difference should be at most a few thousand points from each other. It should be pretty obvious to kill all background apps and processes before benchmarking so apps installed don't matter either. If Facebook is too important to kill for 10 minutes then that person shouldn't worry about benchmarking.
Device to device are obviously going to vary. But a varience of 10k+ points is a pretty good indicator of one set up running slightly better than the other and it's interesting to compare what is the most optimized settings. I can play with my CPU governor all day and get repeatable results +/- 500 - 1000 points. Both me and my wife had a Nexus 5 and with identical settings we both benchmarked very similar. To say it is a useless test is ignorant. If people look at this as a pissing match to see who's "better" then yeah, I see this being a dumb and useless thread. But I think most people who do this want to know what settings, ROM, and kernel are best optimized for performance.
Edit: https://www.phonearena.com/phones/Motorola-Moto-G5-Plus_id10398/benchmarks
63,191
https://www.youtube.com/watch?v=345eKlssdH8
62,769
http://www.fonearena.com/blog/214719/moto-g5-plus-review.html
62,893
https://www.pcmag.com/review/352573/motorola-moto-g5-plus
63,845
http://www.guidingtech.com/65986/moto-g5-plus-vs-redmi-note-4/
62,896
5 different devices, all tested stock within right around 1,000 points of each other.
username8611 said:
That doesn't make any sense. Devices are manufactured to a certain tolerance and winning the "silicon lottery" doesn't make a device faster, it makes it more overclockable. Device to device, stock for stock, the difference should be at most a few thousand points from each other. It should be pretty obvious to kill all background apps and processes before benchmarking so apps installed don't matter either. If Facebook is too important to kill for 10 minutes then that person shouldn't worry about benchmarking.
Device to device are obviously going to vary. But a varience of 10k+ points is a pretty good indicator of one set up running slightly better than the other and it's interesting to compare what is the most optimized settings. I can play with my CPU governor all day and get repeatable results +/- 500 - 1000 points. Both me and my wife had a Nexus 5 and with identical settings we both benchmarked very similar. To say it is a useless test is ignorant. If people look at this as a pissing match to see who's "better" then yeah, I see this being a dumb and useless thread. But I think most people who do this want to know what settings, ROM, and kernel are best optimized for performance.
Edit: https://www.phonearena.com/phones/Motorola-Moto-G5-Plus_id10398/benchmarks
63,191
https://www.youtube.com/watch?v=345eKlssdH8
62,769
http://www.fonearena.com/blog/214719/moto-g5-plus-review.html
62,893
https://www.pcmag.com/review/352573/motorola-moto-g5-plus
63,845
http://www.guidingtech.com/65986/moto-g5-plus-vs-redmi-note-4/
62,896
5 different devices, all tested stock within right around 1,000 points of each other.
Click to expand...
Click to collapse
Thank you for taking the time to write a long response. But, I believe you may have just proved my point. I believe the test results of different roms should be well within 'around 1,000 points of each other'. Unless-
a. the rom is very poorly optimized - score would be lower.
b. the kernel is overclocked - score could be slightly higher.
c. user error (lots of background apps).
suhridkhan said:
Thank you for taking the time to write a long response. But, I believe you may have just proved my point. I believe the test results of different roms should be well within 'around 1,000 points of each other'. Unless-
a. the rom is very poorly optimized - score would be lower.
b. the kernel is overclocked - score could be slightly higher.
c. user error (lots of background apps).
Click to expand...
Click to collapse
I really don't know how else to explain this to you. OP got a lower score than me, yet is overclocked. So it stands to reason that either "a. the rom is very poorly optimized - score would be lower" or "b. the kernel is overclocked - score could be slightly higher" or "c. user error (lots of background apps)" is the reason for it. But wait, the performance should be slightly higher for an overclock except that it isn't. That's the whole reason to benchmark. Another possibility is that since I've heard ElementalX is currently having overclock issues, it may be reverting to its nominal frequency, which I believe is 1.4Ghz. How would this person have known that if not for comparing benchmarks? According to you, they can't compare to stock benchmarks because it's a different set of apps installed and a different ROM and in fact can't compare it to anyone because it's a different device, albeit the same model.
Benchmarks show performance differences, regardless of whether or not they are large enough to even notice on a day to day basis. It shows technical differences and if you think technical differences mean jack squat, then why are you even commenting in this thread? It's the same theory when you throw a car on a dyno. You're going to notice small differences between each run, but when you have two of the same model cars with the same engine, and one consistently puts out 30HP more than the other, there's probably a reason for it.
To reiterate what I said in my first reply, for people who want to compare optimization between different ROMs, kernels, and technical settings such as CPU governors and schedulers, benchmarking is not useless. Not in this method of testing and not across identical devices with different software. The baseline or "stock vs stock" comparison shows that the benchmark is measuring with an adequate amount of accuracy and that multiple devices in stock form are performing equally before being modified. Just because it doesn't mean anything to you doesn't mean that it means nothing at all.
I did some research and things like backround apps running in airplane mode scripts like lightning blade. all these things make a difference. I was running kernel over clocked in interactive mode with lightning script. If I set to performance my score was significantly higher I was hoping this would give users a better way to set up and optimize their device not to compare roms running same device. Yes at first I thought about that then realized it wouldn't make a lot of sense. Im hoping some of u guys will hop on board and help test kernel roms and other mods so maybe we can get the best out of our device thanks guys.