ROMs compared - Benchmarks, Camera & App performance (Updated: 18th Sep) - Sony Ericsson Xperia Mini, Mini Pro, Xperia Pro, A

Formerly: X-Ray USSR (Un-Scientific Study of ROMs)
Update: Background of the tests moved to the lower post. First post contains only the results and analysis.
Part 1
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Part 2
Update: 18th September
Added 2 new ROMs - Jelly Sandwich 5 and GxG Jellybread
Click to expand...
Click to collapse
Update: 10th September
Added 3 new ROMs - Jelly Sandwich 5.1, Superleggera, and Flylight
Click to expand...
Click to collapse
Remarks
Ice Cream Purenesss Build 11
Switching to Airplane mode switched of WiFi
No 'clear' to remove all notifications at once
Uninstalling apps shows a dirty error - android.process.acore has stopped
JJ's Hybrid 3.3b
Antutu benchmarks terminated prematurely twice and the phone was running quite hot
After initial caching and media scanning, the ROM sped up incredibly
Xperia Stock 587
<didn't take down any notes for this>
Xperia Stock 587 (minus bloatware)
There still seems to be bloatware after the bloatware remover is applied - so there is more potential to gain performance
Rayonium ICS v6.2
<no notes here too>
Rayonium ICS v7
Credential manager stopped during one uninstallation, and keeps occurring during apps usage
Feels laggy while using - can't validate for sure
MIUI v4 2.8.17 (NeilDownx)
Great usability. While it seems to be laggy at times, it is not, as the results show
FXP 134 CM9
For some reason, graphics performance doesn't seem to be great though the CPU performance is top notch.
Even using the latest Adreno ICS / JB drivers is not helping
Jelly Sandwich 5
Similar installer like KA21. However looks a lot more customizable - gives you so many customizing options that I was almost lost.
I gather this is what is the Aroma installer. (Can compare this installer to others like -> Mandriva to Slackware Linux)
Ewwww.... bootanimation is gross -> needs a change first if I stick to this ROM
GXG Jellybread
Oh well, it is still Gingerbeard. Never going to be my long(er) term ROM. GB just feels so outdated now after ICS and performance was not so great that it could win me over.
Superleggera V3 ICS
Incredible graphics performance in Antutu. Literally blew everything else out of the water. The first time I have seen the knights scene touch 50fps, and this thing went on till 60fps.
Flylight ICS
Camera takes very fuzzy shots. First focus is slow, and then subsequent focus are quick.
Jelly Sandwich 5.1
This ROM is so much better than its last version. Looks like something to do with the drivers.
The camera seems a tad faster, but I get this feeling that there is some sort of IO issue on the ROM.
On Antutu, the IO scores were quite low and pulled the overall scores down. Also while taking burst shots on the camera, there was a feeling that it as struggling after 3-4 shots - looked like write delays
And as far as bootimages go, this is clean and professional
KA21
Easily the awesomest installation procedure/UI of any ROM that I have seen to date (my first experience with Aroma). The ROM felt very laggy.
KA21 Barebones
Even the barebones version of KA21 feels very laggy and is just not able to multitask

---

Analysis, Background and Testing Process
Analysis
After all the tests, it turns out that good ol' CM ROMs are still the best. CM9 is quite awesome on this phone and my primary 'focus' - the camera runs the best and fastest on CM9/FXP. Only trouble was with the graphics benchmarks - it performed well on some and not so well on others. Wonder why? Even the latest graphics drivers did not help.
I thought MiUI ROMs are heavy, but surprisingly it performed rather well. So did the stock ROMs though it had a lot of bloatware. I guess that the stock ROMs are highly optimized since they are made for the phone in the first place.
On the other hand, a ROM that had been quite hyped - KA21 quite simply sucked. And not just with Benchmarks, but it was just plain unusable and was all laggy and slow. I couldn't even go through with the tests and got fed up midway through.
What was interesting was on that on a whole all ROMs behaved almost the same. So its just a matter of preference whether you install a lot o 3rd party apps - then you need the ones with higher internal memory free after installation. Also, with 3rd party apps - like music, browser, movies etc. behave almost exactly the same - which is why you see it grayed out. The difference between the ROMs seemed to be in how efficient their media scanner and IO throughput was. After the first few runs of each app, the next runs were pretty much identical. So when your app is already buffered, then subsequent access is superfast.
Unfortunately subsequent accesses are limited by the amount of free RAM. And some heavy ROMs here leave very little free RAM to the apps. This is where you need a good balance of free ROM and good memory management in the ROMs.
The best balance that I found - for my needs, was FXP/CM9 and I am sticking to that.
All in all, it will just come down to personal preference of look and feel - whether someone prefers Sony's look, or stock Android's look,or MiUI's convenience and ease of use.
18th September:
I added a couple more ROMs to the mix. I tried the much demanded Jelly Sandwich - it felt nice and looked very promising, but somehow did not deliver. I had a lot of expectations from this ROM. Benchmark scores turned out quite pathetic and this ROM ranks among the low ones. I felt so bad that I even tried a barebones version by removing almost everything and choosing the lightest components but performance did not improve significantly enough even to pull it from the orange or yellow band to one band higher.
I also benchmarked a gingerbread ROM, but even that did not deliver to my expectation. Looks like some of the best ROMs are the ICS roms. In fact I even tried a JB rom, but most of it did not work and I just did not continue with the benchmarking.
20th September:
Looks like some new ICS GPU drivers are doing the rounds. I've never seen the Ray pull off such great FPSes before. Flylight has got the best driver yet, hope it rubs off on the others.
Background
I had thought of starting off a new thread for a personal comparison that I am doing. I felt this would benefit others as well who loves playing around with ROMs, or who is looking for a ROM that suits their needs on paper.
I have tried out almost all ROMs on the forum for the Xperia Ray and like a spoilt child, I am never satisfied with any one. Probably the longest that I have had one is for 2 weeks. When I think back, I realize that most ROMs were chucked away - primarily because of boredom; but an other reason was that the performance deteriorated or I was not able to play some touted games.
I have now settled down to the following facts
1. Xperia Ray is not meant to play games (debatable - but that's my stand)
2. If you fill the last MB on the phone with apps, the performance is bound to drop
So, I have decided to do a un-scientific study of the various ROMs that are currently there to decide on one ROM and settle on it for a while (1-2 months at least :silly: )
The control device is my 5 month old Ray. It has a 16GB Class 10 card. The kernel is a XRay v7.3.431 overclocked to 1.4GHz. (the only exception would probably be the FXP ROMs which comes with its own kernel and I don't suppose it runs on the XRay)
The device will have a factory reset, all possible wipes, all possible formats, before installing the ROM and taking readings. There will be some measurements, some benchmarks, and some 'feeling' measurements. Nothing will be installed other than the benchmark tools and I will remove all apps from the recently-run apps before taking each measurement.
Testing Process
The setup of the phone is as follows for the tests
Preparation
1. Factory reset
2. Wipe cache
3. Wipe dalvik
4. Format system
5. Format data
6. Format cache
ROM Installation
1. Install ROM
2. Install any dependent patches
3. Flash compatible Kernel (if required, else stick to XRay 7.3.431)
4. Do initial setup - without using a google account
5. Overclock to 1.4GHz (use built in configuration, or install No-Frills CPU)
6. Set CPU Governer = SmartAssV2, IO Scheduler = noop (where possible)
7. Let phone rest for 5 minutes
8. Switch on WiFi
9. Activate GPS and location usage
10. Switch on installation from unknown sources
11. Add a google account and sync only browser and contacts
12. Let phone rest for 5 minutes after sync is complete
13. Go into Airplane mode
14. Keep charger plugged in
15. Start tests
Tests
1. Unload all apps from recent apps
2. Install app for testing (other than stock or whatever exists on the phone)
3. Run tests
4. Uninstall app
5. Repeat for next test
Note: All apps (for testing loading times) are opened once to dismiss off with popups and notifications and so that any caching and optimization can be done. They are then unloaded and tests are started. Theory is that the user is concerned with the speed of loading for commonly used apps, so they would have been opened many times and optimized/cached.
Q: Is there a way to integrate the google docs spreadsheet inline in this forum? Will make my life that much easier. Or at least a way to embed a HTML table into this post?

---

Good luck.. I've tried them all (unlocked bootloaders).
And there isn't better combination than
Kernel: Aire V2 + initd
Rom: RayoniumICS 6.2 (just because i don't like nxt pack)

---

stebenwulf said:
Good luck.. I've tried them all (unlocked bootloaders).
And there isn't better combination than
Kernel: Aire V2 + initd
Rom: RayoniumICS 6.2
Click to expand...
Click to collapse
well could you post a simple step by step walkthrough to achieving this result...

---

rom study?
i figured (using gxg jellybread right now) that my ram is getting less.
after flashing and using some certain apps my complete ram was about 300mb, after restoring my (about 200, lot of games) via apk-installer, my ram (as you can check via applications -> running services) went down to about 220-230mb
how is that possible? i dont get it.
with ics i did not have this problem (afair) but ics is too slow for the ray imho.

---

JB is fun, but it is still in infancy. I wouldn't use it as my daily ROM yet. So I will stick to ICS comparisons for now, and will add JB as the ROMs get more stable. I could have added GB to the comparison too, but then GB is soo boring that I wouldn't want to have it around on my phone for even a while.

---

geekoo said:
JB is fun, but it is still in infancy. I wouldn't use it as my daily ROM yet. So I will stick to ICS comparisons for now, and will add JB as the ROMs get more stable. I could have added GB to the comparison too, but then GB is soo boring that I wouldn't want to have it around on my phone for even a while.
Click to expand...
Click to collapse
jellybean is not ready for daily use yet, i agree. the rom (the android version) i was talking about was gingerbreaed.
gxg jellybread is just a gb-rom that looks a little bit like jellybean, nothing more, but its still gingerbread
yeah, gb might be "boring", but its the fastest and fulliest functional rom you can get for ray right now.
if this weird ram-issue wouldnt occur i wrote about...

---

i am currently using Paranoid Android v1.4.1 (ICS Version) and right now, i am pretty satisfied.. it's smooth enough for daily use..
i used 2 extra apps..
automemory killer pro - set to extreme
ram manager pro - set to hard gaming
cpu governor - performance
max- 1ghz
min- 158mhz
this is the result of using FPS2D..

---

Updated the charts with my measurements and comments. Hope you find this useful if you are looking for a new ROM.

---

Nice thread, I really like the evidence-based, unbiased methods. As a scientist myself this is important in any comparison This was something I was contemplating doing myself - now I don't have to!

---

thanks for the chart
I think I will still stick with JJ's 3.3b
JJ's may not be the fastest rom but it still on the fast side and offer a lot feature too.
And I modded the hell out of this so I will miss my mod when install other rom.

---

foggyflute said:
thanks for the chart
I think I will still stick with JJ's 3.3b
JJ's may not be the fastest rom but it still on the fast side and offer a lot feature too.
And I modded the hell out of this so I will miss my mod when install other rom.
Click to expand...
Click to collapse
Oh, JJ is definitely one of the fastest ROMs there. In fact JJ, Stock, and FXP are almost on par (what's 1 or 2 points?) In fact the only thing that let JJ down was the amount of free RAM. Other than that, in the benchmarks, it was indeed consistently the fastest ROM.

---

Voodooguru said:
Nice thread, I really like the evidence-based, unbiased methods. As a scientist myself this is important in any comparison This was something I was contemplating doing myself - now I don't have to!
Click to expand...
Click to collapse
Thanks
Spent almost a week without a life getting this chart ready - wiping, rewiping, installing, uninstalling, measuring etc etc

---

Did you uninstalled the bloatware from stock ROM using any automatic bloatware removal tool or uninstalled the applications manually?
According to my experience : after I removed bloatware one by one using 'root uninstaller' the free internal memory became 332 MB. (I removed facebook inside xperia, google maps, talkback etc also)
and before i started installing the applications and mods the the free RAM was around 200 - 195 MB.
I can play most of the games (which are playable on Ray) very smoothly.
The last game I played was NFS hot persuit. The graphics ran super smooth, but the sound sometimes become choppy though.
I actually don't see a very big reason to jump from stock ROM 4.1.B.0.587 (minus bloatware, plus some of the very very useful mods and tweaks available in this forum) to a custom ROM.
Some people may have different views. I respect their views, but for me, I am happy with 4.1.B.0.587

---

Really nice thread
By Rizal Lovins [email protected]"Love Indonesia"

---

JJ's is the best .. at least for me

---

i've been flashing different rom, but i been missing the GB experience.. GxG's Jellybread is quite amazing for it's ICS-GB-JB experience.. but it's actually GB. i haven't tried the new stock 587 though, because of your chart i can see the performance increase with the new stock.. i will give it a try thanks mate..
tintin100 said:
Did you uninstalled the bloatware from stock ROM using any automatic bloatware removal tool or uninstalled the applications manually?
According to my experience : after I removed bloatware one by one using 'root uninstaller' the free internal memory became 332 MB. (I removed facebook inside xperia, google maps, talkback etc also)
and before i started installing the applications and mods the the free RAM was around 200 - 195 MB.
I can play most of the games (which are playable on Ray) very smoothly.
The last game I played was NFS hot persuit. The graphics ran super smooth, but the sound sometimes become choppy though.
I actually don't see a very big reason to jump from stock ROM 4.1.B.0.587 (minus bloatware, plus some of the very very useful mods and tweaks available in this forum) to a custom ROM.
Some people may have different views. I respect their views, but for me, I am happy with 4.1.B.0.587
Click to expand...
Click to collapse
can i ask for the list of playable games the run in your stock 587? i would be thankful for that

---

tintin100 said:
Did you uninstalled the bloatware from stock ROM using any automatic bloatware removal tool or uninstalled the applications manually?
Click to expand...
Click to collapse
I used just the automatic bloatware removing tool that knight has posted on the forum. I have mentioned in the comments in the chart there is still so much scope to remove bloatware, so obviously a lot more space and memory could be freed up.
I think Sony has done quite a good job with their ROM - not considering the bloatware.

---

Do you really prefer the CM9 camera instead of SE stock camera?

---

can you test Jellysandwich Mr. Tapa ROM