Daniel Sperl, developer of the Sparrow Framework, recently posted a performance comparison on the Apple Developer forum where Sparrow ran 2.5 times faster with MRC code than the version upgraded to ARC.
A curious finding though it seemed very far off from real world observations. Being a synthetic benchmark no less. I decided to do a similar test based on the same code comparing cocos2d v2 and v3.
Fortunately cocos2d-iphone v3 has made a similar switch from MRC (v2.1 and earlier) to ARC (v3 preview). Unfortunately the internals of cocos2d also changed to some extent, for example custom collection classes written in C were replaced by Core Foundation classes. I don’t have a full overview of the changes, but at least the renderer doesn’t seem to have changed in any significant way. Yet.
So while comparability is good, it’s not like Sparrow where truly the only changes made were converting the code from ARC back to MRC. Take the following benchmark results and comparisons with two grains of salt and pepper on the side.
ARC vs MRC
The original benchmark done with Sparrow has seen MRC perform 2.5 times better than ARC in a synthetic “draw as many sprites as possible until framerate has dropped consistently below 30 fps” test:
The discussion seemed to reveal that the issue may be a side-effect of Sparrow’s architecture taking a larger-than-usual hit from the switch to ARC due to insertions of extra retain/release. For example passing an object to a method will have ARC insert retain and release for the object even though the object’s lifetime will always exceed the duration of the method.
Cocos2d on the other hand doesn’t pass a lot of objects around during the update loop, so I was hopeful it wouldn’t take as much a performance penalty from ARC as Sparrow did. Turns out to be true:
ARC is in fact slightly faster for iPod touch 5 and iPad 1, while cocos2d-iphone v2.1 (MRC) is 50% faster on iPad 3. I can’t really explain this difference on the Retina iPad, it needs further investigation when cocos2d v3 is more mature.
I can’t even say whether the iPad 3 test is CPU or GPU bound. It may be CPU bound which might explain a greater negative impact of ARC. However if that’s true, that’s actually good news, because far more games have performance issues because of fillrate, not because the CPU couldn’t keep up. And there’s a simple optimization to be found in multithreading, given that almost all iOS 7 devices have dual-core CPUs.
Now of course that needs a framework which is capable of multithreading. Sprite Kit + OpenGW. Hint, hint. 😉
The effect of Batch Drawing
While I was at it I wanted to show how much of a difference batch drawing makes.
Interestingly the gap is far larger on the Retina iPad 3 than for the slower devices. Goes to show that even a much faster device (iPad 3) will render fewer sprites than an older device (iPad 1) that renders batched sprites.
Cocos2D vs Sparrow
Though the benchmark code I used is based on Sparrow’s benchmark scene, it can’t really be compared. I’m doing it anyway. 😉
What you need to know about this comparison is that Sparrow’s benchmark scene renders slightly differently. It allows some of the nodes to be partially outside the screen, which may skew results slightly in its favor.
The iPad versions of Sparrow’s benchmark can’t be compared at all because the results I see are quite different compared to cocos2d. It looks like internally Sparrow uses a fixed-size scene of 960×640 points for all devices, which leaves a small area of the iPad screen cut off. Plus the scaling is different compared to cocos2d, Sparrow’s sprites are about twice as large on iPads. Hence Sparrow’s iPad version of the benchmark produces far lower results than my ported benchmark.
Overall real world performance difference between the two engines doesn’t really matter.
The Benchmark Code
You’ll find the benchmark projects for cocos2d v2 and v3 in my github repository.
Look for the projects named Cocos2D-v2.1-Performance and Cocos2D-v3.0-alpha-Performance.
Side-note: just getting the two benchmark projects work on both Retina and non-Retina, on small and iPad screen sizes took me about 2 hours to get them decently correct. It’s bugs like disappearing menus on Retina, iPad not loading the correct images, labels wildly offset on each device and resolution and the fact that these behaviors change between v2 and v3 make me a very happy Sprite Kit user these days.
Poll for (future) Sprite Kit users
If you can spare a minute please take this poll, thank you!
Let me know what I should be working on or confirm that what I’m currently working on won’t be for naught … which apparently it isn’t. Good to know. 😉
Also take the preferred Objective-C render engine poll if you haven’t already. Thanks again!
|Follow @gaminghorror||Follow @kobold2d|