Runtime Integrations Performance Plan

[csciguy8]

(currently work in progress, feedback welcome)

Let this issue propose a methodology for determining performance and to answer...

Is our product performant?
What are the questions we should ask?
How do we answer them?

Applies to Cesium's runtime integrations, but also extends to any project that uses cesium-native in the community.
cesium-unreal
cesium-unity
cesium-omniverse

Background

When we say performance, what are we referring to?

• Load times - How quickly tiles appear when using a Cesium app
• Frame rate (FPS) - How smoothly animation is presented
• Efficient use of device resources - Network, Memory, CPUs, GPUs, etc

How do we measure performance?

• It must observable - Either through intuitive use, or relevant tools
• There must be metrics that specify a threshold - Essentially what has been decided as 'good'

The Case for Performance Reporting

We need a baseline. A performance report shows how our products are doing, quickly, without a deep dive of technical data.

The details should be intuitive, whether the reader is in programming, project management, or business development.

Example Report Output

POOR  | Load time when starting an app | 10 sec | [Unreal 5.0][Win][NoCache][Denver]   | CesiumForUnreal
OK    | Load time when starting an app | 2 sec  | [Unreal 5.0][macOS][NoCache][Denver] | CesiumForUnreal
OK    | Load time when starting an app | 1 sec  | [Unreal 5.0][Win][WarmCache][Denver] | CesiumForUnreal 

In the above example, the Cesium for Unreal plugin is being tested for load time, under various conditions. 3 test instances were run with each line corresponding to data about its conditions and passing state.

Reports should help with hard questions

A programmer might ask:

• Is implementing X optimization worth Y amount of development time?
• Did X code change help or hurt performance?

A business developer may ask:

• Are load times ok when running on some-low-spec-device with World Terrain?
• Will I experience similar framerates if I choose Unreal over Unity?

A product owner could ask:

• Is performance competitive with similar products on the market?
• What are the weakest performing use cases that hurt our reputation the most?

Performance Testing

Reports summarize based on data, which comes from performance testing.

Performance tests answer specific questions to predefined use cases. The best tests are simple ones. Ex. "How long did a level take to load?". Add more conditions of the test to create more coverage. Create enough tests with enough coverage, and you are closer to a comprehensive picture of performance.

Test data (even a report) should be generated without user intervention. The ultimate goal would be to run under continuous integration with extensive coverage across device types and use cases.

This is not to be confused with a performance profile, which is invaluable to a programmer fixing a performance problem. Profiles provide information like how long X function took to execute in this frame, the idle time of each CPU, or the number of times memcpy was called. Performance tests on the other hand, focus more on the user situation that was run, and if it fails expectations.

Example Tests

Test 1: Load times when starting an app

      -Common-

      Description: User starts the app at an initial location and waits until the view is fully loaded.
      Measurement: Time from first tile request until final level of detail is presented
      Map Assets:  Cesium Ion World Terrain + Bing Maps Aerial
      Condition Set: [Empty, Warm]             // Cache state
      Condition Set: [Denver, Paris, New York] // Start location

      -Client Specific-

      Condition Set: [Unreal 5.0, Unreal 5.1, Unreal 5.2]
      Condition Set: [Windows, macOS, iOS, Linux, Android]

A pseudo-definition of how we could conceptually define a test.

The common section defines what any runtime integration could implement. It should be described well enough to:

• Understand the start and end of the test
• What is being measured
• Any number of 'Condition Sets' that imply multiple runs of the same test

This shows 2 states the cache can be in (empty or warm) in addition to 3 different locations to start in. This means at minimum 6 (2x3) different instances of the test should be run.

The client specific section would define additional aspects that cesium-native would not know about. This includes the versions of Unreal or platforms that are being built. This would add more conditions sets, multiplying the total runs by 3, then 5, totaling 90 (6x3x5).

Already we see that judicious of use of conditions will be needed to keep report generation time reasonable.

Test 2: Per-tile load times

      Description: Wait until the view is fully loaded, spin around 360 in 10 seconds.
      Measurement: Per-tile loading time (min, max, average)
      Map Assets:  Cesium Ion World Terrain + Bing Maps Aerial
      Condition Set: [Empty, Warm]             // Cache state

Test 3: Frame rate when flying from location to location

      Description: Wait until view is fully loaded. Fly to different locations.
      Measurement: Frames per second during animation (min, max, average)
      Map Assets:  Cesium Ion World Terrain + Bing Maps Aerial
      Condition Set: [Empty, Warm]             // Cache state

The Role of Cesium-Native

Cesium-native does not know the about the project that is using it, but can provide supporting tools and methodology.

It is ultimately the runtime integration's responsibility to integrate performance tests into the project.

Research will be required here:

• How much code can cesium-native provide for performance testing?
• How do we get the integration effort to a minimum?
• How do we share performance tests across multiple different integrations?
• How do we foster a shared methodology across different integrations?

Unknowns

• What will be the integration workflow to adopt performance testing?
  For example, if I'm developing for the Cesium for Unreal plugin and want to see performance reporting, I would:

  1. Choose an appropriate automation system. In this case, let's use Unreal Engine's built in solution
  2. Implement performance tests in Unreal (C++), matching cesium-native's recommended cases, or make my own
  3. Register the results of the tests to cesium-native using provided APIs
  4. Generate a report using cesium-native provided scripts (or APIs), or make my own

  This sounds ok, but could it be better?

• What does this offer implementations on mobile devices (tablets and phones)? Traditionally it is hard to integrate automated performance testing here. Does the shared nature of cesium-native give us an advantage?

• What does this offer VR / AR / XR headset development?

• How far into development workflow does this extend? As prevalent as unit tests in CI?

Proposed Task Schedule

(replace with actual issues as they are created)

• Implement initial performance testing in Cesium for Unreal
• Move integration-independent code to cesium-native
• Integrate performance testing in Cesium for Unity
• Integrate performance testing in Cesium for Omniverse
• Add reporting features (set acceptable thresholds, generate summary, collate across integrations)
• (continuous) Add support for critical use cases
• (continuous) Add support for additional platforms (Linux, macOS, etc)
• (continuous) Add support for additional devices (phones, tablets, headsets)

Reference

Performance improvements for unreal and native
CesiumGS/cesium-unreal#867

Load tracing in cesium native
#267

[timoore]

Just a couple of comments:

I understand Cesium's priorities, but it might be useful to give a nod to native clients of cesium-native in general, not just those maintained by Cesium. I'm selfishly thinking of vsgCs, but there are / will be others, and the performance of cesium-native may be easier to characterize against a lighter-weight client than Unreal et al.

I have had great luck with Tracy and have annotated parts of vsgCs that implement the cesium-native interfaces with the Tracy macros. It would be awesome if the cesium-native tracing annotations "just worked" with Tracy; maybe they do, but I haven't tried it yet.

At least some of the performance impact of cesium-native on the client comes from the quality of the glTF model representation of the tiles. #610 would be useful for testing that, among other things.

[r-veenstra]

One thing that we should consider is identical test content across platforms where this is feasible, so that we can establish and maintain a consistent performance baseline across Unity, Unreal, Omniverse and others.

[csciguy8]

@timoore, great comments. And yes, we should absolutely consider other projects using cesium-native. I've updated the verbiage for this.

@r-veenstra Agreed. Lots of value in finding performance consistency (or inconsistency) among Cesium's integrations.

I've updated the description, adding sections for Performance Reporting and Testing. There are more ideas and details need to be fleshed out...

Comments still very welcome.

[csciguy8]

Updated the description to add 'Unknowns' and 'Proposed Task Schedule' sections.

I think the main ideas are fleshed out. There still unknowns as to the role of cesium-native and how to maintain performance testing consistency. At the very least, we are asking good questions.

[ptrgags]

A couple of notes from my team at Cesium:

• So far we've been exclusively testing our output in CesiumJS
• The important metrics we care about:
  • How long does it take for the tileset to load. (e.g. in CesiumJS elapsed time from Cesium3DTileset.fromUrl() to the initialTilesLoadedEvent). We've been using a single camera view for repeatability. Though if there's a better approach, open to ideas
  • GPU memory usage. Ideally broken out into geometry vs textures
  • How many tiles are loaded

Making these easy to measure would be very helpful to us!

[csciguy8]

Closing this.

I believe we've reaped a good deal of value from this issue, and don't believe the work remaining is worthwhile to pursue, at least right now. And if there is, we should probably write it up in a smaller, more targeted issue. So please do so if any of your ideas got lost.

Here are some highlights since this issue was written:

• Performance tests were created for cesium-unreal with this PR
• Profiling guides were created for cesium-unreal with this PR
• Performance reports were never generated from code (still manual), but have proven to be very useful in quantifying improvements, like in this miracle one-liner PR that sped things up 2.5X
• Although the cesium-unreal contributions aren't part of this repo, performance tests and reports can still help drive improvements in cesium-native, as shown in this PR

Some outstanding questions... (answered to the best of my ability)

• How much code can cesium-native provide for performance testing? Unclear. During performance investigation, I found that a great deal of the code was integration specific. Some was integration agnostic, like the specific test locales, world location, which ion assets to test, etc. But that information was the easiest to just copy / paste.
• Can we performance test just the cesium-native code? Yes, but the integration can also introduce severe performance problems (mentioned before) and there's value in testing all of it at once. Typically, when I look at fixing performance, I want to see the whole picture, and fix the biggest problems first.
• Should we establish and maintain a consistent performance baseline across Unity, Unreal, Omniverse? If there's a business case for this, great. And if all these products have performance tests, fantastic. But does cesium-native have any role to play here? It's hard for me to see it.
• How far into development workflow does this extend? As prevalent as unit tests in CI? For any use case, performance tests should be repeatable, and easy to run. But when you want to run on a virtual machine, regularly, the problem gets a bit harder. If there's business need to justify the cost of this, sure. But until then, we'll pass.
• Will this help phones, tablets, headsets? Again, it takes a bit more development time and cost to extend running these tests on specific devices. There's no work towards this now, but maybe later, if a business case comes up.

Originally this issue created a plan, but along the way supported a methodology:

1. Identify a performance need (who is asking what? is it observable?)
2. Measure performance (test should be easy to run, repeatable)
3. Profile to discover areas of improvement
4. Make changes
5. Go back to 2), repeat until satisfied

Thank you to everyone who contributed to this issue!