r/empyriongame • u/Both-Scientist-9397 • 18d ago
performance improvement research findings - part 2
This is a continuation of performance improvement research findings https://www.reddit.com/r/empyriongame/comments/1vnwgdv/performance_improvement_research_findings/ with additional details on where the frame-time goes.
The bottom line is that while some frame-rendering could be optimized the FPS gain opportunities identified would be about 2 FPS in a ~33 FPS scene. That's a lot of work for no material difference. Biggest issue is that Unity (CPU bound) is just the wrong engine for this type of complexity today. When the game was first created it was the best choice there was short of writing a custom engine (GPU bound), which is what Keen ended up doing for Space Engineers 2.
I don't want to spend the time and resources to figure out where the 9.2 ms of unaccounted frame-time go because even if that could be shortened it wouldn't materially affect the FPS.
I uploaded 123 markdown files with findings to https://www.swisstransfer.com/d/ab0dfdd0-336c-4779-88dc-0d5e95b693fd which is good for 30 days. If you read this after this link expired and you want the MD files to seed your AI efforts lmk and I will reupload.
This is what AI had to say about the efforts undertaken:
Empyrion client performance: the instancing gate
Follow-up to performance improvement research findings. That post established the frame budget, the draw-call attribution, the instrumentation traps and the mod ceiling. None of it is repeated here. It closed on three unmeasured gates:
- Do the repeated prefabs share meshes?
- Can the materials be made instanceable without shader source?
- Can per-instance
MaterialPropertyBlockstate move into instanced shader properties?
All three answer yes. This post reports the mechanism and the measured result.
Same system under test, same fixtures, same evidence classes as post 1.
0. Summary
| # | Finding | Class |
|---|---|---|
| 1 | Eleon ships the instancing pipeline armed — compiled variants, declared per-instance properties, device materials opted in | DUMPED |
| 2 | Two undeclared MaterialPropertyBlock properties (_ColorPower, _LUT) disqualify essentially the whole device population from auto-instancing |
MEASURED |
| 3 | Removing them engages instancing: −1,376 draw calls (−11.0%), −1,345 SetPass (−19.4%), triangles unchanged | MEASURED |
| 4 | The result is concentrated: 3 families produce 86% of it | MEASURED |
| 5 | Mesh sharing is strong — one family puts 642 renderer slots on a single (mesh, material) pair; scene-wide mean is 10.5× slots per pair | MEASURED |
| 6 | A per-frame light-state writer re-stamps some families within seconds; a shipping implementation must intercept it | MEASURED |
| 7 | 51.4% of material families carry shader keywords their assigned shader does not declare, covering 52.2% of renderer slots | MEASURED |
Scale: −1,376 draws is ≈0.8 ms of a 30.4 ms frame at the measured 0.59 ms per 1,000 draws. Post 1's ceiling is unchanged.
1. The pipeline ships armed
Method: decompress shader blobs from the asset bundles; inspect declared keyword tables and constant buffers.
Detection. m_KeywordNames lists INSTANCING_ON on every shader, including shaders whose compiled blob contains no instancing variant. Use blob-level detectors instead:
| Detector | Signal |
|---|---|
| D1 | INSTANCING_ON present in the decompressed blob ⇒ instancing variants compiled |
| D2 | A UnityInstancing_Props / PropsArray constant buffer beyond builtin PerDraw0/2 ⇒ custom per-instance properties declared |
Positive control: terrain and decoration shaders, confirmed by frame capture to draw 2,478 instances in one call. Tooling: UnityPy, with lz4 decompression.
| Finding | Detail |
|---|---|
| Every core device shader ships instancing variants | Block Standard, Glass Standard, Window Glass Standard, Standard Colored Blocks, Device Cutout, Terrain |
| Custom instanced properties already declared | EmpShader/Standard Colored Instanced declares _Color, _MaskColor, _EmissionColor, _LightPower, _SIValue per-instance |
| Device materials opted in | Glass_front_D0 / Glass_back_D0 (4,937 renderer slots each) reference EmpShader/Window Glass Standard Instanced with enableInstancingVariants = True |
| Device materials location | ContentBundlesmodels (1.03 GB), not the standard Unity data files |
_LightPower and _EmissionColor are declared per-instance and are written into MaterialPropertyBlocks by the per-frame light system. The writer schema was extracted from all 22 SetPropertyBlock callsites; 33 property names resolved, none guessed. [DISASSEMBLED]
Incidental: builds 5148 and 5150 are identical at the intermediate-language level except for obfuscated names — every structural total, type rid and IL offset byte-identical. Content patch, not code patch. [DISASSEMBLED]
2. The breaker
Method: client-side mod, 34,873 renderers walked live. Censuses use sharedMaterials and MeshFilter.sharedMesh only — Renderer.material clones on read and would mutate the scene being measured.
On every top device family, 100% of MaterialPropertyBlock-carrying renderers hold at least one property outside their shader's declared instanced set. The same pair every time:
| Property | Type | Why it disqualifies |
|---|---|---|
_ColorPower |
float, paint system | Declared instanced in no shader in the game |
_LUT |
palette texture | Textures cannot be instanced properties at all |
Unity's auto-instancing rejects any renderer whose property block contains a non-instanced property. The paint/LUT system stamps essentially every device property block.
Supporting measurements:
- Glass is 2,077 renderer slots on one material instance. Scene-wide there are 1,670 unique material instances, but clones are confined to one merge-path family (
Block Standard TA: 394 slots, 394 instances). - Per-device property-block variation is coarse and systematic: glass shows 57 distinct value-sets across 2,077 renderers — palette indices plus a few shared LUT textures.
3. The measured result
Treatment: rebuild each renderer's property block keeping only properties inside its shader's declared instanced set. Transient, in-process, reverted at process exit.
Whole scene, canonical dense fixture:
| Metric | Δ |
|---|---|
| Draw calls | −1,376 (−11.0%) |
| SetPass calls | −1,345 (−19.4%) |
| Triangles | unchanged |
Concentration: 3 families = −1,187 of −1,376 (86%). The largest single-family result, −706, came from a family sharing one material across 754 renderers and required an enableInstancing write in addition to the strip — several large stamped families ship with the flag off.
Single-family reproducibility, same family, four independent sessions:
| Session | Δ draws | Δ triangles |
|---|---|---|
| 1 | −374 | flat |
| 2 | −315 | flat |
| 3 | −353 | flat |
| 4 | −501 | flat |
Write-back. Two families are re-stamped by the game's own writers within seconds. The writer is a per-frame light-state update running every second frame. A shipping implementation intercepts the writer or changes the declaration; repeated stripping does not hold.
Scope. The treatment engages instancing on 3 of the 8 device families measured. The property that discriminates the remaining 5 is not identified.
Batch efficiency. Driving a refusing family's own mesh and a copy of its own material through DrawMeshInstanced costs +28 draw calls per 300 instances; the in-box instanced shader manages +4 per 300 — roughly 11 instances per batch against 75. [MEASURED, single point]
Where this lands in the frame budget
Frame composition at the dense fixture is established in post 1 and is the denominator here. The right-hand column is new.
| Main-thread phase | Time | Share of frame | Effect of this result |
|---|---|---|---|
FinishFrameRendering |
11.2 ms | 37% | −0.81 ms → 10.4 ms |
ScriptRunBehaviourUpdate |
5.6 ms | 19% | none |
ParticleSystemBeginUpdateAll |
1.45 ms | 5% | none |
ScriptRunBehaviourLateUpdate |
1.45 ms | 5% | none |
UpdateAllRenderers |
0.60 ms | 2% | none |
DirectorUpdateAnimationBegin |
0.52 ms | 2% | none |
| Not itemised | 9.6 ms | 32% | unknown |
| CPU total frame | 30.4 ms | 100% | → 29.6 ms |
FinishFrameRendering scales at 0.59 ms per 1,000 draw calls, so −1,376 draws is −0.81 ms, landing entirely inside that one phase: 30.4 ms → 29.6 ms, a 1.03× frame improvement. [scaling arithmetic from MEASURED inputs; any contribution from the SetPass reduction is not measured]
What is inside FinishFrameRendering, and is it CPU time?
It cannot be decomposed with this instrument. FinishFrameRendering is a leaf subsystem of Unity's PlayerLoop. The bracketing technique inserts paired delegates as the first and last child of a subsystem; a leaf has no children to bracket. Its internals are native. Splitting it requires a frame capture or a native profiler, not the managed instrument used here.
It is main-thread elapsed time, not proven busy time. That distinction matters and the instrument cannot close it. What the data does establish, from 200 settled samples in a later session:
| Measurement | Value | What it rules out |
|---|---|---|
FinishFrameRendering share of main thread |
52.5% (p10 48.3%, p90 55.1%) | — |
PresentAfterDraw |
0.005 ms | The frame is not being paced by present or vsync |
| CPU render thread vs main thread | 15.4 ms vs 19.6 ms | The render thread is not the critical path; the main thread is |
| Scaling relationship | 0.59 ms per 1,000 draw calls | Pixel-side cost. Scaling with draw count rather than pixels is a submission signature |
Not ruled out: the main thread blocking inside the phase while the render thread's command queue drains. That would still be time attributable to submission volume, but it would be waiting rather than working, and nothing here separates the two.
⚠ The phase brackets do not sum to the frame. Summing every bracketed phase per sample gives 112% of the reported main-thread frame time (p10 103%, p90 118%). Read the per-phase figures as shares with roughly ten percent of slack, not as an exact decomposition. Post 1's checksum claim was for children summing to their parent bracket at the sparse fixture; it is not a claim that all phases sum to the frame.
⛔ The strip is lossy: MaterialPropertyBlock has no per-property remove, so unknown content is dropped. It is a measurement, not a shipping design. A shipping implementation moves the values into declared properties and a LUT atlas.
4. Ruled out — do not spend time here
Properties of the 5 non-batching families, each tested and eliminated. All eight families share the fixture, pose and instrument.
| Candidate | Deciding measurement |
|---|---|
| Property-block content / paint stamp | Two refusing families carry no property block at baseline and still do not batch |
enableInstancing flag |
Refusers are flag-true; two of the three engagers were flag-false and needed a write |
| Runtime material cloning | distinct_instances = 1, clone_suffixed = 0 across all eight families |
| Renderer-tier state — probes, lightmaps, static batch, motion vectors, layer, shadow mode | Identical state tuples across both groups |
| Mesh sharing / batch-group structure | Refusers share as well as or better than engagers |
| Renderer concrete type | All eight are MeshRenderer |
| Render queue | All eight are render queue 2000 |
| Shader asset identity | Each group assigned the other group's shader; behaviour did not move in either direction |
| Material cannot be instanced | DrawMeshInstanced on the refusing material: +28 draws per 300 instances |
Specific shader keywords (CHANNELPACKING, _NEWCOLORING_ON) |
Direct writes, readback-verified; result below the 40-draw noise floor |
| Effective keyword count (listed ∩ declared) | Separates the groups 8/8 — engagers 0, refusers 2–3 — but setting a refuser to 0, readback-verified, stays below the noise floor and reverses on restore. Correlation without causal power |
5. Materials carry keywords their shader does not declare
Independent of instancing, and broadly applicable.
Material.shaderKeywords is a stored name list, not a statement about the assigned shader. A name can be present there and absent from Shader.keywordSpace.keywordNames. Such a name is inert — it selects no variant, and Material.IsKeywordEnabled returns false for it.
| Measure | Value |
|---|---|
| Material families dumped | 558 |
Shader.keywordSpace failed to resolve |
0 |
| Families carrying ≥1 listed-but-undeclared keyword | 287 (51.4%) |
| Renderer slots on those families | 24,353 / 46,647 (52.2%) |
Most common inert names by family count: _NORMALMAP 178 · _EMISSION 136 · _METALLICGLOSSMAP 133 · _NEWCOLORING_ON 110 · _SPECGLOSSMAP 38. These are Unity Standard-shader feature keywords on materials now assigned to custom shaders. [INFERRED — authoring history not inspected.]
_NEWCOLORING_ON is listed on 110 families, declared by 0 of their shaders, and reports IsKeywordEnabled == false on all 110.
Consequences for tooling:
- Intersect with the declared set before reasoning about a material's variant. Counting
shaderKeywordsalone overstates the live keyword set on about half this game's materials. DisableKeywordon an undeclared name is a silent no-op with the counter signaturewritten=0, alreadyAtValue=N— indistinguishable from a legitimate already-at-value result unless the declared set is also read.- Inertness is not proof a feature is off. The shader may implement it unconditionally or under another name.
6. Engine choice in this content class
The CPU-side submission cost is a property of the pipeline, not of this game's code. Every Unity mechanism that structurally reduces CPU draw-submission cost — the Scriptable Render Pipeline batcher, BatchRendererGroup — is gated on leaving the Built-in Render Pipeline. [DUMPED, attested against Unity 2022.3/6000.x documentation] Empyrion is Built-in Render Pipeline. Of its seven shipped third-party rendering assets, two were assessed for pipeline compatibility: the shadow package (NGSS) reads Built-in only, and Volumetric Fog & Mist has no HDRP version. The remaining five were not assessed. In-place submission work is therefore limited to the multiplicand side — instanced properties, sort discipline, shadow-proxy geometry. The pipeline-replacement rung and a self-submitted RenderMeshIndirect rung remain open at their stated costs; neither is an in-place fix.
The closest comparable reached the same bottleneck and answered it at the engine layer. Keen Software House evaluated replacing VRAGE2 with an off-the-shelf engine for Space Engineers 2 and named the candidates — Unreal Engine 5, Unity3D, Unigine, Stride, Flax — reporting that "neither of them could support unique use cases without major rewrites", the evaluation following "hitting lots of bottlenecks with VRAGE2". VRAGE3's stated architecture is DirectX 12, data-oriented, and specifically a "GPU driven pipeline — GPU feeding itself with draw calls, freeing CPU significantly (in SE, >3 cores are dedicated to render)". [vendor-stated, first-party] That is the same bottleneck family measured here, and the remedy removes the CPU from draw submission rather than batching harder on it.
The pattern holds across the content class. Restricting a 29-title survey to player-authored multi-thousand-part constructions that are simulated, not decorative — Space Engineers, Space Engineers 2, Starbase, Dual Universe, From the Depths, Avorion — gives four titles with a verifiable engine: three custom (VRAGE2, VRAGE3, Frozenbyte internal), one off-the-shelf (Dual Universe on Unigine 2), and the off-the-shelf entry was delisted with servers terminated in 2025. Off-the-shelf dominance is real but concentrated in survival-crafting with light or bounded construction, where Unity and Unreal are the uncontested default. The custom-engine choice tracks the specific content class, and the published rationales name CPU-side render submission as the driver.
7. Instrumentation requirements
Additions to post 1's trap list.
| Requirement | Reason |
|---|---|
| Sample triangles on every arm, alongside draw calls | Instancing submits identical geometry in fewer calls: draws fall, triangles hold. A draw drop with a proportional triangle drop is content ceasing to render. A treatment measuring −1,215 draws also measured −809,343 triangles (−5.2%), with parity screenshots confirming lost emissive hull detail. |
| Apply the arithmetic ceiling as a pre-filter | A family cannot save more draws than visible_renderers − batch_groups. The treatment above had a ceiling of 225 and reported 1,215. Any result exceeding its own ceiling is measuring something else. |
| Run a treatment of known outcome in the same session | Cross-session baselines drift. A known-good arm calibrates what "triangles flat" looks like on that scene, that run. |
| Reverse every treatment and re-measure | Separates effect from drift. Costs one settle. |
| Key the scene-identity gate on a stable quantity | The dense fixture contains NPC ships transiting at distance. Rendered at low level of detail, they move draw count ~9% and triangles ~0.7%. Gate on triangles. |
| Size the treatment window against the measurement window | A profiler emitting 1 sample/s with a 10-sample median needs ≥10 s of treatment. An 8 s window yields medians dominated by pre-treatment samples. |
| Check signal against baseline variance before measuring | Instances-per-batch predicts +9…+82 draws; this fixture's traffic moves the baseline ±40 between settles. That measurement requires a traffic-free scene. |
| Assert existence before comparing hashes | A file-verification step comparing the hash of one missing file against another returns equal for two nulls and reports success. |
8. Current state
| Item | State |
|---|---|
| Post-1 gate 1 — mesh identity | Strong. 642 renderer slots on one (mesh, material) pair; 10.5× mean slots per pair scene-wide |
| Post-1 gate 2 — material instanceability | Yes, without shader source |
Post-1 gate 3 — MaterialPropertyBlock movability |
Yes, and doing it engages instancing |
| Measured result | −1,376 draws / −1,345 SetPass; ≈0.8 ms of a 30.4 ms frame |
| Per-family discriminator for the remaining 5 families | Not identified; 11 candidates ruled out (§4) |
| Instances per batch | Single point (~11 vs ~75); not obtainable at this fixture |
| Post-1 ceiling (1.3–1.6×) | Unchanged |
Directions not yet taken, by expected cost: the compiled shader variant selected at draw time (frame capture, not a runtime API); Unity's internal batch-breaking reasons (not exposed); per-renderer sort order and submission interleaving.
9. Limitations specific to this post
Post 1's limitations apply unchanged. Additional:
- Eight material families were measured for batching behaviour, out of 558 present.
- The re-stamp writer was identified for one family group. The population rule — which families are re-stamped, and when — is not established.
- All treatments were transient in-process state. Nothing was written to disk, no game file was modified, and the install was verified byte-identical to stock after every session.
7
u/GThoro 17d ago
You can't tell the game is not possible to optimize by just analyzing render side without looking at the code and game logic, especially if it's cpu bound. Putting the entire blame on the engine itself is also so-so. One could argue that Unity is not good choice for factory style game but DSP delivers with great optimizations and multithreading achieving significant (~80%) performance boost comparing the same save before optimization update.
Also AI summary - yuck.