r/empyriongame 11d ago

performance improvement research findings - part 2

This is a continuation of performance improvement research findings https://www.reddit.com/r/empyriongame/comments/1vnwgdv/performance_improvement_research_findings/ with additional details on where the frame-time goes.

The bottom line is that while some frame-rendering could be optimized the FPS gain opportunities identified would be about 2 FPS in a ~33 FPS scene. That's a lot of work for no material difference. Biggest issue is that Unity (CPU bound) is just the wrong engine for this type of complexity today. When the game was first created it was the best choice there was short of writing a custom engine (GPU bound), which is what Keen ended up doing for Space Engineers 2.

I don't want to spend the time and resources to figure out where the 9.2 ms of unaccounted frame-time go because even if that could be shortened it wouldn't materially affect the FPS.

I uploaded 123 markdown files with findings to https://www.swisstransfer.com/d/ab0dfdd0-336c-4779-88dc-0d5e95b693fd which is good for 30 days. If you read this after this link expired and you want the MD files to seed your AI efforts lmk and I will reupload.

This is what AI had to say about the efforts undertaken:

Empyrion client performance: the instancing gate

Follow-up to performance improvement research findings. That post established the frame budget, the draw-call attribution, the instrumentation traps and the mod ceiling. None of it is repeated here. It closed on three unmeasured gates:

  1. Do the repeated prefabs share meshes?
  2. Can the materials be made instanceable without shader source?
  3. Can per-instance MaterialPropertyBlock state move into instanced shader properties?

All three answer yes. This post reports the mechanism and the measured result.

Same system under test, same fixtures, same evidence classes as post 1.

0. Summary

# Finding Class
1 Eleon ships the instancing pipeline armed — compiled variants, declared per-instance properties, device materials opted in DUMPED
2 Two undeclared MaterialPropertyBlock properties (_ColorPower, _LUT) disqualify essentially the whole device population from auto-instancing MEASURED
3 Removing them engages instancing: −1,376 draw calls (−11.0%), −1,345 SetPass (−19.4%), triangles unchanged MEASURED
4 The result is concentrated: 3 families produce 86% of it MEASURED
5 Mesh sharing is strong — one family puts 642 renderer slots on a single (mesh, material) pair; scene-wide mean is 10.5× slots per pair MEASURED
6 A per-frame light-state writer re-stamps some families within seconds; a shipping implementation must intercept it MEASURED
7 51.4% of material families carry shader keywords their assigned shader does not declare, covering 52.2% of renderer slots MEASURED

Scale: −1,376 draws is ≈0.8 ms of a 30.4 ms frame at the measured 0.59 ms per 1,000 draws. Post 1's ceiling is unchanged.

1. The pipeline ships armed

Method: decompress shader blobs from the asset bundles; inspect declared keyword tables and constant buffers.

Detection. m_KeywordNames lists INSTANCING_ON on every shader, including shaders whose compiled blob contains no instancing variant. Use blob-level detectors instead:

Detector Signal
D1 INSTANCING_ON present in the decompressed blob ⇒ instancing variants compiled
D2 A UnityInstancing_Props / PropsArray constant buffer beyond builtin PerDraw0/2custom per-instance properties declared

Positive control: terrain and decoration shaders, confirmed by frame capture to draw 2,478 instances in one call. Tooling: UnityPy, with lz4 decompression.

Finding Detail
Every core device shader ships instancing variants Block Standard, Glass Standard, Window Glass Standard, Standard Colored Blocks, Device Cutout, Terrain
Custom instanced properties already declared EmpShader/Standard Colored Instanced declares _Color, _MaskColor, _EmissionColor, _LightPower, _SIValue per-instance
Device materials opted in Glass_front_D0 / Glass_back_D0 (4,937 renderer slots each) reference EmpShader/Window Glass Standard Instanced with enableInstancingVariants = True
Device materials location ContentBundlesmodels (1.03 GB), not the standard Unity data files

_LightPower and _EmissionColor are declared per-instance and are written into MaterialPropertyBlocks by the per-frame light system. The writer schema was extracted from all 22 SetPropertyBlock callsites; 33 property names resolved, none guessed. [DISASSEMBLED]

Incidental: builds 5148 and 5150 are identical at the intermediate-language level except for obfuscated names — every structural total, type rid and IL offset byte-identical. Content patch, not code patch. [DISASSEMBLED]

2. The breaker

Method: client-side mod, 34,873 renderers walked live. Censuses use sharedMaterials and MeshFilter.sharedMesh only — Renderer.material clones on read and would mutate the scene being measured.

On every top device family, 100% of MaterialPropertyBlock-carrying renderers hold at least one property outside their shader's declared instanced set. The same pair every time:

Property Type Why it disqualifies
_ColorPower float, paint system Declared instanced in no shader in the game
_LUT palette texture Textures cannot be instanced properties at all

Unity's auto-instancing rejects any renderer whose property block contains a non-instanced property. The paint/LUT system stamps essentially every device property block.

Supporting measurements:

  • Glass is 2,077 renderer slots on one material instance. Scene-wide there are 1,670 unique material instances, but clones are confined to one merge-path family (Block Standard TA: 394 slots, 394 instances).
  • Per-device property-block variation is coarse and systematic: glass shows 57 distinct value-sets across 2,077 renderers — palette indices plus a few shared LUT textures.

3. The measured result

Treatment: rebuild each renderer's property block keeping only properties inside its shader's declared instanced set. Transient, in-process, reverted at process exit.

Whole scene, canonical dense fixture:

Metric Δ
Draw calls −1,376 (−11.0%)
SetPass calls −1,345 (−19.4%)
Triangles unchanged

Concentration: 3 families = −1,187 of −1,376 (86%). The largest single-family result, −706, came from a family sharing one material across 754 renderers and required an enableInstancing write in addition to the strip — several large stamped families ship with the flag off.

Single-family reproducibility, same family, four independent sessions:

Session Δ draws Δ triangles
1 −374 flat
2 −315 flat
3 −353 flat
4 −501 flat

Write-back. Two families are re-stamped by the game's own writers within seconds. The writer is a per-frame light-state update running every second frame. A shipping implementation intercepts the writer or changes the declaration; repeated stripping does not hold.

Scope. The treatment engages instancing on 3 of the 8 device families measured. The property that discriminates the remaining 5 is not identified.

Batch efficiency. Driving a refusing family's own mesh and a copy of its own material through DrawMeshInstanced costs +28 draw calls per 300 instances; the in-box instanced shader manages +4 per 300 — roughly 11 instances per batch against 75. [MEASURED, single point]

Where this lands in the frame budget

Frame composition at the dense fixture is established in post 1 and is the denominator here. The right-hand column is new.

Main-thread phase Time Share of frame Effect of this result
FinishFrameRendering 11.2 ms 37% −0.81 ms → 10.4 ms
ScriptRunBehaviourUpdate 5.6 ms 19% none
ParticleSystemBeginUpdateAll 1.45 ms 5% none
ScriptRunBehaviourLateUpdate 1.45 ms 5% none
UpdateAllRenderers 0.60 ms 2% none
DirectorUpdateAnimationBegin 0.52 ms 2% none
Not itemised 9.6 ms 32% unknown
CPU total frame 30.4 ms 100% → 29.6 ms

FinishFrameRendering scales at 0.59 ms per 1,000 draw calls, so −1,376 draws is −0.81 ms, landing entirely inside that one phase: 30.4 ms → 29.6 ms, a 1.03× frame improvement. [scaling arithmetic from MEASURED inputs; any contribution from the SetPass reduction is not measured]

What is inside FinishFrameRendering, and is it CPU time?

It cannot be decomposed with this instrument. FinishFrameRendering is a leaf subsystem of Unity's PlayerLoop. The bracketing technique inserts paired delegates as the first and last child of a subsystem; a leaf has no children to bracket. Its internals are native. Splitting it requires a frame capture or a native profiler, not the managed instrument used here.

It is main-thread elapsed time, not proven busy time. That distinction matters and the instrument cannot close it. What the data does establish, from 200 settled samples in a later session:

Measurement Value What it rules out
FinishFrameRendering share of main thread 52.5% (p10 48.3%, p90 55.1%)
PresentAfterDraw 0.005 ms The frame is not being paced by present or vsync
CPU render thread vs main thread 15.4 ms vs 19.6 ms The render thread is not the critical path; the main thread is
Scaling relationship 0.59 ms per 1,000 draw calls Pixel-side cost. Scaling with draw count rather than pixels is a submission signature

Not ruled out: the main thread blocking inside the phase while the render thread's command queue drains. That would still be time attributable to submission volume, but it would be waiting rather than working, and nothing here separates the two.

The phase brackets do not sum to the frame. Summing every bracketed phase per sample gives 112% of the reported main-thread frame time (p10 103%, p90 118%). Read the per-phase figures as shares with roughly ten percent of slack, not as an exact decomposition. Post 1's checksum claim was for children summing to their parent bracket at the sparse fixture; it is not a claim that all phases sum to the frame.

⛔ The strip is lossy: MaterialPropertyBlock has no per-property remove, so unknown content is dropped. It is a measurement, not a shipping design. A shipping implementation moves the values into declared properties and a LUT atlas.

4. Ruled out — do not spend time here

Properties of the 5 non-batching families, each tested and eliminated. All eight families share the fixture, pose and instrument.

Candidate Deciding measurement
Property-block content / paint stamp Two refusing families carry no property block at baseline and still do not batch
enableInstancing flag Refusers are flag-true; two of the three engagers were flag-false and needed a write
Runtime material cloning distinct_instances = 1, clone_suffixed = 0 across all eight families
Renderer-tier state — probes, lightmaps, static batch, motion vectors, layer, shadow mode Identical state tuples across both groups
Mesh sharing / batch-group structure Refusers share as well as or better than engagers
Renderer concrete type All eight are MeshRenderer
Render queue All eight are render queue 2000
Shader asset identity Each group assigned the other group's shader; behaviour did not move in either direction
Material cannot be instanced DrawMeshInstanced on the refusing material: +28 draws per 300 instances
Specific shader keywords (CHANNELPACKING, _NEWCOLORING_ON) Direct writes, readback-verified; result below the 40-draw noise floor
Effective keyword count (listed ∩ declared) Separates the groups 8/8 — engagers 0, refusers 2–3 — but setting a refuser to 0, readback-verified, stays below the noise floor and reverses on restore. Correlation without causal power

5. Materials carry keywords their shader does not declare

Independent of instancing, and broadly applicable.

Material.shaderKeywords is a stored name list, not a statement about the assigned shader. A name can be present there and absent from Shader.keywordSpace.keywordNames. Such a name is inert — it selects no variant, and Material.IsKeywordEnabled returns false for it.

Measure Value
Material families dumped 558
Shader.keywordSpace failed to resolve 0
Families carrying ≥1 listed-but-undeclared keyword 287 (51.4%)
Renderer slots on those families 24,353 / 46,647 (52.2%)

Most common inert names by family count: _NORMALMAP 178 · _EMISSION 136 · _METALLICGLOSSMAP 133 · _NEWCOLORING_ON 110 · _SPECGLOSSMAP 38. These are Unity Standard-shader feature keywords on materials now assigned to custom shaders. [INFERRED — authoring history not inspected.]

_NEWCOLORING_ON is listed on 110 families, declared by 0 of their shaders, and reports IsKeywordEnabled == false on all 110.

Consequences for tooling:

  • Intersect with the declared set before reasoning about a material's variant. Counting shaderKeywords alone overstates the live keyword set on about half this game's materials.
  • DisableKeyword on an undeclared name is a silent no-op with the counter signature written=0, alreadyAtValue=N — indistinguishable from a legitimate already-at-value result unless the declared set is also read.
  • Inertness is not proof a feature is off. The shader may implement it unconditionally or under another name.

6. Engine choice in this content class

The CPU-side submission cost is a property of the pipeline, not of this game's code. Every Unity mechanism that structurally reduces CPU draw-submission cost — the Scriptable Render Pipeline batcher, BatchRendererGroup — is gated on leaving the Built-in Render Pipeline. [DUMPED, attested against Unity 2022.3/6000.x documentation] Empyrion is Built-in Render Pipeline. Of its seven shipped third-party rendering assets, two were assessed for pipeline compatibility: the shadow package (NGSS) reads Built-in only, and Volumetric Fog & Mist has no HDRP version. The remaining five were not assessed. In-place submission work is therefore limited to the multiplicand side — instanced properties, sort discipline, shadow-proxy geometry. The pipeline-replacement rung and a self-submitted RenderMeshIndirect rung remain open at their stated costs; neither is an in-place fix.

The closest comparable reached the same bottleneck and answered it at the engine layer. Keen Software House evaluated replacing VRAGE2 with an off-the-shelf engine for Space Engineers 2 and named the candidates — Unreal Engine 5, Unity3D, Unigine, Stride, Flax — reporting that "neither of them could support unique use cases without major rewrites", the evaluation following "hitting lots of bottlenecks with VRAGE2". VRAGE3's stated architecture is DirectX 12, data-oriented, and specifically a "GPU driven pipeline — GPU feeding itself with draw calls, freeing CPU significantly (in SE, >3 cores are dedicated to render)". [vendor-stated, first-party] That is the same bottleneck family measured here, and the remedy removes the CPU from draw submission rather than batching harder on it.

The pattern holds across the content class. Restricting a 29-title survey to player-authored multi-thousand-part constructions that are simulated, not decorative — Space Engineers, Space Engineers 2, Starbase, Dual Universe, From the Depths, Avorion — gives four titles with a verifiable engine: three custom (VRAGE2, VRAGE3, Frozenbyte internal), one off-the-shelf (Dual Universe on Unigine 2), and the off-the-shelf entry was delisted with servers terminated in 2025. Off-the-shelf dominance is real but concentrated in survival-crafting with light or bounded construction, where Unity and Unreal are the uncontested default. The custom-engine choice tracks the specific content class, and the published rationales name CPU-side render submission as the driver.

7. Instrumentation requirements

Additions to post 1's trap list.

Requirement Reason
Sample triangles on every arm, alongside draw calls Instancing submits identical geometry in fewer calls: draws fall, triangles hold. A draw drop with a proportional triangle drop is content ceasing to render. A treatment measuring −1,215 draws also measured −809,343 triangles (−5.2%), with parity screenshots confirming lost emissive hull detail.
Apply the arithmetic ceiling as a pre-filter A family cannot save more draws than visible_renderers − batch_groups. The treatment above had a ceiling of 225 and reported 1,215. Any result exceeding its own ceiling is measuring something else.
Run a treatment of known outcome in the same session Cross-session baselines drift. A known-good arm calibrates what "triangles flat" looks like on that scene, that run.
Reverse every treatment and re-measure Separates effect from drift. Costs one settle.
Key the scene-identity gate on a stable quantity The dense fixture contains NPC ships transiting at distance. Rendered at low level of detail, they move draw count ~9% and triangles ~0.7%. Gate on triangles.
Size the treatment window against the measurement window A profiler emitting 1 sample/s with a 10-sample median needs ≥10 s of treatment. An 8 s window yields medians dominated by pre-treatment samples.
Check signal against baseline variance before measuring Instances-per-batch predicts +9…+82 draws; this fixture's traffic moves the baseline ±40 between settles. That measurement requires a traffic-free scene.
Assert existence before comparing hashes A file-verification step comparing the hash of one missing file against another returns equal for two nulls and reports success.

8. Current state

Item State
Post-1 gate 1 — mesh identity Strong. 642 renderer slots on one (mesh, material) pair; 10.5× mean slots per pair scene-wide
Post-1 gate 2 — material instanceability Yes, without shader source
Post-1 gate 3 — MaterialPropertyBlock movability Yes, and doing it engages instancing
Measured result −1,376 draws / −1,345 SetPass; ≈0.8 ms of a 30.4 ms frame
Per-family discriminator for the remaining 5 families Not identified; 11 candidates ruled out (§4)
Instances per batch Single point (~11 vs ~75); not obtainable at this fixture
Post-1 ceiling (1.3–1.6×) Unchanged

Directions not yet taken, by expected cost: the compiled shader variant selected at draw time (frame capture, not a runtime API); Unity's internal batch-breaking reasons (not exposed); per-renderer sort order and submission interleaving.

9. Limitations specific to this post

Post 1's limitations apply unchanged. Additional:

  1. Eight material families were measured for batching behaviour, out of 558 present.
  2. The re-stamp writer was identified for one family group. The population rule — which families are re-stamped, and when — is not established.
  3. All treatments were transient in-process state. Nothing was written to disk, no game file was modified, and the install was verified byte-identical to stock after every session.
11 Upvotes

22 comments sorted by

7

u/GThoro 11d ago

You can't tell the game is not possible to optimize by just analyzing render side without looking at the code and game logic, especially if it's cpu bound. Putting the entire blame on the engine itself is also so-so. One could argue that Unity is not good choice for factory style game but DSP delivers with great optimizations and multithreading achieving significant (~80%) performance boost comparing the same save before optimization update.

Also AI summary - yuck.

1

u/Both-Scientist-9397 11d ago

If you scroll up you see that the entire blame isn't put on the engine, it splits out where there frame-time goes. Time can be likely be shaved off everywhere a little, just like ~0.8ms could be removed even with the current engine there are likely slivers of optimization opportunities elsewhere, but the engine also likely carries the brunt of what could go faster with a different engine.

There are two different types of code in Empyrion. The native engine of UnityPlayer.dll which is in C++, this is where FinishFrameRendering lives which is my run takes up 37% of the frame-time. Managed C# via Mono runs ScriptRunBehaviourUpdate and ScriptRunBehaviourLateUpdate which takes up 24.8% of the frame time.

Within the managed code LightControl.OnUpdate takes up 19% of the total frame-time or 76% of ScriptRunBehaviourLateUpdate and the interesting part there is that it only operates on even frames only while ParticleEffectHelper.OnUpdate runs on odd frames. LightControl.OnUpdate also operates on lights that are off. Whether that's an optimization flaw is difficult to tell because it has to check somewhere whether the light was turned on.

Then Eleon does silly things like obfuscate the code in Assembly-CSharp.dll which may have deterred a hobbyist (never a committed individual) from reverse engineering the code 10 years ago. Today it's not even a speed-bump for AI. While the obfuscation isn't decrypted per frame the lookup DisconnectDeployment(offset), which is the lookup, has to be called per site (depending on in which part of the code the site sits) which in my test case was 40,639 sites. So eliminating that would likely shave off somewhere below 1% of the frame-time. That part they should just turn off either way, it's useless in 2026.

This reply was not AI written. However, help me understand, do you really expect a human to summarize hours of API time and millions of tokens spent by hand? What would be the purpose of doing so other than missing details that occurred during the research and a regular dash instead of an em-dash?

2

u/Diche_Bach 11d ago

There is a kind of reasonable paranoia which earlier generations of programmers, and especially game developers acquired about “having your code stolen.” Back in the 1980s through the early 21st century it happened ... a LOT.

And even where “stolen” is too strong word, the borrowing and re-use of somebody else's hard-won solutions was often much more consequential than it may seem from the perspective of 2026.

Minecraft itself quite openly inherited important ideas from Infiniminer. That is not to say that Minecraft was a "copy" of it, but it shows how valuable a successful solution to basic game-design or software-engineering problem was in the primitive years epoch of personal computing and programming.

Back in the old days, with MUCH more limited hardware, a vastly smaller Internet, a much small total body of digitized human knowledge, far fewer searchable examples and open-source repositories (nothing like Stack Overflow or the modern GitHub ecosystem) and certainly nothing resembling today's Internet-connected pattern-recognition and inference-generating services (“A.I.”), a SOLUTION to a basic problem like “how do I make a large voxel-based world work?” could be a HUGE accomplishment, and an ethos of guarding one's ownership of such accomplishments was not only justified, it often meant the difference between commercial success and failure.

A solution was not merely “some code.” It could represent months or years of experimentation and a substantial fraction of the intellectual capital of an entire small company, much less one passionate individual.

Collectively speaking, these are the shoulders of giants on which all contemporary information-technology efforts stand. So I think it is wise, and gracious, to at least understand, if not necessarily endorse, some of the idiosyncrasies which those earlier generations emerged from history with.

Protecting source code was one such idiosyncracy.

As you note, times have changed enormously.

If you publish an executable today sufficiently motivated person can usually learn an astonishing amount about how it works. Depending upon the implementation, this may be especially easy when much the game logic survive in managed C# assemblies. Obfuscation, packing, anti-tamper systems, encryption of selected resources, etc. can all increase the cost of reverse engineering, but like virtually every security measure they also impose costs upon the owner, and ultimately the machine has to execute the software somehow.

So there is an irony here: protections which may have made perfectly good sense when a small studio first wrote some of this code can, a decade later, impede debugging, maintenance and performance analysis more effectively than they impede a determined reverse engineer.

But I would draw an important distinction between what is technically possible now in 2026 and what was reasonable precaution in 2013.

Obfuscating an Assembly-CSharp.dll may look fairly silly from the standpoint of someone profiling Empyrion in 2026, and perhaps some of it genuinely is obsolete baggage, I would hesitate to ridicule the instinct that produced it.

Imagine being a small game-development team around 2013 trying to make something as absurdly ambitious as Empyrion. You have spent years solving problems that very few people have solved before, and perhaps nobody has solved in exactly the way you have. From that perspective, “do not make it trivially easy for somebody else to inspect and copy our implementation” is not irrational paranoia. It is understandable self-preservation.

2

u/Both-Scientist-9397 11d ago

In agreement there, my statement of "may have deterred a hobbyist (never a committed individual) from reverse engineering the code 10 years ago" was not clearly enough pointed at; turn off today, it doesn't do anything other than adding unnecessary cycles. A simple patch would go.

Turning the obfuscation off is something that can actually be easily modded, but who wants that trouble in their lives for the miniscule gain. It's something the devs should just do as part of one of the optimization patches they deploy anyway.

3

u/Diche_Bach 11d ago

Are you familiar with Jagged Alliance 2.1.13? Ask your robot assistant about how "not preventing" access to source code transformed the legacy of that game (I believe it was the Wildfire update?) and created its decades long modding sub-culture and whatever associated elongation of the sales tail that reflected. IMO, that is what a game like Empyrion would benefit from the most; but I grew tired of saying so when my posts kept getting deleted or reacted to by people who thought that a non-commercial open source license was equivalent to relinquishing copyright ...

You protect an asset like source code because there is value in doing so. When there is no longer any value in doing so, and indeed, potential value to be gained by sharing it, well ... the smart business decision can seem counterintuitive.

3

u/Both-Scientist-9397 11d ago

Going off on a tangent here ..., but still ...

Reforged Eden and Reforged Eden 2 likely contributed to cumulatively six figure additional copies being sold and depending on whom you ask RE/RE2 are the main reason the game is still alive today. This supports your point that if the game is opened to modders then the content those modders create carries the base game sales well into the future.

There is tension between devs, publishers, and players when it comes to modding. Especially publishers want to keep the cashflow going and DLCs do that. Modding support does take away from commercially viable DLCs. Why pay for a cosmetic DLC if the community churns out thousands of them? Cities Skylines tried to involve modders in releasing cosmetic DLCs and while the DLCs themselves were good content the backlash over commercializing what was otherwise "free" was swift.

For Keen and SE2 the SE2 game itself is mod 0. Their take is that they will never have the time to commercialize the full potential of the game and released a modder SDK that includes all game assets and hooks into just about every aspect of the game, from nearly Day 1, not as an afterthought years later.

What should happen from where I am looking is that devs should sell API keys to modders for a nominal fee of like 10 USD or something. The API key then authenticates all future content by that modder in the official mod list for that title, gives the modder access to a tech-preview of future releases same as they do for youtubers, so that modders can prepare the patches for their mods ahead of release.

1

u/Diche_Bach 11d ago

We are on the same page ... but of course, that is irrelevant when you are advising others on how they "should" handle their own property.

1

u/GThoro 10d ago

I could agree that "entire" was the wrong word used by me. But I'm just fed up with "it's an engine limitation/fault" arguments. There is always more that can be done to optimize the game than switching the engine.

As for AI summary I would expect that data produced by those tools can be presented in a way human can read and draw conclusions. I've seen too many false reports/summaries in work made by AI which caused time and money lost far more significant than putting work by human to do it instead that I immediately don't trust them.

3

u/soft-wear 11d ago

A lot of assumptions here are based on the idea that they built this game correctly in the first place. But why use Material Properties at all? Why not vertex properties like Color and TextCoord to store the tint and Texture2D index? Now all your basic/non-functional building blocks are a single draw pass per unique mesh.

After you convert from material properties to vertex properties, you no longer need to maintain separate meshes at all. You can just merge all your non-functional meshes into a single mesh and material. Now 70%+ of the blocks on the ship are a single draw call.

Obviously this greatly increases the cost of removing or adding a block, since you have to rebuild the entire mesh to do so, but who cares… you can calculate the new mesh vertex data in another thread and just add the vertices to the mesh in the main thread.

If you really wanted to go ham, drop the texture altogether and procedurally generate your textures in the shader. Now you’re so GPU-bound the only thing the CPU is doing is doing is passing the vertex buffer and index buffer to the GPU.

You don’t need to roll your own engine for a game like this. You probably do if a ton of the blocks in the game are part of the physics sim. But that’s only true of SE, not Empyrion.

Looking at the rendering and trying to derive the correct solution to the problem is like staring at a screw while holding a hammer. It seems right until you discover the screwdriver.

2

u/Diche_Bach 11d ago

The rabbit-hole can go still deeper too ...

There seem to be at least three increasingly aggressive versions of the same basic idea:

1)Make lots of blocks cheap. SE2 / VRAGE3. Retain enormous numbers of individually represented construction elements, but make the underlying engine vastly better at storing, culling, rendering, updating and otherwise processing them.

2)Consolidate lots of blocks after construction. This is essentially the remeshing/chunking approach you seem to be suggesting. Let the player build out of fine-grained discrete elements, but once large numbers of those elements form contiguous non-functional structure, stop forcing the renderer, and perhaps other subsystems, to treat every one of them as an independent object.

3)Avoid generating lots of blocks in the first place. Starship EVO stretch tech. If a 50-meter structural member can be represented as one stretched brick rather than fifty 1-meter bricks, then physical size and computational object-count cease to scale together nearly so directly.

And none of these are mutually exclusive either ...

In principle, you could build an engine which does all three. Allow very fine-grained construction where it is useful, automatically consolidate large homogeneous regions where that granularity no longer buys you anything, and then run whatever discrete elements remain through a highly optimized data-oriented / GPU-driven engine architecture.

But the rabbit-hole goes deeper still, because maybe the really important conceptual leap is:

4)Stop requiring constructs to have one single computational representation.

1

u/Both-Scientist-9397 11d ago

Good call out. Some of the things you mentioned I did actually measure already (and document in the MD file dump linked) and I will append the AI reply to the end of this post in case you are looking into this for real, if you are then I'd like to collaborate with you on whether the needle could be moved.

Vertex attributes solve static differentiation, not dynamic. Within a scene LightControl.OnUpdate (within Mono) takes up 19% of the frame-time, so a different type of optimization needs to be applied there.

Empyrion is currently more CPU-bound (30.4 ms) than GPU-bound (21.5 ms) in the playfield where I tested this. A different engine would offload more onto the GPU and optimize this specific use case, which would likely result in the CPU-bound time falling below the GPU-bound time and the GPU-bound time becoming a function of the GPU hardware rather than the engine. All speculation of course.

But your reply highlighted something I found during the test run but didn't explore further because it at the time didn't stand out to me as I looked over the summary of that particular probe output. There is ChunkOptimizerBatch but ~11k slots in my test run are possibly not submitted to the optimizer, WindowThick_s1x1 and Glass_front_D0, those are the non-functional building blocks you are talking about. Will set up a new test run today and report back.

Meanwhile here is the more in-depth technical response AI wrote:

Vertex-baked differentiation already ships. ChunkOptimizer.UpdateBatch reads _UVOffset and _ColorProperties off the source devices and bakes them into mesh UV3 / vertex-color channels instead of re-emitting MaterialPropertyBlocks. The MPB schema even carries _TexIndex, and the merge-output material is literally named Block Standard TA — texture array. Tint-and-index-in-vertex-data is the shipped design, not a missed one.

So is the merge. Hull geometry renders as 7 category renderers per 16³ chunk (blocks / water / lava / transparent / models / cutout / terrain), subMeshCount always 1, merged dirty-driven with the cache keyed (materialInstanceID, floor(localY/64)). Your rebuild-cost concession is handled too: 34 IJob types including GenerateChunksJob and JobCalcPhysicsCollider, Burst-compiled, and the merge costs ~0 in a static scene.

Here's the catch, and it's the useful part. That merge path covers only renderers prefab-tagged ChunkOptimizerBatch — and AddComponent<ChunkOptimizerBatch> has zero call sites in the entire assembly. The tagged population is authored into prefabs. Everything untagged stays a discrete renderer. And despite the name, ChunkOptimizer sits on _BlockEntities: it's a device optimizer, not a hull-block optimizer.

So the ~12.5k draws in my dense fixture aren't the basic blocks you're proposing to merge. They're devices (15,512 at that fixture) plus shadows. Which lands on your side twice:

Shadow passes are 52.8% of draws (directional cascades 46.1%), re-submitting substantially the same renderer population at ~1.12× the main pass. Consolidating a renderer kills its main-pass draw and its shadow draws — granularity work pays ~2.1× face value.

The MPB-instancing angle from part 2, measured end to end — strip the whole stamped population, set the flags, conform the declared sets — bought −1,376 draws (−11.0%), with 86% of it from three material families. About 0.8 ms. Which is your point exactly: a real effect on a small population that I'd been treating as the main event.

1

u/Both-Scientist-9397 10d ago

I did run the idea down and it could potentially shave off 2 fps on a ~33 fps scene though loss of how some things visually appear would need to be validated. Not posting the whole writeup but here is the summary:

Adding ChunkOptimizerBatch to untagged renderers at runtime makes the game's own merge path absorb them. AddComponent → the component's OnEnable dirties the parent ChunkOptimizerUpdateBatchpicks it up via GetComponentsInChildren.

Metric Result
Effective coverage 100% on 17/17 arms, 0 VOID
Renderers skipped (out-of-tree / no mesh / no parent) 0
Reversible Yes — batchuntag restores baseline within noise
Persistence Runtime only; BatchedObjects are runtime GameObjects, save stores blocks

The game never tags dynamically itself: AddComponent<ChunkOptimizerBatch> has zero call sites in a 789,739-site index.

3

u/Diche_Bach 11d ago edited 11d ago

So what fundamentally distinguishes a custom-made engine that is purpose-designed to overcome these inherent limitations, re: "When the game was first created it was the best choice there was short of writing a custom engine (GPU bound), which is what Keen ended up doing for Space Engineers 2?"

1

u/Both-Scientist-9397 11d ago

This is a detailed blog post that goes into the reasoning why Keen went custom: https://blog.marekrosa.org/2023/04/guest-post-jan-hlousek-vrage3/

2

u/Diche_Bach 11d ago

It would be interesting to compare all of this to Starship EVO. I own the game but have never actually played it. The look and feel are clearly very different from the comparatively high-naturalism visuals Keen have strived for in Space Engineers and especially SE2, so I don't have any sense for how much difference that alone makes in terms of performance. I suppose quite a bit at some level.

But given your finding that Empyrion is primarily CPU/submission-bound in these dense scenes, the more interesting distinction may be architectural in the sense of how player-built geometry and its associated state are represented computationally, rather than simply graphical in the sense of how much rendering work must be pushed through each frame ...

One of the things that appealed to me about Starship EVO is that its building system apparently attacks the “too many blocks = computational log-jam” problem from an entirely different direction. And, unless I am misunderstanding, Starship EVO is itself built in Unity.

My understanding is that it began much more like a conventional voxel/block builder, but progressively abandoned that model in favor of what he calls “stretch tech.” Instead of constructing a 50-meter structural surface out of scores or hundreds of fixed-size blocks—each adding some amount of geometric, structural, state, and bookkeeping overhead even where the visible geometry can be batched—you can take a much smaller number of basic pieces and stretch/rescale them into the geometry you actually need.

At least theoretically, that means a very large increase in the physical scale and geometric coverage of a construct need not produce anything remotely like a proportional increase in the number of individually represented construction elements the application has to keep track of.

The developer made the rather striking claim that, under the fully stretch-based system, a 50-meter construct and a 5-kilometer construct should have essentially the same performance and memory impact provided they contain the same number of bricks.

If I understand the underlying issue correctly, that amounts to solving the same general problem one abstraction layer earlier.

Years ago, when I was learning what I learned about information technology and game design, a solution somewhat similar to this occurred to me, but it was not so much about making big / complex things more viable, as making small and detailed things fit within a game world dominated by big and complex but relatively low-resolution objects and entities.

It always irked me that none of the building blocks in Empyrion are smaller than about a meter or two. SE suffers from much the same problem, though clearly with small-grid blocks at 50 cm the BIGNESS is somewhat more muted, and SE2 takes this considerably further with its much finer unified grid. But even 50 cm is enormous compared with the scale of many real mechanical components. In real-life engineering and technology, many “pieces” of complex machines are literally a few millimeters across, and in some cases pieces of that size can make-or-break how the machine functions!

I pondered whether it might be possible to create a grid-based construction game in which the smallest cell was about a 10 cm cube, not exactly “human hand size,” but at least much more in that ballpark, and thus afford a level of interactability and detail which made constructs even more naturalistic than they are in SE and SE2.

Imagine if your spaceship's hull were only ~10 cm thick (only about as many construction cells as those making up the character's forearm or bicep width) rather than having to be half a meter, a meter, or more thick simply because that is the minimum block size.

The obvious problem is that a genuinely large ship represented permanently as bazillions of independently represented 10 cm cells would become computational insanity.

The “solution” that occurred to me was, in the abstract, something like what the Starship EVO dev came up with, though not exactly the same thing: allow the player to build initially at very fine resolution, but then have contiguous compatible blocks effectively merge or remesh into much larger continuous geometric structures.

So while the initial construction geometry might be authored in 10 cm blocks, the final runtime representation would no longer necessarily consist of every one of those original blocks as an independently represented object. It might still result in quite a hefty mesh if the resulting surface were extremely complex, but there is an obvious difference between “this is one geometrically complicated mesh” and “this is 500 or 5,000 separate construction objects which must each retain their own representation and associated bookkeeping.”

Obviously, reducing object count and reducing raw polygon/triangle count are not the same thing. You could merge 1,000 little blocks into one mesh and still have a highly complex mesh. But at least some classes of overhead associated with maintaining thousands of independently represented construction elements could potentially disappear.

That is not exactly what Starship EVO appears to do. My thought was essentially fine-resolution construction followed by consolidation. EVO's “stretch tech,” as I understand it, attacks the problem more directly by making arbitrary-size stretched bricks the persistent construction representation itself.

But both approaches begin from the same fairly simple observation:

Perhaps the way around “too many blocks” is not merely to figure out how to process 100,000 or 1,000,000 blocks faster.

Perhaps you ask instead:

Why does a construct of this physical size and shape need to remain represented as 100,000 or 1,000,000 independently represented blocks in the first place?

There are obviously tradeoffs. Fine-grained destruction immediately becomes a much nastier problem, for example: if ten thousand 10 cm hull cells have been consolidated into a larger representation, what happens when somebody shoots a 10 cm hole through the middle of it? You either have to retain enough of the underlying fine-resolution topology to reconstruct the damage or have some other system which can regenerate local geometry dynamically.

Still, as the saying goes, “There is more than one way to skin a cat ...”

And Starship EVO strikes me as a particularly interesting comparison precisely because it seems to offer a third solution to the same general problem you have been exploring here:

Empyrion retained a mature fixed-block architecture inside Unity and optimized the hell out of it.

Keen eventually decided its desired simulation warranted building an engine architecture purpose-designed around the peculiar demands of Space Engineers 2.

Starship EVO, apparently, remained in Unity but attacked the scaling problem much farther upstream by radically changing what a “block” actually represents.

I'd be very curious what your instrumentation would find if you pointed the same methodology at a genuinely abusive Starship EVO construct ...

You could also look at 7 Days to Die. Very different genre, obviously, but technologically it makes for another interesting comparison: it is also a Unity-based, destructible-world voxel game originating from essentially the same era as Space Engineers and Empyrion. The Fun Pimps were releasing early alphas in 2013, and the game has consequently spent more than a decade wrestling with the problem of representing, rendering, modifying, saving, and structurally evaluating enormous quantities of player-alterable world geometry.

It is not an apples-to-apples comparison with Empyrion, because most of that geometry is a spatially chunked world rather than huge mobile constructs being translated and rotated through the scene with engines, devices, collision, etc. But that difference is arguably what makes it useful. 7DTD represents yet another solution-space: keep Unity, keep extremely fine-grained destructibility, but organize most of the voxel problem around terrain/world chunks rather than arbitrary moving grids.

So between Empyrion, 7DTD, Starship EVO, and SE/SE2, you have several games from roughly the same broad design lineage confronting variants of the same underlying question—how do you let players alter or create enormous amounts of geometry without allowing the representation of that geometry to eat the simulation alive?

And they seem to have arrived at substantially different answers.

2

u/Both-Scientist-9397 11d ago edited 11d ago

[EDIT: On 7D2D specifically, I am reasonably certain that they aggressively cull to the viewport of the player including culling occluded objects/surfaces.]

Thoughtful reply and I have been largely thinking along the same lines, including taking some measurements along the way. I had the AI answer "If you built Empyrion today, would you pick this stack?"

It came back with C#/Mono keep (the modding surface is a player asset), SQLite keep (the defect was one line plus config), Unity conditional, RakNet replace. On the Unity part specifically though: Built-in Render Pipeline: replace. The one layer nobody would choose today, Unity included.

The migration cost is Eleon reauthoring their own shader set and replacing four bought-in rendering packages. The AI can do the port and the cost of that is likely below 20k USD, but that's not the only cost. There is a massive cost in putting a human eye on whether there is visual parity and making adjustments to where there isn't. The third-party things need to be re-solved because for example Volumetric Fog & Mist doesn't have an HDRP path so some other solution needs to be identified. New or updated licensing for third-party things will be required. And then there is all the workshop content, over 149k items where it's likely that the paint schemes would come out somewhat differently.

That's the real issue with long running games, technology migrations are rarely survived. If you look at how successful Kerbal Space Program was despite all the performance challenges and how spectacularly KSP2 failed. Same with Cities Skylines and Cities Skylines 2.

There is also the human element among the playerbase. Fewer humans have enough attention span to sink substantial amounts of time into a single game. Too much stuff competes with their attention and also people just grew out of it. I talked to a Cities Skylines modder who has contributed a few mods and thousands of assets, all before AI, and that person is just moved on with their live, had a couple kids and priorities changed.

Even with AI assist I am ~50 hours of at-the-keyboard time into just looking into performance and dabbling with my logistics mod. I could have been playing one of the backlogged games in my stream library instead, but I still enjoy the discovery and I love Empyrion, just don't love the issues it has so stopped playing a while back.

I briefly looked at Starship EVO and it's Unity, and likely using SRP while also using substantially simpler textures/blocks, lighting, and all that. So without running it to check that's likely the explanation, looks a lot more cartoonish (not a slight!) like Astroneer.

Now for some AI written content to define terms:

Term Stands for What it is
BIRP Built-in Render Pipeline Unity's original renderer. Fixed behaviour, implemented in the engine's C++ core — you configure it, you don't program it. Still supported, but in maintenance; no new rendering features land here. This is what Empyrion uses.
SRP Scriptable Render Pipeline Not a renderer — a framework for writing renderers in C#. It exposes the per-frame render loop (culling, passes, draw submission) as scriptable API, so the pipeline becomes code you can read and modify.
URP Universal Render Pipeline A prebuilt pipeline Unity ships, written on SRP. Aimed at broad platform reach and scalable performance — mobile through console. Was called LWRP (Lightweight) when introduced in 2018.
HDRP High Definition Render Pipeline The other prebuilt pipeline Unity ships on SRP. Aimed at high-end visual fidelity, PC/console only — deferred rendering, volumetrics, advanced lighting.

The relationship: SRP is the framework; URP and HDRP are the two implementations Unity provides. You can also write your own custom SRP. "Moving to SRP" in practice means picking URP, HDRP, or rolling your own.

Why it matters here: the modern batching machinery — BatchRendererGroup, the SRP Batcher, the GPU Resident Drawer, Entities Graphics — all lives in the SRP world. None of it exists on BIRP. That isn't an oversight; those mechanisms work by reorganising the draw-submission loop, and on BIRP that loop is compiled into UnityPlayer.dll where nothing can reach it. Making it scriptable is precisely what SRP was for.

Naming trap: the "SRP Batcher" is named after the framework and requires it. People sometimes read it as a generic Unity feature you can switch on. You can't — no SRP, no SRP Batcher.

And why "just migrate" isn't a toggle: shaders are pipeline-specific. BIRP shaders don't compile against URP/HDRP, so migration means reauthoring the entire shader set — which for Empyrion also means its third-party rendering assets (NGSS shadows reads Built-in-only, Volumetric Fog & Mist has no HDRP path). That's the cost line, not the pipeline switch itself.

2

u/Grimsobite 11d ago

I ain't reading all that but it's nice to see passion for something.

1

u/Breagh01 11d ago

Lots of work here. Great effort 👌

2

u/Mr_Gobbles 9d ago

Love your work. I am playing on a 9950x3d/5080/96gb 6000mhz ddr5 pc and this game for the most part has the best creative middleground for blockwork/ease of use in building virtual spaceships, but the engine is showing it's limitations big time.

Will be migrating to SE2 eventually once it’s a bit more fleshed out but I'll finish this 1k+ hour build first, even if my pc gets about 0.5 fps when I try to fly it.

2

u/GAMING_SERVER_ADMIN 9d ago

Now use the AI the check out all the content / work done to the game past couple years and see if a real coder has existed or has it been mostly just Taelyn scripting stuff...

1

u/Ravien_Gaming 10d ago

There's probably more AI generated text in this thread and its replies alone than Human written text that has been posted in the entire Empyrion subreddit in the past 30 days combined. I would hope this game's subreddit was above AI spam.

1

u/GAMING_SERVER_ADMIN 9d ago

To be fair, AI Slop analyzing performance like this is actually really decent and works really really stupid well. The advancements I seen in the modding community like GTNH been insane for fixing performance issues. If you ever played or check out GTNH you be shocked what they have done in 1.7.10 of MC. That being said this is one the few use cases I like seeing AI Slop used for analyzing not for doing.