Random crashes when playing our game

#1
Hi all

We are working on the Indie game, Aporia, and it is soon to be released on Steam. However, we still have on major issue. The game crashes at random points while playing in the launcher or the compiled Steam version of the game. We are optimizing all we can to see if that can get rid of the crash. We have quite big levels, but have divided the game in 3 different levels, for performance and loading times, while using layerswitching as well.

We suspect it is some sort of memory leak and we can guarantee that the crash is completely random. We used the built-in "demo recording" in the engine, without crashing and it crashed when replaying it.
When crashing, our virtual memory is around 5500mb up to 7900mb, which is quite alot. could this be it?

Here is a video of the crash happening. There is no connection between where the player is, what he is doing or where he is looking. it is completely random, however it usually happens after 1-2 hours of playing the game.

If you have suggestions, anything is very much welcome, since we are desperate to get rid of this crash and release a stable Cryengine game.
Attached is different errorlogs from different crashes and computers.
AporiaCrashes2.rar
(237.75 KiB) Downloaded 40 times
AporiaCrashes.rar
(118.08 KiB) Downloaded 45 times

Re: Random crashes when playing our game

#2
Hi,

Not sure we can say anything using these logs tbh. All I see is something trying to access a pointer that is for some reason -1. (hence AV ax FFFFFFFFF etc).
Use the debugger to catch exactly where in source code it is crashing, what pointer/variable it crashed using etc.
---------

I assume this crash is in profile x64?
cheers.

---

PS: For memory leaks there are many tools, but you can use the CRT Debugger to quickly check on "unclaimed" memory on shutdown/exit. Google will help with this.
Uniflare
CRYENGINE Technical Community Manager
Here to help the community and social channels grow and thrive.

My personal belongings;
Beginner Guides | My GitHub | Splash Plugin

Re: Random crashes when playing our game

#3
It seems that the issue lies in shadows/particles. Our game runs smooth when disabling those.

We are getting a number of warnings that makes the game crash::

13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Vegetation@Common_ZPassVS(%GRASS|%NORMAL_MAP|%TEMP_VEGETATION|%VERTCOLORS (%_RT_ALPHATEST|%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY)(0)(0)(0)(0)(VS)

<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Vegetation@Common_SG_VS(%TEMP_VEGETATION|%VERTCOLORS)(%_RT_ALPHATEST|%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY|%_RT_HW_PCF_COMPARE|%_RT_CUBEMAP0)(0)(0)(0)(0)(VS)
<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Vegetation@Common_SG_VS(%TEMP_VEGETATION|%VERTCOLORS)

(%_RT_ALPHATEST|%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY|%_RT_HW_PCF_COMPARE|%_RT_SAMPLE4)(0)(0)(0)(0)(VS)
<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Illum@Common_ZPassVS(%SUBSURFACE_SCATTERING)(%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY)(0)(100)(0)(0)(VS)

<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Illum@Common_ZPrepassVS(%SUBSURFACE_SCATTERING)(%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY)(0)(100)(0)(0)(VS)

<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Illum@Common_SG_VS(%SUBSURFACE_SCATTERING)(%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY|%_RT_HW_PCF_COMPARE|%_RT_SAMPLE4)(0)(100)(0)(0)(VS)

<13:22:49> [Warning] [SHADERS] GCM Global Cache Miss: <2>Illum@Common_SG_VS(%SUBSURFACE_SCATTERING)(%_RT_PER_INSTANCE_CB_TEMP|%_RT_NO_TESSELLATION|%_RT_OBJ_IDENTITY|%_RT_HW_PCF_COMPARE|%_RT_CUBEMAP0|%_RT_SAMPLE4)(0)(100)(0)(0)(VS)

<13:22:49> [Assert] [File=D:/jk/gamesdk_release/Code/CryEngine/RenderDll/XRenderD3D9/CompiledRenderObject.cpp]

does anyone know what is going on here?

Re: Random crashes when playing our game

#4
Here is the call stack from one of the random crashes. we've also tried the Github fix for CompiledRenderObjects.cpp...


<22:35:32> [Assert] [File=c:\aporia\bintemp\uber_files\cryrenderd3d11\..\..\..\Code\CryEngine\RenderDll\XRenderD3D9\CompiledRenderObject.cpp]
<CRITICAL EXCEPTION>
Exception: 0xC0000005, at Address: 0x0033:0x00007FF89F978555
Virtual memory usage: 6272Mb
Debug Status:
Last D3D debug message: No message queued. Debug runtime might be inactive or not installed.
Exception Code: 0xC0000005
Exception Addr: 0x0033:0x00007FF89F978555
Exception Module: CryRenderD3D11.dll
Exception Name : EXCEPTION_ACCESS_VIOLATION
Exception Description: Attempt to read from address 0x000000003F800000

The memory could not be "read"
Exception Code: 0xC0000005
Exception Addr: 0x0033:0x00007FF89F978555
Exception Module: CryRenderD3D11.dll
Exception Description: EXCEPTION_ACCESS_VIOLATION, Attempt to read from address 0x000000003F800000

The memory could not be "read"
Memory in use: 6778.1MB
Debug Status:
Out of Memory: 0
MainThread Id: 18408
RenderThread Id: 16368
[Crashed] Thread Id: 18408
[Crashed] ThreadName: Main

[Crashed] Call Stack: ["Main" [18408]]

=============================================================================
29) stl::SharedSizePoolAllocator<stl::HeapAllocator<CRenderObjectsPools::PoolSyncCriticalSection,stl::HeapSysAllocator> >::Allocate() [poolallocator.h:101]
28) CRenderView::AllocCompiledObjectTemporary() [renderview.cpp:111]
27) CRenderView::AddRenderItem() [renderview.cpp:699]
26) CRenderer::EF_AddEf_NotVirtual() [renderer_jobs.cpp:177]
25) CRenderMesh::Render() [rendermesh.cpp:4987]
24) CStatObj::RenderRenderMesh() [statobjrend.cpp:1376]
23) CStatObj::RenderObjectInternal() [statobjrend.cpp:1296]
22) CStatObj::RenderInternal() [statobjrend.cpp:1128]
21) CStatObj::Render() [statobjrend.cpp:74]
20) CGeomCacheRenderNode::Render() [geomcacherendernode.cpp:317]
19) C3DEngine::RenderRenderNode_ShadowPass() [3dengine.cpp:6309]
18) ShadowMapFrustum::Job_RenderShadowCastersToView() [shadow_renderer.cpp:33]
17) ShadowMapFrustum::RenderShadowFrustum() [shadow_renderer.cpp:142]
16) SShadowRenderer::RenderFrustumsToView() [shadow_renderer.cpp:188]
15) CD3D9Renderer::EF_PrepareShadowGenRenderList() [d3dshadows.cpp:105]
14) C3DEngine::RenderScene() [3denginerender.cpp:1823]
13) C3DEngine::RenderInternal() [3denginerender.cpp:1508]
12) C3DEngine::RenderWorld() [3denginerender.cpp:915]
11) CSystem::Render() [systemrender.cpp:739]
10) CCryAction::PostUpdate() [cryaction.cpp:3022]
9) CCryAction::Update() [cryaction.cpp:2263]
8) CCryAction::Run() [cryaction.cpp:2179]
7) CCryAction::StartEngine() [cryaction.cpp:2083]
6) RunGame() [main.cpp:170]
5) WinMain() [main.cpp:228]
4) __scrt_common_main_seh() [exe_common.inl:264]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]
=============================================================================


Call Stack: ["RemoteConsoleServer" [6500]]
10) ZwWaitForSingleObject() [:0]
9) Tcpip4_WSHOpenSocket2() [:0]
8) AcceptEx() [:0]
7) WSAAccept() [:0]
6) accept() [:0]
5) SRemoteServer::ThreadEntry() [remoteconsole.cpp:384]
4) CThreadManager::RunThread() [systemthreading.cpp:218]
3) o__strtoui64() [:0]
2) BaseThreadInitThunk() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["JobSystem_Worker_1" [15824]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) JobManager::ThreadBackEnd::CThreadBackEndWorkerThread::ThreadEntry() [threadbackend.cpp:318]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["JobSystem_Worker_3" [13980]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) JobManager::ThreadBackEnd::CThreadBackEndWorkerThread::ThreadEntry() [threadbackend.cpp:318]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["JobSystem_Worker_2" [6268]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) JobManager::ThreadBackEnd::CThreadBackEndWorkerThread::ThreadEntry() [threadbackend.cpp:318]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["ShaderCompile" [2224]]
8) ZwDelayExecution() [:0]
7) SleepEx() [:0]
6) CAsyncShaderTask::CShaderThread::ThreadEntry() [d3dhwshadercompiling.cpp:4370]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["JobSystem_Worker_0" [1912]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) JobManager::ThreadBackEnd::CThreadBackEndWorkerThread::ThreadEntry() [threadbackend.cpp:318]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["GFxMeshCacheReset" [9176]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) CryModuleGetMemoryInfo() [:0]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["Streaming AsyncCallback" [3840]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) CStreamingWorkerThread::ThreadEntry() [streamiothread.cpp:717]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["JobSystem_Worker_0 (Blocking)" [9672]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) JobManager::BlockingBackEnd::CBlockingBackEndWorkerThread::ThreadEntry() [blockingbackend.cpp:245]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["NotificationNetwork" [9632]]
10) ZwWaitForSingleObject() [:0]
9) Tcpip4_WSHOpenSocket2() [:0]
8) NSPStartup() [:0]
7) select() [:0]
6) CNotificationNetwork::ProcessSockets() [notificationnetwork.cpp:1258]
5) CNotificationNetwork::CThread::ThreadEntry() [notificationnetwork.cpp:1055]
4) CThreadManager::RunThread() [systemthreading.cpp:218]
3) o__strtoui64() [:0]
2) BaseThreadInitThunk() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["Streaming File IO HDD" [15820]]
10) ZwWaitForSingleObject() [:0]
9) WaitForSingleObjectEx() [:0]
8) CryEvent::Wait() [crythreadimpl_win32.h:243]
7) CStreamEngineWakeEvent::Wait() [streamasyncfilerequest.h:495]
6) CStreamingIOThread::ThreadEntry() [streamiothread.cpp:133]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["Streaming File IO InMemory" [16604]]
10) ZwWaitForSingleObject() [:0]
9) WaitForSingleObjectEx() [:0]
8) CryEvent::Wait() [crythreadimpl_win32.h:243]
7) CStreamEngineWakeEvent::Wait() [streamasyncfilerequest.h:495]
6) CStreamingIOThread::ThreadEntry() [streamiothread.cpp:133]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["RenderThread" [16368]]
10) NtWaitForAlertByThreadId() [:0]
9) RtlAcquireSRWLockExclusive() [:0]
8) stl::SizePoolAllocator<stl::HeapAllocator<CRenderObjectsPools::PoolSyncCriticalSection,stl::HeapSysAllocator> >::Deallocate() [poolallocator.h:237]
7) CRenderView::ClearTemporaryCompiledObjects() [renderview.cpp:1072]
6) CRenderView::Clear() [renderview.cpp:94]
5) CRenderView::SShadows::Clear() [renderview.cpp:1464]
4) CRenderView::Clear() [renderview.cpp:90]
3) CD3D9Renderer::RT_RenderScene() [d3drendpipeline.cpp:4101]
2) SRenderThread::ProcessCommands() [renderthread.cpp:2947]
1) SRenderThread::Process() [renderthread.cpp:3292]



Call Stack: ["ZLibCompressor" [7928]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) XMLCPB::CCompressorThread::ThreadEntry() [xmlcpb_zlibcompressor.cpp:145]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["UserAnalytics" [15676]]
9) NtWaitForAlertByThreadId() [:0]
8) RtlSleepConditionVariableSRW() [:0]
7) SleepConditionVariableSRW() [:0]
6) CUserAnalyticsSendThread::ThreadEntry() [useranalytics.cpp:43]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["NetAddressSolver" [2824]]
10) NtWaitForAlertByThreadId() [:0]
9) RtlSleepConditionVariableSRW() [:0]
8) SleepConditionVariableSRW() [:0]
7) CryConditionVariable::TimedWait() [crythreadimpl_win32.h:269]
6) CNetAddressResolver::ThreadEntry() [netresolver.cpp:779]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["ServiceNetwork" [7352]]
8) ZwDelayExecution() [:0]
7) SleepEx() [:0]
6) CServiceNetwork::ThreadEntry() [servicenetwork.cpp:1723]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["InputWorker" [12716]]
8) ZwDelayExecution() [:0]
7) SleepEx() [:0]
6) CInputConnectionThreadTask::ThreadEntry() [xinputdevice.cpp:94]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["Network" [10400]]
10) ZwRemoveIoCompletion() [:0]
9) GetQueuedCompletionStatus() [:0]
8) CSocketIOManagerIOCP::PollWait() [socketiomanageriocp.cpp:87]
7) CNetwork::DoMainTick() [network.cpp:1607]
6) CNetwork::UpdateTick() [network.cpp:1403]
5) CNetwork::CNetworkThread::ThreadEntry() [network.cpp:161]
4) CThreadManager::RunThread() [systemthreading.cpp:218]
3) o__strtoui64() [:0]
2) BaseThreadInitThunk() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["Physics" [17668]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) CPhysicsThreadTask::ThreadEntry() [system.cpp:998]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["PhysicsWorkerThread_0" [4644]]
8) ZwWaitForSingleObject() [:0]
7) WaitForSingleObjectEx() [:0]
6) CPhysicalWorld::ThreadProc() [physicalworld.cpp:2232]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["MainAudioThread" [5112]]
10) CAudioObjectManager::HasActiveData() [audioobjectmanager.cpp:445]
9) CAudioObjectManager::Update() [audioobjectmanager.cpp:163]
8) CAudioTranslationLayer::Update() [atl.cpp:191]
7) CAudioSystem::InternalUpdate() [audiosystem.cpp:331]
6) CAudioThread::ThreadEntry() [audiosystem.cpp:22]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]



Call Stack: ["CaptureCrashScreenShot" [700]]
9) NtWaitForAlertByThreadId() [:0]
8) RtlSleepConditionVariableSRW() [:0]
7) SleepConditionVariableSRW() [:0]
6) CCaptureCrashScreenShot::ThreadEntry() [debugcallstack.cpp:98]
5) CThreadManager::RunThread() [systemthreading.cpp:218]
4) o__strtoui64() [:0]
3) BaseThreadInitThunk() [:0]
2) RtlUserThreadStart() [:0]
1) RtlUserThreadStart() [:0]

<22:35:33> Saved dump file to 'error.dmp'
Attempting to create error screenshot "error.jpg"
<22:35:33>
<22:35:33> Screenshot: error.jpg
<22:35:33> [CONSOLE] Executing console command 'goto'
<22:35:33> [Error] [CVARS]: [EXECUTE] command goto is marked [VF_CHEAT]
<22:35:33> GOTO 1935.25 1518.54 65.95 -5.61 0.00 77.44
<22:35:33>
<22:35:33> Drawcalls: 3848
<22:35:33> FPS: 62.0 (33.3 ms)
<22:35:33> Tris: 1142,923
<22:35:33> CGF streaming: Loaded:1332 InProg:0 All:2084 Act:2084 MemUsed:280.88 MemReq:0.00 PoolSize:128
<22:35:33> TexStreaming: MemUsed:1749.97MB(113%%) PoolSize:1536.00MB Trghput:18014398509477296.00KB/s
<22:35:33> [CONSOLE] Executing console command 'sys_RestoreSpec test*'
<22:35:33>
<22:35:33> sys_spec = 3
<22:35:33>
<22:35:33>
Successfully created screenshot.
<22:46:03> [Assert] [File=c:\aporia\bintemp\uber_files\cryrenderd3d11\..\..\..\Code\CryEngine\RenderDll\XRenderD3D9\DevBuffer.cpp]
<22:46:03> [Assert] CResFile - File still open at shutdown: Shaders/Cache/D3D11/CGPShaders/Terrain@Common_ZPassPS.fxcb (streaming: no) [File=c:\aporia\bintemp\uber_files\cryrenderd3d11\..\..\..\Code\CryEngine\RenderDll\Common\ResFile.cpp]

Re: Random crashes when playing our game

#7
Thing is there is really not much we can assume from a call stack that crashes on an allocation - could literally be anything.
You're going to have to narrow it down to *at least* either an asset or engine component.

So, my advice;
Test on a blank level.
If it still does it: Swap out and/or disable system components... Test using DX12 dll. (r_driver=DX12) etc.
If it seemingly fixes the issue: Copy your level and swap/delete w/e assets until you have 1 or more that seem to cause the issue.

I assume this is game launcher and not sandbox? Have you tested in release mode? (Pure game client).

You need to find the something that is either using a component that causes the issue, or maybe a malformed asset or even your own game code.
Either way you really need to do some testing and narrow it down.

good luck

- EDIT;
Just noticed you mentioned this only happens with shadows enabled?
Can you narrow down which mode?
Also try to narrow down if it is an asset causing it, or the level setup itself etc etc. Basically still do same as above :P.
Uniflare
CRYENGINE Technical Community Manager
Here to help the community and social channels grow and thrive.

My personal belongings;
Beginner Guides | My GitHub | Splash Plugin

Re: Random crashes when playing our game

#8
thanks for the input, we will try some of the things out!

Another factor we found out: The game does not do this crash on Windows 8.1... the only difference I know of, between Windows 10 and 8.1 is that Windows 8.1 has DirectX11 and Win10 has DirectX12

We tried with DirectX12 enabled but it killed our framerate completely and crashed after a minute or so (another type of crash).. hmmm

Who is online

Users browsing this forum: No registered users and 3 guests

cron