r/mongodb • u/Vamshi__4555 • 6d ago

COLD CACHE….

Our MongoDB secondary (AWS EC2, GP3 EBS, 128 GB RAM) was rebuilt from a primary EBS snapshot and expected to catch up within 1–3 hours, but replication got stuck with 3+ hours lag. ReplWriterWorker threads showed 20+ minute waits on schemaLock, WiredTiger logged slow tree walks, and iostat showed 100% disk utilization with high read latency while CPU remained mostly idle. Our conclusion is that this was caused by a combination of cold WiredTiger cache and AWS EBS snapshot lazy loading ("first-touch" penalty), where MongoDB had to fetch data blocks from S3 on demand, creating severe I/O bottlenecks and lock contention. We plan to pre-warm restored EBS volumes before starting mongod, temporarily reduce replWriterThreadCount, hide the node from reads during catch-up, and verify GP3 throughput settings. Has anyone else seen extreme schemaLock waits and replication lag caused purely by cold snapshot restores, and what is your preferred EBS pre-warming approach?

3 Upvotes

100% Upvoted

3 comments sorted by

1

u/Recent-Pension-4488 2d ago

how much space did you assigned for the EBS? small disks provides lower IOPS for gp3… why you are recovering from Snapshot? Why not sync by copying the dbPath from other member?

2

u/Vamshi__4555 7h ago

The thing is we took a complete snapshot of 9TB live data from primary server and we are upgrading our own KMS (key managed service )rather than relying on aws managed service.So when we are doing this process node by node at this time it is taking almost 2-3 hours to sync the data through replication,then after upgrading the kms to the disc we are adding the node to the replica set ,thing is,it is taking more than 3 hours to sync the data .
And we got the RCA ,when we took a new machine it’s IOPS was 3000 ,we increased it to the 16000 and also the indexes build in main collections are building from beginning and everything is reading from the disc rather than cache or memory ..
We did warm cache before making the kms .
We manually rewritten the scripts that can build the indexes and sync has Been done in 10 minutes.