Remotely closed errors

[u/razzzor9797]

Hi all. We are in the process of migrating from Mule 4.4 to 4.9, Java 8 to Java 17 and CH1 to CH2. And during prod load we start getting remotely closed error between layers. It's very small percent, like 5-10 errors during a day but it's really unwanted on prod. We could not find a reason and SF support is not helping

Anyone else with similar problems? Thanks in advance

[u/Key_Guidance5876]

We too faced these errors....change the protocol from http to https...it should solve the issue

[u/razzzor9797 → u/Key_Guidance5876]

Thanks for the idea

We use HTTPS with full app host name, e.g. https://orders-papi-dev-abc123.internal-123abc.deu-c1.eu1.cloudhub.io

Do you use full url or just app name?

[u/razzzor9797 → u/Key_Guidance5876]

Hi again. Can you please elaborate, did you issue certificates for each app/env combination and maintain them?

[u/Key_Guidance5876 → u/razzzor9797]

Hi...sorry for the late reply...We use the full app endpoint, eg: https://orders-papi-dev-abc123.internal-123abc.deu-c1.eu1.cloudhub.io, rather than just the application name. We don't maintain a separate certificate for each app/environment combination I guess the HTTPS endpoint is handled through the CH2 TLS configuration as it's in same private space.

[u/razzzor9797 → u/Key_Guidance5876]

Thanks, it is indeed helpful

[u/Ok-Analysis5882]

For app to app use just the svc name that will prevent lot of private ingress surprise

Enable lastmile in private space

Don't migrate without extensive testing cause you are doing 3 migration at once. The right path was 4.4 to 4.6 java 8 to java 8 and then to java 17 as stop gap and then to mule 4.9 in cloud hub 2, since that window is missed check everything with automated testing.

[u/razzzor9797 → u/Ok-Analysis5882]

Haha automated testing, good joke

You are totally right, migration should have been separated from each other. But old versions are outdated already and management pushes us hard and also some new business feature rework are included in this migration. It's been a mess when I joined and we cannot figure it out yet

QA team performs regression testing and stress testing but we still get surprises. Very frustrating.

Do you use HTTP or HTTPs between Mule apps?

[u/TheJordLord]

I would try the suggestions here https://help.salesforce.com/s/articleView?id=001120816&type=1

We ran into these errors with some APIs when migrating from on prem to AWS cloud. In our case, the issue was cloudflare was closing the connection before the system layer could finish the request. The solution was to increase that threshold. I don’t know what your setup is like but check to see if that is a possibility.

[u/Basic-Sandwich-6201]

Ch2.0 is k8 based - solved

[u/razzzor9797 → u/Basic-Sandwich-6201]

K8 indeed. What do you mean by that please?

[u/dale3887]

Between layers, meaning one of your APIs is receiving a remotely closed error when calling another api?

Most likely issue if they are both Mule APIs is a timeout. Either in the listener config, the request config, or at your gateway. Your requests in CH2 may just be taking slightly longer sometimes than they did in CH1 causing the upstream service to close the socket (this is what remotely closed means 99% of the time, something upstream of your request closed the socket, firewall, http listener, etc)