4lowTheRabbit.github.io

Workshop - Java Web App slow boot up

Short Story

Issue:

A few number of java apps are not starting up successfully. This seems to be happening with the ANT 79 upgrade. This does not appear to be related to any particular version of java. Java apps felt the impact of this the most if a startup time wasn’t long enough. It would result in a startup loop.

Customer impact:

The java process tries to start up but ends up restarting. Requests to the web app return 500 errors.

How to mitigation:

Long Story

1) One day….

Google’s Project Zero team discovered serious security flaws caused by “speculative execution,” a technique used by most modern processors (CPUs) to optimize performance.

The Project Zero researcher, Jann Horn, demonstrated that malicious actors could take advantage of speculative execution to read system memory that should have been inaccessible. For example, an unauthorized party may read sensitive information in the system’s memory such as passwords, encryption keys, or sensitive information open in applications. Testing also showed that an attack running on one virtual machine was able to access the physical memory of the host machine, and through that, gain read-access to the memory of a different virtual machine on the same host.

These vulnerabilities affect many CPUs, including those from AMD, ARM, and Intel, as well as the devices and operating systems running on them.

2) Azure deployed the security patch:

Links about How Azure handle this vulnerablity:

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/mitigate-se https://support.microsoft.com/en-us/help/4073119/protect-against-speculative-execution-side-channel-vulnerabilities-in

Other than what is in that article, we won’t go into detail on exactly what we do. It impacted the Java apps that had low start up timeouts. We won’t go into detail on what exact change caused that - just that if they had lower start up times, to increase them and they will be fine.

“Note Enabling mitigations that are off by-default may affect performance. The actual performance effect depends on multiple factors, such as the specific chipset in the device and the workloads that are running.”

3) Someone else run some tests, got some results:

How meltdown and spectre patches drag down older hardware https://www.pcworld.com/article/3250645/laptop-computers/how-meltdown-and-spectre-patches-drag-down-older-hardware.html

“Generally, performance in common tasks will be hard for the average person to notice most of the time. So yeah, breathe a sigh of relief.”

And then sometimes, it’ll just hit you in the face, with wait times taking 25 percent more on I/O-intensive tasks, such as decompressing a file. “

Troubleshooting and resolution

How to handle the issue: