At ARender, we believe in transparency and sharing the challenges we face on a daily basis. This commitment not only keeps us accountable, but also enriches the community by sharing knowledge and solutions that might be useful to others facing similar hurdles. In this blog, we are highlighting a recent challenge tackled by our tech lead that posed a potential risk to our customers.
Diving into the Challenge
When ARender is deployed via Docker/Kubernetes and used in conjunction with NFS (a shared file system), it exhibits malfunctions in some features. These problems only occur in this specific environment, which is also used by some of our clients, potentially exposing them to the same difficulties.
Using java.net.URL in our code creates a temporary lock on the specified file, preventing NFS from updating it if it is created by another NFS client. Here is an example illustrating this problem:
URL url = new File(“/efs-mount-point/hello-world.txt”) . toUri() . toURL() ;
Whether the code above is executed or not, it results in the update of the file hello-world.txt on the NFS being blocked for up to a minute, often around 20 seconds.
It appears that the use of java.net.URL is not suited to our needs, which primarily involve transmitting source or target file paths to microservices. Replacing this practice with the direct use of a string representing the absolute path of the file could avoid this blocking issue when used with NFS.
Problem Identification:
- Origin of detection: The problem was initially identified in a demonstration environment.
- Expansion of the problem: The extent of the problem was recognized when clients reported experiencing the same issue, which extended the problem beyond the demo environment.
- Collaboration with partners: Discussions with our partners, identifying the same usage parameters (Kubernetes + NFS), allowed us to determine that the problem was related to NFS.
- Diagnostic tests: Tests carried out on an earlier version of ARender, which functioned without anomalies, confirmed that it was the latest version of the software that was incompatible with NFS.
A word from Maxime Tran - ARender Tech Lead who worked on this issue:
"These problems are a bit tricky because they're more about the architecture than the app itself. They're pretty rare but can really shake things up, especially with our new version. So, I rolled up my sleeves and tried to sort it out as best I could!"
This issue will be solved in the next ARender version
Comments