Fixing “Cannot remove directory: File exists” error
The error “Cannot remove directory File exists” occurs sometimes while deleting directories from NFS file systems.
In this article, we’ll take into account a scenario where I faced this show and will present an understanding of why it happened and finally how I eventually fixed it.
Fixing “Cannot Remove directory: File exists”Error
Recently a DBA asked me to delete a directory which was residing in a directory tree mounted under an NFS file system. The DBAs had permissions on that directory but still, they weren’t able to remove it so they asked us to investigate. The directory had a couple of strangely named hidden files in it.
[ssuri@sahil-redhat:~] $ sudo ls -ltr /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn total 0 [ssuri@sahil-redhat:~] $ sudo ls -ltra /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn total 32 drwx------ 3 oemms12 dba 4096 Dec 21 11:41 .. -rw-r----- 1 oemms12 dba 1753 Dec 21 11:42 .nfsF7B2 -rw-r----- 1 oemms12 dba 154 Dec 21 11:42 .nfs08B2 drwx------ 2 oemms12 dba 4096 Dec 21 11:44 .
I considered it a routine request and ran the rm -rf command on the directory.
But to my surprise, I received an error message.
[ssuri@sahil-redhat:~] $ sudo rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn rm: Unable to remove directory /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn: File exists
I thought it might have something to do with the hidden files so I executed the below command to remove the hidden files but I ran into the same error again.
[ssuri@sahil-redhat:~] $ sudo rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn/.* rm: Unable to remove directory /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn/: File exists
Strangely, the names of the hidden files had changed now.
[ssuri@sahil-redhat:~] $ sudo ls -ltra /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn total 32 drwx------ 3 oemms12 dba 4096 Dec 21 11:41 .. -rw-r----- 1 oemms12 dba 1753 Dec 21 11:42 .nfs58B2 -rw-r----- 1 oemms12 dba 154 Dec 21 11:42 .nfs48B2 drwx------ 2 oemms12 dba 4096 Dec 21 11:59 .
The files were simple text files and I was able to view their content.
sahil-redhat# file .nfs48B2 .nfs48B2: ascii text sahil-redhat# cat .nfs48B2 [2017-12-21T11:42:03+00:00] [opmn] [TRACE:32]  [internal] Server reloading [2017-12-21T11:42:06+00:00] [opmn] [TRACE:32]  [internal] Server reloading
Simple Solution to Fix
One solution to fix this problem was to remove the files from the NFS server.
The file system that I was working with was residing on a Netapp filer which I didn’t have access to.
The feasible solution was to identify any processes using that directory and indeed there were two.
On checking with the fuser command I was able to identify two application user processes which appeared to be accessing the directory in consideration.
sahil-redhat# fuser -u /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn: 19888c(root) 16004c(oemms12) 16059c(oemms12)
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND oemms12 16059 0.0 0.14777629328 ? S 11:42:06 0:00 /oems12ohsap/fmw/Oracle_WT2/opmn/bin/opmn -d oemms12 16004 0.0 0.142976 8400 ? S 11:41:37 0:00 /oems12ohsap/fmw/Oracle_WT2/opmn/bin/opmn -d
The third root owner process was generated when I ran the fuser command on the directory to list out the processes using it.
Further investigation revealed that this is the way NFS client handles an open file that has been deleted (unlinked).
In order to emulate this behavior using the functionality provided by the NFS protocol, the client implements what is called a “silly rename”, leaving the deleted file in place, under a different name, i.e., “.nfs<unique_file_id>”.
Once the file handle has been closed (by the program that still has opened it), the client will effectively complete the last stage of the unlink operation by deleting the file in question, i.e., that renamed file will disappear.
After talking to the DBAs I killed the two processes that were still using the directory.
sahil-redhat# kill -9 16059 16004
After this, I was able to remove the directory successfully.
sahil-redhat# rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
In this article we showed why you might receive a “Cannot remove directory File exists” error while trying to remove a directory and how to fix it. We hope you found this article to be useful and would now be aware as to how to fix this error if you come across it in future.