Fixing “Cannot remove directory: File exists” error

The error “Cannot remove directory File exists” occurs sometimes while deleting directories from NFS file systems.
In this article, we’ll take into account a scenario where I faced this show and will present an understanding of why it happened and finally how I eventually fixed it.

Fixing “Cannot Remove directory: File exists”Error

Recently a DBA asked me to delete a directory which was residing in a directory tree mounted under an NFS file system. The DBAs had permissions on that directory but still, they weren’t able to remove it so they asked us to investigate. The directory had a couple of strangely named hidden files in it.

[ssuri@sahil-redhat:~] $ sudo ls -ltr /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
total 0

[ssuri@sahil-redhat:~] $ sudo ls -ltra /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
total 32
drwx------ 3 oemms12 dba 4096 Dec 21 11:41 ..
-rw-r----- 1 oemms12 dba 1753 Dec 21 11:42 .nfsF7B2
-rw-r----- 1 oemms12 dba 154 Dec 21 11:42 .nfs08B2
drwx------ 2 oemms12 dba 4096 Dec 21 11:44 .

I considered it a routine request and ran the rm -rf command on the directory.
But to my surprise, I received an error message.

[ssuri@sahil-redhat:~] $ sudo rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
rm: Unable to remove directory /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn: File exists

I thought it might have something to do with the hidden files so I executed the below command to remove the hidden files but I ran into the same error again.

[ssuri@sahil-redhat:~] $ sudo rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn/.* 
rm: Unable to remove directory /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn/: File exists

Strangely, the names of the hidden files had changed now.

[ssuri@sahil-redhat:~] $ sudo ls -ltra /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
total 32
drwx------ 3 oemms12 dba 4096 Dec 21 11:41 ..
-rw-r----- 1 oemms12 dba 1753 Dec 21 11:42 .nfs58B2
-rw-r----- 1 oemms12 dba 154 Dec 21 11:42 .nfs48B2
drwx------ 2 oemms12 dba 4096 Dec 21 11:59 .

The files were simple text files and I was able to view their content.

sahil-redhat# file .nfs48B2
.nfs48B2: ascii text
sahil-redhat# cat .nfs48B2
[2017-12-21T11:42:03+00:00] [opmn] [TRACE:32] [] [internal] Server reloading
[2017-12-21T11:42:06+00:00] [opmn] [TRACE:32] [] [internal] Server reloading

Simple Solution to Fix

One solution to fix this problem was to remove the files from the NFS server.
The file system that I was working with was residing on a Netapp filer which I didn’t have access to.

The feasible solution was to identify any processes using that directory and indeed there were two.

On checking with the fuser command I was able to identify two application user processes which appeared to be accessing the directory in consideration.

sahil-redhat# fuser -u /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn
/oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn: 19888c(root) 16004c(oemms12) 16059c(oemms12)
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
oemms12 16059 0.0 0.14777629328 ? S 11:42:06 0:00 /oems12ohsap/fmw/Oracle_WT2/opmn/bin/opmn -d
oemms12 16004 0.0 0.142976 8400 ? S 11:41:37 0:00 /oems12ohsap/fmw/Oracle_WT2/opmn/bin/opmn -d

The third root owner process was generated when I ran the fuser command on the directory to list out the processes using it.

Further investigation revealed that this is the way NFS client handles an open file that has been deleted (unlinked).
In order to emulate this behavior using the functionality provided by the NFS protocol, the client implements what is called a “silly rename”, leaving the deleted file in place, under a different name, i.e., “.nfs<unique_file_id>”.
Once the file handle has been closed (by the program that still has opened it), the client will effectively complete the last stage of the unlink operation by deleting the file in question, i.e., that renamed file will disappear.

After talking to the DBAs I killed the two processes that were still using the directory.

sahil-redhat# kill -9 16059 16004

After this, I was able to remove the directory successfully.

sahil-redhat#  rm -rf /oem12runtime/OHS/ORACLE_WT2/instances/EPM_OHS2/diagnostics/logs/OPMN/opmn

Conclusion

In this article we showed why you might receive a “Cannot remove directory File exists” error while trying to remove a directory and how to fix it. We hope you found this article to be useful and would now be aware as to how to fix this error if you come across it in future.

Related Articles

25 mostly used commands

15 scp command practical examples

20 Network troubleshooting commands

Basic Commands

Thanks for your wonderful Support and Encouragement

Sahil Suri

I am a system administrator who loves to learn and share my knowledge with the community. I've been working in the IT industry since 2011.

Leave a Reply

Your email address will not be published. Required fields are marked *