umount: /u01: device is busy: even when Grid Infrastructure is down (12c)

Sometimes back, I noticed that even after shutting down GI (12.1.0.2.x) in a RAC setup, I still can't unmount the filesystem where the GI binaries are located.

This is applicable for GI version 12.1.0.2.x and even in Exadata too. Reproducible every time when GI shutdown completes.

Below is the scenario -

After shutting down the GI (12.1.0.2.x) using crsctl stop crs. It's not possible to unmount the /u01 mount point (S/W location of GI).

 # umount /u01   
 umount: /u01: device is busy.   
     (In some cases useful info about processes that use   
      the device is found by lsof(8) or fuser(1))  
 # cat /etc/oracle-release  
 Oracle Linux Server release 6.6  
 # uname -r   
 3.8.13-68.2.2.el6uek.x86_64  

 -- No output when running lsof/fuser:   
 # lsof /u01   
 # fuser -mv /u01   
 # lsof | grep /u01   
 # lsof | grep u01  

 -- No loop device, nor any NFS mounts   
 # losetup -a   
 # exportfs -v  

 -- Nothing is running after "crsctl stop crs"   
 # ps -ef |grep d.bin   
 root   22343 3617 0 11:26 pts/0  00:00:00 grep d.bin   
 # ps -ef |grep oracle   
 root   22345 3617 0 11:26 pts/0  00:00:00 grep oracle   

 -- Filesystem Mount Options   
 # mount | grep /u01   
 /dev/mapper/VolGroup-LVDbOra1 on /u01 type ext4 (rw,nodev)   

 -- Its been observed that following modules are currently loaded into the kernel   
 # lsmod | grep oracle  
 oracleacfs      3308260 0  
 oracleadvm      508030 0  
 oracleoks       506741 2 oracleacfs,oracleadvm  

It looks like the drivers oracleadvm and oracleacfs is using /u01.
File referenced by drivers will not show up on the lsof nor fuser output.

From the MOS Note # Unable to Umount /u01 on Exadata X5 machine (Oracle 12.1.0.2 BP6) (Doc ID 2001832.1) -
After removing the driver modules from the Kernel using modprobe, it is possible to unmount the filesystem:

 # modprobe -r oracleadvm  
 # modprobe -r oracleacfs  
 # umount /u01  
 # df -h  
 Filesystem      Size Used Avail Use% Mounted on  
 /dev/mapper/VolGroup-LVDbSys1  
             12G 1.4G 9.8G 12% /  
 tmpfs         1.8G   0 1.8G  0% /dev/shm  
 /dev/sda1       477M  40M 408M  9% /boot  

BUT, it turns out that the recommended method is to -

• First stop the Oracle stack: crsctl stop crs
• Then unload the drivers: acfsload stop
• Lastly unmount the native file systems (e.g., "/u01"): umount /u01