Tomcat 空间占用增长异常分析一例

有朋友问：

RHEL AS 3.0 上，df -k发现 /opt 在不断增长，而在 /home 里面 du -sh 却发现空间大小没有变化， /opt 下面是一个网站，把 tomcat 停了就不会增长了。这是怎么回事?

第一个感觉以为是 df 和 du 这两个命令显示的有差异。没错，的确有差异。但是，现在的主要问题是 df -k 命令显示的空间在增长。分析研究了 /proc 下相关进程对应的目录下的信息，未发现异常。图劳无功。

猜测tomcat 打开了某些文件描述符，虽然文件已经修改了，但是文件描述符如果没有关闭的话，一样会占用空间甚至增大空间的占用。用 lsof(LiSt Opened File) 进行分析。

注：对一些特定的路径进行了一点处理。首先获取/opt 对应的设备名字。

# lsof /dev/sdb2 | sort +6nr | head -5
java      21247 root    1w   REG  104,6 96538188 917506 /opt/tomcat/logs/catalina.out (deleted)
java      21247 root    2w   REG  104,6 96538188 917506 /opt/tomcat/logs/catalina.out (deleted)
java      21247 root  mem    REG  104,6  1394764 901145 /opt/tomcat/common/lib/nls_charset12.jar
java      21247 root  mem    REG  104,6  1394295 901143 /opt/tomcat/common/lib/nls_charset11.jar
java      21247 root  mem    REG  104,6  1159907 901140 /opt/tomcat/common/lib/classes12dms.jar
[root@stat root]#

经确认，/opt/tomcat/logs/catalina.out (deleted) 项占用的空间在疯狂增长。建议跟踪该文件的变化。

# tail -f /opt/tomcat/logs/catalina.out
java.net.UnknownHostException: auto at
java.net.InetAddress.getAllByName0(InetAddress.java:1016) at
java.net.InetAddress.getAllByName0(InetAddress.java:981) at
java.net.InetAddress.getAllByName(InetAddress.java:975) at
java.net.InetAddress.getByName(InetAddress.java:889)
org.apache.catalina.cluster.mcast.McastMember.getData(McastMember.jav a:122) at
org.apache.catalina.cluster.mcast.McastServiceImpl.send(McastServiceI mpl.java:221) at
org.apache.catalina.cluster.mcast.McastServiceImpl$SenderThread.run(M castServiceImpl.java:259)

注意到

java.net.InetAddress.getAllByName0(InetAddress.java:1016

测试可能是 Tomcat 的 Bug,把 java.net.InetAddress.getAllByName0(InetAddress.java:1016 作为搜索短语查找一下，果不其然：

http://issues.apache.org/bugzilla/show_bug.cgi?id=27259

Updated:2005/12/12 . 空间不释放的问题:

When you open a file, you get a pointer. Subsequent writes to this file
references this file pointer. The write call does not check to see if the file
is there or not. It just writes to the specified number of characters starting
at a predetermined location. Regardless of whether the file exist or not, disk
blocks are used by the write operation.
The df command reports the number of disk blocks used while du goes through the
file structure and and reports the number of blocks used by each directory. As
far as du is concerned, the file used by the process does not exist, so it does
not report blocks used by this phantom file. But df keeps track of disk blocks
used, and it reports the blocks used by this phantom file.

下载 LSOF 工具的使用文档

–EOF–