Summariztion for tomcat stress testing: DBCP could cause DDos Crash with no resume
Submitted by corel123 on Fri, 01/04/2005 - 03:13.
java
I noticed that the working system crashed with no resume. I try to repeat the crash by similation of extrem stress testing. the results happened now and then, but it always can be resumed after 5-15 minutes. At last, I use the WAS to launch 50 threads * 5 multiper sockets, ( I count it should be 250, however, why I collect the sessesion more than 500 in log?), in testing system. Except for other reasons, the main one cause the tomcat hault is the DBCP, the pool manager should return the connection which has been closed to the client request, so that it cause the JDBC client throw Exception of :
java.io.IOException
MESSAGE: Unexpected end of input stream
STACKTRACE:
java.io.IOException: Unexpected end of input stream
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1405)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1775)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1752)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:833)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1734)
at com.mysql.jdbc.Connection.(Connection.java:562)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:361)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:82)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
at dao.ConnectionManager.getConnection(ConnectionManager.java:28)
at dao.Lister.open(Lister.java:92)
at tags.list.ListContainerTag.doStartTag(ListContainerTag.java:142)
at org.apache.jsp.bbs.hcare_jsp._jspService(hcare_jsp.java:985)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at system.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:233)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:245)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:199)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:195)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:164)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:156)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:972)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:211)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:536)
** END NESTED EXCEPTION **
then all threads waiting for the connection just wait on query, it is the reason. once I changed to the single connection by one to one, it was still hault for minutes ( waiting for the connection empty, but I has set in the server.xml by the maxwait one 10 seconds?!), then it resume as usual.
The DBCP usage can increase the access speed in 100%-150%, however, if it can not guarantee the system resume work after the DDos or the max peak access, It is terrible for the bussiness website.
ch.test
java.io.IOException
MESSAGE: Unexpected end of input stream
STACKTRACE:
java.io.IOException: Unexpected end of input stream
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1405)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1775)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1752)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:833)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1734)
at com.mysql.jdbc.Connection.(Connection.java:562)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:361)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:82)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:119)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
at dao.ConnectionManager.getConnection(ConnectionManager.java:28)
at dao.Lister.open(Lister.java:92)
at tags.list.ListContainerTag.doStartTag(ListContainerTag.java:142)
at org.apache.jsp.bbs.hcare_jsp._jspService(hcare_jsp.java:985)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at system.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:42)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:233)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:245)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:199)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:195)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:164)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:156)
at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:564)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:972)
at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:211)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:309)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:387)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:673)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:615)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:786)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:677)
at java.lang.Thread.run(Thread.java:536)
** END NESTED EXCEPTION **
then all threads waiting for the connection just wait on query, it is the reason. once I changed to the single connection by one to one, it was still hault for minutes ( waiting for the connection empty, but I has set in the server.xml by the maxwait one 10 seconds?!), then it resume as usual.
The DBCP usage can increase the access speed in 100%-150%, however, if it can not guarantee the system resume work after the DDos or the max peak access, It is terrible for the bussiness website.
ch.test
