Troubleshooting "java.lang.OutOfMemoryError: unable to create new native thread" error in VPS installations
Symptoms
Fisheye cannot be started and the following appears in the atlassian-fisheye.log:
2013-10-01 13:11:16,900 ERROR - Crucible Database setup failed.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at com.jolbox.bonecp.ConnectionPartition.<init>(ConnectionPartition.java:236)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:356)
at com.jolbox.bonecp.provider.BoneCPConnectionProvider.createPool(BoneCPConnectionProvider.java:168)
at com.jolbox.bonecp.provider.BoneCPConnectionProvider.configure(BoneCPConnectionProvider.java:142)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:143)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84)
at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:438)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:91)
at org.hibernate.cfg.Configuration.buildSettingsInternal(Configuration.java:2163)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2159)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1383)
at com.cenqua.crucible.hibernate.HibernateUtil.setUp(HibernateUtil.java:100)
at com.cenqua.crucible.hibernate.DefaultDBControl.start(DefaultDBControl.java:243)
at com.cenqua.crucible.hibernate.DefaultDBControl.start(DefaultDBControl.java:221)
at com.cenqua.crucible.hibernate.DBControlFactoryImpl.<init>(DBControlFactoryImpl.java:40)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at com.cenqua.fisheye.web.jetty.FishEyeWebApplicationContext.doStart(FishEyeWebApplicationContext.java:28)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.cenqua.fisheye.web.WebServer.start(WebServer.java:276)
at com.cenqua.fisheye.ctl.Run.mainImpl(Run.java:233)
at com.cenqua.fisheye.ctl.Run.main(Run.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.cenqua.fisheye.FishEyeCtl.mainImpl(FishEyeCtl.java:98)
at com.cenqua.fisheye.FishEyeCtl.main(FishEyeCtl.java:41)
Cause
Specifically, whatever user needs to run Fisheye needs to be able to open at least 200 files at a time, and be able to execute >100 processes at a time, since 100+ threads are created by Fisheye.
Basically the aproximated maximum number of threads of a JVM can be calculated this way:
(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads
Resolution
- Modify the limits.conf file with the following:
sudo vim /etc/security/limits.confAdd the following for the user that runs FISHEYE.
limits.conf#<domain> <type> <item> <value> # #* soft core 0 #root hard core 100000 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #ftp - chroot /ftp #@student - maxlogins 4 fisheye soft nofile 4096 fisheye hard nofile 8192- Modify the common-session file with the following:
sudo vim/etc/pam.d/common-session Add the following line:
common-sessionsession required pam_limits.so- Restart Fisheye.
Last modified on Jul 31, 2018
Powered by Confluence and Scroll Viewport.