近期weblogic 11g永久代內存溢出，分析JVM dump文件是沒有用處的，因為那只是堆內存，永久代不在里面。目前永久代設置是1G,遙想當年，只有400M，這么多年來一直在漲，現在一次full gc需要10多秒。可以增大到1.5G，但從GC日志中可以看到永久代在不斷的增長，如果二周不重啟weblogic，內存又不夠用了，不是長久之計。java.lang.OutOfMemoryError: PermGen spaceat java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.6.0_30]at java.lang.Class.PRivateGetDeclaredMethods(Class.java:2427) ~[na:1.6.0_30]at java.lang.Class.getDeclaredMethod(Class.java:1935) ~[na:1.6.0_30]at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382) ~[na:1.6.0_30]at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52) ~[na:1.6.0_30]at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438) ~[na:1.6.0_30]at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_30]可以看到是方法區有問題，創建了大量的類，從GC日志上看，啟動之后永久代有700M，隨著時間的推移，會緩慢增長，這就是內存泄露，堆內存泄露也是這樣。如何定位哪些類一直在漲呢，此時就用到兩個重要的JVM參數，一個顯示每個加載的類，一個顯示每個卸載的類：-XX:+TraceClassLoading -XX:+TraceClassUnloadingweblogic里面會出現這種日志，需要自己寫程序解析最后還有哪些類存活著。[Loaded MM.statistics.mmbillstatistics.exception.MmBillStatisticalTypeException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/mmbillstatistics/exception/MmBillStatisticalTypeException.class][Loaded MM.statistics.mmbillstatistics.appservice.impl.MmBillStatisticalTypeSes_1k61gv_IMmBillStatisticalTypeServiceImpl_1035_WLStub from file:/data/wls1035/modules/com.bea.core.utils.wrapper_1.4.0.0.jar][Loaded MM.statistics.amfmHuiXuFinance.appservice.AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf from file:/data/Domain/mm_Domain/servers/mmServer1/cache/EJBCompilerCache/1nhs7towub2hs/mm/statistics/amfmHuiXuFinance/appservice/AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf.class][Loaded MM.statistics.amfmHuiXuFinance.exception.AmfmHuiXuFinanceException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/amfmHuiXuFinance/exception/AmfmHuiXuFinanceException.class][Unloading class mm.purchase.purchaSEOrder.model.PurchaseItemVO][Unloading class mm.inventory.reservereplenishplan.appservice.impl.ReserveReplenishPlanBizService][Unloading class mm.inventory.reservequota.dao.ReserveQuotaBatchDAO]解析程序開始：drop table load_class purge;truncate table load_class;create table load_class( nu number, action varchar2(20), class_name varchar2(1000), class_file varchar2(1000));import java.io.BufferedReader;import java.io.File;import java.io.FileInputStream;import java.io.InputStreamReader;import java.sql.Connection;import java.sql.DriverManager;import java.sql.PreparedStatement;/** * [Loaded java.lang.ClassNotFoundException from /usr/local/jdk1.6/jre/lib/rt.jar] * [Unloading class com.MM.externalinterface.fmis.impl.FmisBizService] * */public class PermAla { static final String driver_class = "Oracle.jdbc.driver.OracleDriver"; static final String connectionURL = "jdbc:oracle:thin:@10.10.15.164:1521:orcl"; static final String userID = "SPROC"; static final String userPassWord = "SPROC"; public static void readTxtFile(String filePath){ Connection con = null; String s_sql = "insert into load_class values(?,?,?,?)"; PreparedStatement pstmt = null; int i=0; try { Class.forName (driver_class).newInstance(); con = DriverManager.getConnection(connectionURL, userID, userPassword); pstmt = con.prepareStatement(s_sql); con.setAutoCommit(false); String encoding="GBK"; File file=new File(filePath); InputStreamReader read = new InputStreamReader( new FileInputStream(file),encoding);//考慮到編碼格式 BufferedReader bufferedReader = new BufferedReader(read); String lineTxt = null; String[] lineTxtArray= null; while((lineTxt = bufferedReader.readLine()) != null){ if(lineTxt.indexOf("[Loaded")==0 ) { lineTxtArray = lineTxt.split(" "); pstmt.setInt(1, i); pstmt.setString(2, lineTxtArray[0]); pstmt.setString(3, lineTxtArray[1]); if(lineTxtArray.length ==4) { pstmt.setString(4, lineTxtArray[3]); }else{ System.out.println(lineTxt); } pstmt.addBatch(); i++; } else if(lineTxt.indexOf("[Unloading")==0){ lineTxtArray = lineTxt.split(" "); pstmt.setInt(1, i); pstmt.setString(2, lineTxtArray[0]); pstmt.setString(3, lineTxtArray[2]); pstmt.addBatch(); i++; } if(i % 10000 == 0){ pstmt.executeBatch(); con.commit(); } } con.commit(); read.close(); } catch (Exception e) { System.out.println("讀取文件內容出錯,行數："+i); e.printStackTrace(); }finally{ if(pstmt != null){ try { pstmt.close(); } catch (Exception e) { e.printStackTrace(); }finally{ pstmt = null; } } if(con != null){ try { con.close(); } catch (Exception e) { e.printStackTrace(); }finally{ con = null; } } } } public static void main(String argv[]){ String filePath = "E://永久代內存溢出//20161213//Server1.log"; readTxtFile(filePath); }}--執行java代碼導入之后，處理一些特殊的格式update load_class set action = replace(action, '[', ''), class_file = replace(class_file, ']', ''), class_name = replace(class_name, ']', '');commit;update load_class set class_name = replace(class_name, 'file:', ''), class_file = replace(class_file, 'file:', '');commit;SQL> select * from load_class where rownum <10; NU ACTION CLASS_NAME CLASS_FILE--- ------- --------------------------------------- -------------------------------- 96 Loaded sun.misc.SharedSecrets /data/jdk1.6.0_45/jre/lib/rt.jar 97 Loaded sun.misc.Unsafe /data/jdk1.6.0_45/jre/lib/rt.jar 98 Loaded java.lang.IncompatibleClassChangeError /data/jdk1.6.0_45/jre/lib/rt.jar 99 Loaded java.lang.NoSuchMethodError /data/jdk1.6.0_45/jre/lib/rt.jar100 Loaded sun.reflect.Reflection /data/jdk1.6.0_45/jre/lib/rt.jar101 Loaded java.util.Collections /data/jdk1.6.0_45/jre/lib/rt.jar102 Loaded java.lang.Iterable /data/jdk1.6.0_45/jre/lib/rt.jar103 Loaded java.util.Collection /data/jdk1.6.0_45/jre/lib/rt.jar104 Loaded java.util.Set /data/jdk1.6.0_45/jre/lib/rt.jar --顯示加載且沒有卸載的類 drop table purge;create table aswith res as(select count(*) over(partition by action,class_name order by nu asc) rn, t.* from load_class t),res1 as (select rn,class_name from res a where a.action='Loaded' minus select rn,class_name from res b where b.action='Unloading')select b.* from res1 a, res b where a.rn= b.rn and a.class_name= b.class_name;查看數據，對數據摸底 select count(1) from where class_file='__JVM_DefineClass__';select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/lib%';select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/classes%' or class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/ejb%';select count(1) from where class_file like '/data/wls1035/%'; select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/cache/EJBCompilerCache/%'; select count(1) from where class_file like '/data/jdk1.6.0_45/jre%'; select count(1) from where class_file in ('weblogic.utils.classloaders.GenericClassLoader', 'sun.misc.Launcher$AppClassLoader', 'weblogic.utils.classloaders.ChangeAwareClassLoader','instance');select count(1) from where class_file like '/data/Domain/MMDomain/jsp_temp%'; 可以看到反射的類很多，重點研究__JVM_DefineClass__和sun.reflect.GeneratedMethodAccessor，發現是反射產生的一個問題。參考http://stackoverflow.com/questions/16130292/java-lang-outofmemoryerror-permgen-space-java-reflection
When using Java reflection, the JVM has two methods of accessing the information on the class being reflected. It can use a JNI accessor, or a Java bytecode accessor.
If it uses a Java bytecode accessor, then it needs to have its own Java class and classloader (sun/reflect/GeneratedMethodAccessor class and sun/reflect/DelegatingClassLoader). Theses classes and classloaders use native memory.
The accessor bytecode can also get JIT compiled, which will increase the native memory use even more.
If Java reflection is used frequently, this can add up to a significant amount of native memory use. The JVM will use the JNI accessor first, then after some number of accesses on the same class, will change to use the Java bytecode accessor. This is called inflation, when the JVM changes from the JNI accessor to the bytecode accessor. Fortunately, we can control this with a Java property.
The sun.reflect.inflationThreshold property tells the JVM what number of times to use the JNI accessor. If it is set to 0, then the JNI accessors are always used. Since the bytecode accessors use more native memory than the JNI ones, if we are seeing a lot of Java reflection, we will want to use the JNI accessors. To do this, we just need to set the inflationThreshold property to zero.
如果節點使用sun公司jdk，配置-Dsun.reflect.inflationThreshold=2147483647 如果是IBM的jdk，配置-Dsun.reflect.inflationThreshold=0
以下是配置的前后類數量對比，反射類有明顯的下降，從gc日志上看也有回收的跡象：
| 加載類的出處 | Server1 | server2 |
| 加參數前 | 加參數后 | 加參數前 | 加參數后 |
| __JVM_DefineClass__(反射產生的類) | 42372 | 3632 | 19961 | 2822 |
| EAR/APP-INF/lib | 25552 | 23162 | 23613 | 23579 |
| EAR/APP-INF/classes | 21531 | 15684 | 15167 | 15176 |
| wls1035 | 19017 | 19420 | 19661 | 19168 |
| MMServer1/cache/EJBCompilerCache | 5996 | 5996 | 5996 | 5996 |
| /data/jdk1.6.0_45/jre | 3494 | 3493 | 3175 | 3458 |
| weblogic.utils.classloaders.GenericClassLoadersun.misc.Launcher$AppClassLoaderweblogic.utils.classloaders.ChangeAwareClassLoader | 2550 | 2574 | 2508 | 2516 |
| jsp_temp | 843 | 1395 | 843 | 849 |
當然我遇到的情況是反射類引起的問題，如果你的永久代設置太小，不用糾結，直接把永久代內存加大；如果是類太多導致（不是反射類），那就想辦法瘦身，想辦法 改造去掉一些依賴的jar包，可能是一個大工程。
