<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:103429128;
mso-list-type:hybrid;
mso-list-template-ids:-1562843516 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:457451214;
mso-list-type:hybrid;
mso-list-template-ids:-1006879114 1354636930 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F06E;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Memory Issue 1: JNOS using massive (>1G) amount of memory = overgrown mailboxes:<o:p></o:p></b></p><p class=MsoNormal><b><o:p> </o:p></b></p><p class=MsoNormal>The JNOS hangs and use of massive (>1GB) amounts of memory turned out to be a problem with JNOS corrupting/overgrowing (for lack of a better word) one or more mailboxes. On the machines exhibiting the hangs, I discovered that one or more JNOS mailboxes had grown to 100s of MB or even > 1GB. When a forwarding session occurred on machines in this condition, the temp files used by JNOS (/tmp/fileXXXXXX) would also grow to 100s of MB or > 1GB, free memory would drop to almost nothing, and swap space utilization would start to creep up. In all cases that I’ve seen, the mailboxes were bulletin mailboxes. To relieve the symptom, I deleted the mailbox(s) (mailbox-name.*) and deleted the temp files (/tmp/file*). After that, JNOS would behave normally.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Some may recall that I reported this several months ago. The problem mailboxes at that time contained multiple copies of each message, making it enormously large. But the problem did not happened since then, until this past week. It just so happened that it occurred when I updated to 2.0k. Because I suspected 2.0k, I backed our production machines off of 2.0j.7v. But it happened again, after that. Therefore, this is NOT something new in 2.0k. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I don’t have any hard evidence because I don’t know how to replicate the problem. But one suspect is the “expire future” function. If I remember correctly, the first occurrence of this problem was when Maiko first updated expire future to deal with different time zones. Perhaps something in the way the indexes were updated created a side-effect? This is pure speculation. But it’s based on the following sequence of events. <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I had expire future turned off because it was ignoring the timezone and “future”-ing legitimate messages<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Maiko did some work on expire future to deal with different timezones<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I turned on expire future<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I *<b>THINK</b>* that the first occurrence of the problem happened after that (but I don’t recall exactly and I didn’t correlate the two issues at the time)<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I turned off expire future because the first fix didn’t quite work<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I didn’t experience any problems for months<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>Maiko fixed expire future in 2.0k<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>When I updated to 2.0k, I also turned on expire future<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>When I experienced the hangs, I backed off to 2.0j.7v, but left expire future turned on<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l1 level1 lfo2'><![if !supportLists]><span style='font-family:Wingdings'><span style='mso-list:Ignore'>n<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]>I had another mailbox overgrowth with 2.0j.7v<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Workaround:<o:p></o:p></p><p class=MsoNormal>I created a shell script that checks the JNOS mailbox sizes and sends mail if any mailbox is larger than a specified size (a few MB). It is run by cron every hour. I’m hoping this will allow me to monitor the situation before it gets too large. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Suggestions:<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>1)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>As previously requested, change the name of the temp files used by jnos to show that they below to jnos. Something like “/tmp/jnos-XXXXXX” would be more appropriate than the current /tmp/fileXXXXXX. This would at least let us know where the problem file is coming from.<o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>2)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Perhaps JNOS could be enhanced with configurable warning and error levels of max mailbox size. Upon reaching the warning size, it would log a warning. Upon reaching the max error size, it would log an error and not add anymore to the mailbox. This wouldn’t fix the problem, but it would prevent the problem from becoming a hang. <o:p></o:p></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='mso-list:Ignore'>3)<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>Tracking the root cause will be difficult without knowing how to reproduce the problem. Perhaps some added diagnostic logging is in order?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Michael<o:p></o:p></p><p class=MsoNormal>N6MEF<o:p></o:p></p></div></body></html>