[Mimedefang] Mimedefang with postfix. Process memory usage over time.

Discussion:

Robert Theisen

2017-11-10 15:33:10 UTC

We have an environment with postfix-2.10.1-6.el7.x86_64 and
mimedefang-2.78-6.el7.x86_64 . Postfix is configured to interact with
mimedefang via smtpd_milters =
unix:/var/spool/MIMEDefang/mimedefang.sock

[***@ext-mx02 ~]# uname -a
Linux ext-mx02 3.10.0-514.21.2.el7.x86_64 #1 SMP Sun May 28 17:08:21
EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[***@ext-mx02 ~]# free -h
total used free shared buff/cache available
Mem: 15G 10G 2.5G 692M 2.6G 4.0G
Swap: 8.1G 5.8G 2.3G

The mimedefang process looks like this after running for approximately 3 weeks

defang 26810 0.3 42.8 12680732 6961512 ? Sl Oct17 109:15
/usr/bin/mimedefang -P /var/spool/MIMEDefang/mimedefang.pid -m
/var/spool/MIMEDefang/mimedefang-multiplexor.sock -y -R -1 -U defang
-T -r -H -s -t -G -q -p /var/spool/MIMEDefang/mimedefang.sock

12.1 GB VIRT
6.6 GB RES

I checked the smap data for that mimedefang process and selected one
of the many 64MB blocks that was currently in swap.

7f895c000000-7f895fffc000 rw-p 00000000 00:00 0
Size: 65520 kB
Rss: 65520 kB
Pss: 65520 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 65520 kB
Referenced: 37632 kB
Anonymous: 65520 kB
AnonHugePages: 0 kB
Swap: 0 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
VmFlags: rd wr mr mw me nr sd

I did a memory dump of that block and ran strings against it and I got
a quarter of a million lines that look like this ...

_ {auth_authen} {auth_author} {auth_ssf} {auth_type} {cert_issuer}
{cert_subject} {cipher} {cipher_bits} {daemon_name} {daemon_port} i
{if_addr} {if_name} j {mail_addr} {mail_host} {mail_mailer}
{tls_version} {verify} {rcpt_addr} {rcpt_host} {rcpt_mailer}

That seems to correspond to this code snippet in mimedefang.c line 161
(in version 2.78) ...

/* Standard Sendmail macros */
/* We can't make it char const * because libmilter
is not const-correct. */
static char *StandardSendmailMacros[] = {
"_", "auth_authen", "auth_author", "auth_ssf", "auth_type",
"cert_issuer", "cert_subject", "cipher", "cipher_bits", "daemon_name",
"daemon_port",
"i", "if_addr", "if_name", "j", "mail_addr", "mail_host", "mail_mailer",
"tls_version", "verify", "rcpt_addr", "rcpt_host", "rcpt_mailer",
/* End of macros MUST be marked with NULL! */
NULL
};

and that macro gets copied to dbuf at line 952

/* Write the standard macros */
macro = StandardSendmailMacros;
while (*macro) {
append_macro_value(&dbuf, ctx, *macro);
macro++;
}

Is it possible that that dbuf is not getting freed properly on down the line?
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://list

Dianne Skoll

2017-11-10 17:15:29 UTC

Permalink

On Fri, 10 Nov 2017 10:33:10 -0500
Robert Theisen <***@redhat.com> wrote:

[snip]

Post by Robert Theisen
and that macro gets copied to dbuf at line 952
/* Write the standard macros */
macro = StandardSendmailMacros;
while (*macro) {
append_macro_value(&dbuf, ctx, *macro);
macro++;
}
Is it possible that that dbuf is not getting freed properly on down the line?

I can't see how. A bit further down:

if (data->cmdFD < 0) {
dbuf_free(&dbuf);
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
if (write_dbuf(&dbuf, data->cmdFD, data, "COMMANDS") < 0) {
dbuf_free(&dbuf);
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
dbuf_free(&dbuf);

and there are no branches or returns between the code you posted
and the code I posted above. However, I just checked one of our
busy servers and the mimedefang process is 8GB which seems large.
I'll investigate to see if I can find a memory leak.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mai

Robert Theisen

2017-11-10 18:18:15 UTC

Permalink

Something that might help. I noticed that the format of the string in
memory is ...

_ {auth_authen} {auth_author} {auth_ssf} {auth_type} {cert_issuer}
{cert_subject} {cipher} {cipher_bits} {daemon_name} {daemon_port} i
{if_addr} {if_name} j {mail_addr} {mail_host} {mail_mailer}
{tls_version} {verify} {rcpt_addr} {rcpt_host} {rcpt_mailer}

It leads with the _ and does have the curly braces around each item
longer than a single char.
It also terminates at {rcpt_mailer}

The curly braces are not added until the string is processed by
append_macro_value() and then immediately after append_macro_value()
runs, we add another value to dbuf called "client_port". But client
port does not show up in the memory dump.

/* Write the standard macros */
macro = StandardSendmailMacros;
while (*macro) {
append_macro_value(&dbuf, ctx, *macro);
macro++;
}

/* Fake client_port: We don't get the macro, but we have the connection
info cached in our private data area. */
dbuf_putc(&dbuf, '=');
append_percent_encoded(&dbuf, "client_port");

Is it possible that during this function ...

static void
append_macro_value(dynamic_buffer *dbuf,
SMFICTX *ctx,
char *macro)
{
struct privdata *data;
char *val;
char buf[256];

data = DATA;
if (!data) return;

if (*macro && *(macro+1)) {
/* Longer than 1 char -- use curlies */
snprintf(buf, sizeof(buf), "{%s}", macro);
val = smfi_getsymval(ctx, buf);
} else {
val = smfi_getsymval(ctx, macro);
}
if (!val) return;
dbuf_putc(dbuf, '=');
append_percent_encoded(dbuf, macro);
dbuf_putc(dbuf, ' ');
append_percent_encoded(dbuf, val);
dbuf_putc(dbuf, '\n');
}

at snprintf(buf, sizeof(buf), "{%s}", macro);

that buf[256] is not getting cleaned up even though it is a local
variable that is falling out of scope?

The strike against that theory would be that my memory dump shows that
the "i" and "j" variables are included in that memory chunk. But the
single character i and j values are never copied to the local buf[256]
variable.

Post by Dianne Skoll
On Fri, 10 Nov 2017 10:33:10 -0500
[snip]

if (data->cmdFD < 0) {
dbuf_free(&dbuf);
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
if (write_dbuf(&dbuf, data->cmdFD, data, "COMMANDS") < 0) {
dbuf_free(&dbuf);
cleanup(ctx);
DEBUG_EXIT("envfrom", "SMFIS_TEMPFAIL");
return SMFIS_TEMPFAIL;
}
dbuf_free(&dbuf);
and there are no branches or returns between the code you posted
and the code I posted above. However, I just checked one of our
busy servers and the mimedefang process is 8GB which seems large.
I'll investigate to see if I can find a memory leak.
Regards,
Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/lis

Dianne Skoll

2017-11-10 21:09:23 UTC

Permalink

On Fri, 10 Nov 2017 13:18:15 -0500

Post by Robert Theisen
{
struct privdata *data;
char *val;
char buf[256];
if (*macro && *(macro+1)) {
/* Longer than 1 char -- use curlies */
snprintf(buf, sizeof(buf), "{%s}", macro);
val = smfi_getsymval(ctx, buf);
} else {
val = smfi_getsymval(ctx, macro);
}
...
}
at snprintf(buf, sizeof(buf), "{%s}", macro);
that buf[256] is not getting cleaned up even though it is a local
variable that is falling out of scope?

Um what? There's nothing to clean up. CLeaning that up simply
involves the C compiler incrementing the stack pointer by sizeof(buf);
it's an automatic variable.

Post by Robert Theisen
The strike against that theory would be that my memory dump shows that
the "i" and "j" variables are included in that memory chunk. But the
single character i and j values are never copied to the local buf[256]
variable.

If you're linking against a version of libmilter older than 8.14.4, the
library has a memory leak in it that could be causing this.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
htt

Robert Theisen

2017-11-10 22:33:52 UTC

Permalink

sendmail-milter-8.14.7-5.el7.x86_64

[ext-mx09] [10:31:02 PM]
[***@ext-mx09 tmp]# rpm -qa | grep milter
sendmail-milter-8.14.7-5.el7.x86_64
[ext-mx09] [10:31:10 PM]
[***@ext-mx09 tmp]# rpm -ql sendmail-milter
/usr/lib64/libmilter.so.1.0
/usr/lib64/libmilter.so.1.0.1
/usr/share/doc/sendmail-8.14.7/README.libmilter
/usr/share/doc/sendmail-milter-8.14.7
/usr/share/doc/sendmail-milter-8.14.7/LICENSE
[ext-mx09] [10:32:37 PM]
[***@ext-mx09 tmp]# ldd /usr/bin/mimedefang
linux-vdso.so.1 => (0x00007ffc78ff7000)
libmilter.so.1.0 => /lib64/libmilter.so.1.0 (0x00007ff833eb6000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff833c9a000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007ff833a80000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff8336bd000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff8340d0000)
[ext-mx09] [10:32:45 PM]

Post by Dianne Skoll
On Fri, 10 Nov 2017 13:18:15 -0500

Um what? There's nothing to clean up. CLeaning that up simply
involves the C compiler incrementing the stack pointer by sizeof(buf);
it's an automatic variable.

If you're linking against a version of libmilter older than 8.14.4, the
library has a memory leak in it that could be causing this.
Regards,
Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang

Robert Theisen

2017-11-10 23:44:20 UTC

Permalink

Did you find anything of concern in the mimedefang process on your
host that was consuming 8GB RAM?
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
htt

Dianne Skoll

2017-11-12 16:30:14 UTC

Permalink

On Fri, 10 Nov 2017 18:44:20 -0500

Post by Robert Theisen
Did you find anything of concern in the mimedefang process on your
host that was consuming 8GB RAM?

No, and here's the weird thing. I attached using gdb and forced a
core dump. The core file was only 98MB in size. I have no idea what
would cause that.

The mail server is very busy and mimedefang would have hundreds of threads
going, so I can easily see a 98MB process image with all of the threads'
stacks.

I'm sorry, but this has me stumped.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://

Dianne Skoll

2017-11-10 20:52:45 UTC

Permalink

Hi again,

What version of libmilter are you linking against? There was a memory leak
in libmilter in versions older then 8.14.4.

https://groups.google.com/forum/#!topic/linux.debian.bugs.dist/I3QpUbpdwSM

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo