[Mimedefang] REVISED: postfix/mimedefang socket

[Michael Fox]

Sorry if this has been asked/answered before. I've searched and searched and found no consistent, complete answers.

What are the steps to configure MIMEDefang and Postix to share a socket?

Here's what I tried:

Option 1: Use inet socket
------------------------------------------------
/etc/default/mimedefang:
SOCKET=inet:***@localhost

/etc/postfix/main.cf
smtpd_milters = inet:localhost:8899

This works for Ubuntu 14.04/MIMEDefang 2.73.
But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already running, and MIMEDefang is restarted, I get the following in /var/log/mail.err:

mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port inet:***@localhost: Address already in use
mimedefang[27605]: MIMEDefang-2.78: Unable to create listening socket on conn inet:***@localhost


Option 2: Use unix socket in MIMEDefang directory
--------------------------------------------------
/etc/default/mimedefang:
<leave default SOCKET line (/var/spool/MIMEDefang/mimedefang.sock)>
MD_ALLOW_GROUP_ACCESS=yes

add user "postfix" to group "defang"
usermod -a -G defang postfix

/etc/postfix/main.cf:
smtpd_milters=unix:/var/spool/MIMEDefang/mimedefang.sock

This fails, presumably because Postfix is chroot'ed and can't access anything outside it's chroot directory (/var/spool/postfix). In /var/log/mail.err:

postfix/smtpd[31110]: warning: connect to Milter service unix:/var/spool/MIMEDefang/mimedefang.sock: No such file or directory


Option 3: Use unix socket in Postfix chroot jail
-------------------------------------------------
I suppose I could configure /etc/default/mimedefang:SOCKET to point to a file in /var/spool/postfix/private, where the other sockets are located. But that directory is owned by postfix, group=root, permissions 0700. So I don't know how mimedefang could write to it.


So, I'm stuck.

Any help would be appreciated.
Michael




_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://l

[Richard Laager]

Post by Michael Fox
Option 3: Use unix socket in Postfix chroot jail

This looks to be what I do. I'm running Postfix and MIMEDefang on
Ubuntu, both from packages. Postfix runs as the postfix user, and
there's a defang group. I run Postfix in a chroot.

These appear to be the relevant parts of my install script:

adduser --quiet postfix defang

install -d -o defang -g defang -m 750 \
/var/spool/postfix/var/spool/MIMEDefang

chown -R defang:defang \
/var/lib/MIMEDefang \
/var/spool/MIMEDefang \
/var/spool/postfix/var/spool/MIMEDefang

sed -i 's|^\(# \)\?\(SOCKET\)=.*|\2=/var/spool/postfix/var/spool/MIMEDefang/mimedefang.sock|' \
/etc/default/mimedefang

I believe we have some sort of trouble if one of the daemons is
restarted, but not the other, or if it's done in the wrong order or
something. I don't have a lot of specifics off the top of my head. In
practice, we hardly ever restart one or the other. It's usually either
stopping both (and starting MIMEDefang first, to give slaves a chance to
spin up), or rebooting the server.

I hope this helps. If you have specific questions, I'll try to dig into
my config if I can. I'm currently out of the office, though.

[Michael Fox]

Thanks Richard,

BTW, I also installed from packages.

So, with a unix socket you still have the same problem I do with the inet socket, which is: the order of start/restart matters and can break things. That's not good.

This seems like a clear bug in MIMEDefang.

Evidence:
1) Other milter(s) share a socket with Postfix and don't care which order they are started/restarted (example: OpenDKIM)
2) Dovecot shares a socket with Postfix and doesn't care which order they are started/restarted
3) Amavisd-new is a different mechanism, but it also doesn't care which order it is started/restarted
4) And even MIMEDefang didn't care about the order in v2.73

I'm new to the list. What's the process for reporting/resolving issues?

Thanks,
Michael

-----Original Message-----
Sent: Friday, September 22, 2017 9:51 PM
Subject: Re: [Mimedefang] REVISED: postfix/mimedefang socket

Post by Michael Fox
Option 3: Use unix socket in Postfix chroot jail

This looks to be what I do. I'm running Postfix and MIMEDefang on
Ubuntu, both from packages. Postfix runs as the postfix user, and
there's a defang group. I run Postfix in a chroot.
adduser --quiet postfix defang
install -d -o defang -g defang -m 750 \
/var/spool/postfix/var/spool/MIMEDefang
chown -R defang:defang \
/var/lib/MIMEDefang \
/var/spool/MIMEDefang \
/var/spool/postfix/var/spool/MIMEDefang
sed -i 's|^\(#
\)\?\(SOCKET\)=.*|\2=/var/spool/postfix/var/spool/MIMEDefang/mimedefang.so
ck|' \
/etc/default/mimedefang
I believe we have some sort of trouble if one of the daemons is
restarted, but not the other, or if it's done in the wrong order or
something. I don't have a lot of specifics off the top of my head. In
practice, we hardly ever restart one or the other. It's usually either
stopping both (and starting MIMEDefang first, to give slaves a chance to
spin up), or rebooting the server.
I hope this helps. If you have specific questions, I'll try to dig into
my config if I can. I'm currently out of the office, though.
--
Richard

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mime

[Dianne Skoll]

On Fri, 22 Sep 2017 23:59:38 -0700

Post by Michael Fox
This seems like a clear bug in MIMEDefang.

Not at all. This problem is not present with Sendmail, so I have no
idea what would cause it with Postfix. Removing the MIMEDefang socket
prior to starting MIMEDefang should fix this. The sample init script
given with MIMEDefang does exactly this.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimede

[Bill Cole]

Post by Michael Fox
Sorry if this has been asked/answered before. I've searched and
searched and found no consistent, complete answers.
What are the steps to configure MIMEDefang and Postix to share a socket?

The workable solutions depend on your OS (and with Linux, your
distribution,) your filesystem, and whether you chroot the Postfix smtpd
program.

Post by Michael Fox
Option 1: Use inet socket
------------------------------------------------
/etc/postfix/main.cf
smtpd_milters = inet:localhost:8899
This works for Ubuntu 14.04/MIMEDefang 2.73.
But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already
running, and MIMEDefang is restarted, I get the following in
mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port
mimedefang[27605]: MIMEDefang-2.78: Unable to create listening

That's odd. Given your other options, I'd try to figure out what exactly
is going on here and fix it. I.e. what process actually has taken that
port.

As for using a unix socket, you don't need to create the socket inside
the chroot if it's on the same filesystem as /var/spool/MIMEDefang/ ,
you can just hardlink the existing socket.

Also, you may not need to add the postfix user to the defang group, if
you have a filesystem that supports BSD default group semantics for
setgid directories, e.g.:

chgrp postfix /var/spool/MIMEDefang/
chmod g+ws /var/spool/MIMEDefang/

With the setgid bit set on a directory, newly created files (or sockets)
take their group from the parent directory rather than the creating
process.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists

[Michael Fox]

Thanks Bill,

After some more research I found the bug and I think I've "fixed" it. See below.

Post by Bill Cole

Post by Michael Fox
Option 1: Use inet socket
------------------------------------------------
/etc/postfix/main.cf
smtpd_milters = inet:localhost:8899
This works for Ubuntu 14.04/MIMEDefang 2.73.
But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already
running, and MIMEDefang is restarted, I get the following in
mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port
mimedefang[27605]: MIMEDefang-2.78: Unable to create listening

That's odd. Given your other options, I'd try to figure out what exactly
is going on here and fix it. I.e. what process actually has taken that
port.

Yes, odd.

I've continued to research and just found this, which is my problem.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807078

The problem occurs for me in MIMEDefang 2.78 (package for Ubuntu 16.04), which is the version reported in the above bug. Apparently, the problem is that when using "systemctl restart mimedefang" it doesn't wait for itself to completely stop before it tries to start again. And I can confirm that. If I use "systemctl stop ...", then wait for it to stop, then "systemctl start ...", then it works.

The bug report says it's fixed in v2.79 and v2.79 is in the Ubuntu 17.04 package.
According to https://ubuntu.pkgs.org/17.04/ubuntu-universe-i386/mimedefang_2.79-2_i386.deb.html , on 2016-12-23 the init stop action was changed to always wait.

So here's what I did to "fix" it in Ubuntu 16.04/MIMEDefang 2.78:

In /etc/init.d/mimedefang, in the stop_it() function:

# if [ "$1" = "wait" ] ; then
....
# fi

In other words, just comment out the conditional wait so that the "waiting" part inside the IF statement always runs.

When I tried to run systemctl restart mimedefang, I got:
Warning: mimedefang.service changed on disk. Run 'systemctl daemon-reload' to reload units.

So, I did that. And then: systemctl restart mimedefang seems to work just fine.

I'd appreciate it if any of the developers can verify that what I did shouldn't cause some other unintended consequence.

Thanks much,
Michael


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinf

[Bill Cole]

Post by Michael Fox
Thanks Bill,
After some more research I found the bug and I think I've "fixed" it.
See below.

Post by Bill Cole

Post by Michael Fox
Option 1: Use inet socket
------------------------------------------------
/etc/postfix/main.cf
smtpd_milters = inet:localhost:8899
This works for Ubuntu 14.04/MIMEDefang 2.73.
But it fails with Ubuntu 16.04/MIMEDefang 2.78. If Postfix is already
running, and MIMEDefang is restarted, I get the following in
mimedefang[27605]: MIMEDefang-2.78: Unable to bind to port
mimedefang[27605]: MIMEDefang-2.78: Unable to create listening

That's odd. Given your other options, I'd try to figure out what exactly
is going on here and fix it. I.e. what process actually has taken that
port.

Yes, odd.
I've continued to research and just found this, which is my problem.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807078
The problem occurs for me in MIMEDefang 2.78 (package for Ubuntu
16.04), which is the version reported in the above bug. Apparently,
the problem is that when using "systemctl restart mimedefang" it
doesn't wait for itself to completely stop before it tries to start
again. And I can confirm that. If I use "systemctl stop ...", then
wait for it to stop, then "systemctl start ...", then it works.

I kinda wanted to blame the abomination of systemd and/or the
Debian/Ubuntu packagers who regularly 'fix' things that are not broken
but since I had nothing more solid than my own bile, I refrained...

Post by Michael Fox
The bug report says it's fixed in v2.79 and v2.79 is in the Ubuntu 17.04 package.
According to
https://ubuntu.pkgs.org/17.04/ubuntu-universe-i386/mimedefang_2.79-2_i386.deb.html
, on 2016-12-23 the init stop action was changed to always wait.

As the "restart" action has been in the distribution's included init
scripts for as long as I've used MD (over a decade) and I suspect
forever. There's no rationale I can think of to always wait on a 'stop'
action but it isn't really harmful, beyond delaying shutdown for
(normally) a few seconds.

Post by Michael Fox
# if [ "$1" = "wait" ] ; then
....
# fi
In other words, just comment out the conditional wait so that the
"waiting" part inside the IF statement always runs.
Warning: mimedefang.service changed on disk. Run 'systemctl
daemon-reload' to reload units.
So, I did that. And then: systemctl restart mimedefang seems to work just fine.
I'd appreciate it if any of the developers can verify that what I did
shouldn't cause some other unintended consequence.

Dianne will likely give a definitive authoritative answer but having
adapted the MD startup script to multiple system control environments, I
don't see anything especially risky. The only potential issue I can
think of is if you're trying to shut down or reboot in a pathological
circumstance where the mimedefang.pl worker perl sub-processes refuse to
exit and the supervisory process (e.g. init) has no last resort
mechanism to arbitrarily slaughter hung children. There's nothing quite
so annoying as a "hung going down" system.

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roarin

[Michael Fox]

Post by Bill Cole
Dianne will likely give a definitive authoritative answer but having
adapted the MD startup script to multiple system control environments, I
don't see anything especially risky. The only potential issue I can
think of is if you're trying to shut down or reboot in a pathological
circumstance where the mimedefang.pl worker perl sub-processes refuse to
exit and the supervisory process (e.g. init) has no last resort
mechanism to arbitrarily slaughter hung children. There's nothing quite
so annoying as a "hung going down" system.

Thanks Bill,

That makes sense.

One option would be for the MIMEDefang team to provide the necessary systemd control files (whatever they're called). That way, they could define restart any way they like, such as "stop wait" followed by "start". I know systemd is hated by many. But it is what Ubuntu 16.04 LTS uses.

Without the proper files, apparently what happens is systemd auto-generates the file(s) it needs from the init script. And in the autogenerated file, a 'restart' is simply a 'stop' followed by a 'start'. I guess they can depend on at least those two basic commands being in every init script. And because 'stop' doesn't wait unless the added "wait" option is given, we have the problem.

A friend of mine found the actual "fix" that was submitted. It's a bit different than what I did. Instead of commenting out the if statement in stop_it() like I did, they changed the line that calls stop_it from "stop_ip $2" to "stop_it wait", so stop_it is always called with the "wait" option. See below. Same result - the (formerly conditional) wait part always runs. 1 line changed instead of 2.

Michael


*** mimedefang-16.04 2017-09-23 17:53:51.046863700 -0700
--- mimedefang-17.04 2017-09-23 17:50:18.456864400 -0700
***************
*** 412,418 ****
;;

stop)
! stop_it $2
;;

restart|force-reload)
--- 412,418 ----
;;

stop)
! stop_it wait
;;

restart|force-reload)


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailm

[Bill Cole]

Post by Michael Fox

Post by Bill Cole
Dianne will likely give a definitive authoritative answer but having
adapted the MD startup script to multiple system control
environments, I
don't see anything especially risky. The only potential issue I can
think of is if you're trying to shut down or reboot in a pathological
circumstance where the mimedefang.pl worker perl sub-processes refuse to
exit and the supervisory process (e.g. init) has no last resort
mechanism to arbitrarily slaughter hung children. There's nothing quite
so annoying as a "hung going down" system.

Thanks Bill,
That makes sense.
One option would be for the MIMEDefang team to provide the necessary
systemd control files (whatever they're called).

They are called "unit files" but including one in the base code would
not help.

Post by Michael Fox
That way, they could define restart any way they like, such as "stop
wait" followed by "start".

Nope. A service definition in a unit file has an ExecStart definition
and maybe an ExecStop definition, but there is no support in systemd for
an ExecRestart attribute. It's a design flaw in systemd, which ALWAYS
implements "systemctl restart" as a stop then a start.

Post by Michael Fox
I know systemd is hated by many.

Don't get me started...

Post by Michael Fox
But it is what Ubuntu 16.04 LTS uses.

So it's the Ubuntu/Debian package maintainer's job to create a suitable
unit file just as they've created their own init script.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http

[Michael Fox]

Post by Bill Cole

Post by Michael Fox
That way, they could define restart any way they like, such as "stop
wait" followed by "start".

Nope. A service definition in a unit file has an ExecStart definition
and maybe an ExecStop definition, but there is no support in systemd for
an ExecRestart attribute. It's a design flaw in systemd, which ALWAYS
implements "systemctl restart" as a stop then a start.

Ah. I guess I was thinking of reload.

Michael


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists

[Richard Laager]

Post by Bill Cole
A service definition in a unit file has an ExecStart definition
and maybe an ExecStop definition

ExecStop must not return until the service has actually stopped:
https://www.freedesktop.org/software/systemd/man/systemd.service.html#ExecStop=

There are a couple ways to address this. As discussed, the init script
can be changed to always wait on stop. Then the automatic mapping in
systemd should Just Work.

Otherwise, as discussed, we need to create a unit file.

For a temporary measure, if one is willing to depend on the SysV init
script, it should be possible to do something simple like:
ExecStop=/etc/init.d/mimedefang stop ; /etc/init.d/mimedefang wait

The fully native approach would be to include (either directly, or in a
small helper script) the wait loop (but not the status nor the killing
bits, as systemd handles those things).

Unless someone beats me to it, I'll whip up a systemd unit file in the
next few days, test it, and submit a patch to Debian.

[Dianne Skoll]

On Mon, 25 Sep 2017 16:08:46 -0500

Post by Richard Laager
There are a couple ways to address this. As discussed, the init script
can be changed to always wait on stop. Then the automatic mapping in
systemd should Just Work.

The sample init script that ships with MIMEDefang does wait on stop.
If you have a script that doesn't, that's a distribution bug and not a
MIMEDefang bug and it's definitely more productive to take this up with
the distribution maintainer.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinf

[Bill Cole]

Post by Dianne Skoll
On Mon, 25 Sep 2017 16:08:46 -0500

Post by Richard Laager
There are a couple ways to address this. As discussed, the init script
can be changed to always wait on stop. Then the automatic mapping in
systemd should Just Work.

The sample init script that ships with MIMEDefang does wait on stop.
If you have a script that doesn't, that's a distribution bug and not a
MIMEDefang bug and it's definitely more productive to take this up with
the distribution maintainer.

What am I missing? This is in a pristine MD distribution tree:

bigsky:mimedefang-2.82 bill$ grep -A3 'stop)' examples/init-script.in
redhat/mimedefang-init.in
examples/init-script.in: stop)
examples/init-script.in- stop_it $2
examples/init-script.in- ;;
examples/init-script.in-
--
redhat/mimedefang-init.in: stop)
redhat/mimedefang-init.in- stop $2
redhat/mimedefang-init.in- RETVAL=$?
redhat/mimedefang-init.in- ;;
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpen

[Dianne Skoll]

On Tue, 26 Sep 2017 09:38:18 -0400

Post by Bill Cole
bigsky:mimedefang-2.82 bill$ grep -A3 'stop)'

The stop_it function in init-script.in and the stop function in
redhat/mimedefang-init.in both wait for the daemons to exit.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/li

[Bill Cole]

Post by Dianne Skoll
On Tue, 26 Sep 2017 09:38:18 -0400

Post by Bill Cole
bigsky:mimedefang-2.82 bill$ grep -A3 'stop)'

The stop_it function in init-script.in and the stop function in
redhat/mimedefang-init.in both wait for the daemons to exit.

I'm uneasy telling any author that they're wrong about their own code
but as far as I can tell, that is only conditionally true:

$ grep -C2 -n Waiting examples/init-script.in
redhat/mimedefang-init.in
examples/init-script.in-331-
examples/init-script.in-332- if [ "$1" = "wait" ] ; then
examples/init-script.in:333: printf "Waiting for daemons to
exit."
examples/init-script.in-334- WAITPID=""
examples/init-script.in-335- test -f $PID && WAITPID=`cat
$PID`
--
redhat/mimedefang-init.in-227-
redhat/mimedefang-init.in-228- if [ "$1" = "wait" ] ; then
redhat/mimedefang-init.in:229: printf "Waiting for daemons to
exit"
redhat/mimedefang-init.in-230- WAITPID=""
redhat/mimedefang-init.in-231- test -f /var/run/$prog.pid &&
WAITPID=`cat /var/run/$prog.pid`

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mime

[Dianne Skoll]

On Tue, 26 Sep 2017 15:48:42 -0400

Post by Bill Cole
I'm uneasy telling any author that they're wrong about their own code

Oh dear... you are correct. :)

Seeing as I don't use those scripts myself, I've never encountered the
problem. :(

Mea culpa.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://list

[Michael Fox]

Post by Bill Cole
I'm uneasy telling any author that they're wrong about their own code

Bill, you and Dianne are both right, depending on the version.

See my previous message. I provided a diff between v2.78 and v2.79.

In v2.79, 'wait' is always passed to stop_it, so the if statement you showed will always evaluate to true

But in v2.78, which is where I had my problem, the main part of the script passes $2 to stop_it. So stop_it may or may not see "wait".


Michael


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mail

[Bill Cole]

Post by Michael Fox
Bill, you and Dianne are both right, depending on the version.
See my previous message. I provided a diff between v2.78 and v2.79.

That's only in the Debian variant, not Dianne's original code.
Everything in the top-level 'debian' directory of the source package for
Debian (and Debian derivatives like Ubuntu) is created and maintained by
the Debian packager. I tried to figure out where they inserted the bug,
but it seems to have been in a version control gap between 2.75 and 2.78
(or maybe between Debian and Ubuntu???) that I don't understand. It may
simply be that in earlier versions the distribution init scripts'
"restart" command was used directly rather than generating a generic
systemd unit file that synthesizes restart with plain "start" and "stop"
commands.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http:/

[Michael Fox]

Post by Bill Cole
It may
simply be that in earlier versions the distribution init scripts'
"restart" command was used directly rather than generating a generic
systemd unit file that synthesizes restart with plain "start" and "stop"
commands.

That makes sense.

In my own case, I started with MD on Ubuntu 14.04 (MD v2.73). The documentation I found said to use "/etc/init.d/mimedefang reread" after changes to the filter file. But that had no effect (that I could see). So I started using "/etc/init.d/mimedefang restart".

Once I got things working, I installed on Ubuntu 16.04 (v2.78). Ubuntu 16.04 uses systemd. So I simply converted to "systemctl restart mimedefang". And that's when I saw the problem caused by no "wait"ing.

BTW, in Ubuntu 16.04:
-- systemctl reread mimedefang results in "Unknown operation reread"
-- systemctl reload mimedefang does run. But I haven't tested to see if it actually picks up the changes to the filter file.

I wonder: Is the Debian package maintainer on this list? We seem to be speculating on questions that the packager could answer definitively.

Michael




_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpeng

[Richard Laager]

For those of you that don't use systemd, you can just ignore this. None
of this affects the SysV init scripts in any way.

Here's a first run at the diff from the generated unit to a manual unit
to fix the issue. The relevant change is making ExecStop use "stop wait"
instead of just "stop".

The init script (at least on my system, which is Ubuntu 16.04) does not
do a "wait" on stop, but only on restart. That arguably seems fine in
the SysV world.

The other changes I made in the [Service] section weren't required for
this particular problem, but seem like things we'd want.

This fixes the problem for me.

I think a more correct solution for systemd is to make this two units,
one for mimedefang and one for mimedefang-multiplexor. I'm going to work
on that next, before submitting anything to Debian.

--- /run/systemd/generator.late/mimedefang.service 2017-09-16 01:25:06.772000000 -0500
+++ /lib/systemd/system/mimedefang.service 2017-09-28 18:41:38.651481894 -0500
@@ -1,10 +1,5 @@
-# Automatically generated by systemd-sysv-generator
-
[Unit]
-Documentation=man:systemd-sysv-generator(8)
-SourcePath=/etc/init.d/mimedefang
-Before=multi-user.target
-Before=multi-user.target
+Documentation=man:mimedefang(8)
Before=multi-user.target
Before=graphical.target
Before=shutdown.target
@@ -14,12 +9,11 @@

[Service]
Type=forking
-Restart=no
+Restart=on-failure
TimeoutSec=5min
-IgnoreSIGPIPE=no
-KillMode=process
+KillMode=control-group
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/mimedefang start
-ExecStop=/etc/init.d/mimedefang stop
+ExecStop=/etc/init.d/mimedefang stop wait
ExecReload=/etc/init.d/mimedefang reload

[Richard Laager]

I have attached fully split, native, Type=simple (not forking) unit
files, which I have tested on Ubuntu 16.04.

They honor the settings in /etc/default/mimedefang. This includes the
MX_USER setting (i.e. systemd does not directly control the service user).

They are configured such that if you do start/stop/reload/restart the
mimedefang service (as before), the actions propagate to
mimedefang-multiplexor appropriately.

Some questions (for Dianne?):

1) Is it necessary to rm $SOCKET and $MX_SOCKET *before starting*?

2) Is it necessary to rm $SOCKET and $MX_SOCKET *after stopping*?

3) The init script (at least on Ubuntu) says, '# Locale should be set to
"C" for generating valid date headers'. I see a strftime() in
mimedefang-multiplexor.c. Plus, the Perl code inherits from the
multiplexor, so I can understand this. Is setting LC_ALL=C necessary for
the mimedefang process too, or just mimedefang-multiplexor?

4) The mimedefang-multiplexor process exits with status 1 on SIGTERM. I
am absolutely sure (because I'm doing it by hand for testing) that I am
only sending a SIGTERM, and only to the main mimedefang-multiplexor
process (not the children). It logs 'Received SIGTERM: Stopping slaves
and terminating' and then reaps the slaves as expected; there are no
errors in the log. Is that exit status expected?

I have built the unit files assuming the following answers:
1) Yes.
2) Yes.
3) Not necessary for mimedefang, only mimedefang-multiplexor.
4) "Yes", as this happens (whether expected or not).

[Dianne Skoll]

On Thu, 28 Sep 2017 23:11:18 -0500

Post by Richard Laager
1) Is it necessary to rm $SOCKET and $MX_SOCKET *before starting*?

It's best to do so because if mimedefang or mimedefang-multiplexor
crashes, the socket files will be left lying around and startup will likely
fail.

Post by Richard Laager
2) Is it necessary to rm $SOCKET and $MX_SOCKET *after stopping*?

It's best to do so in order that stale socket files are not left
lying around.

Post by Richard Laager
3) The init script (at least on Ubuntu) says, '# Locale should be set
to "C" for generating valid date headers'. I see a strftime() in
mimedefang-multiplexor.c. Plus, the Perl code inherits from the
multiplexor, so I can understand this. Is setting LC_ALL=C necessary
for the mimedefang process too, or just mimedefang-multiplexor?

I don't know; it's probably safer for both.

Post by Richard Laager
4) The mimedefang-multiplexor process exits with status 1 on SIGTERM.

This is a bug that was corrected in MIMEDefang 2.82.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/ma

[Richard Laager]

Post by Richard Laager
I have attached fully split, native, Type=simple (not forking) unit
files, which I have tested on Ubuntu 16.04.

I have submitted the slightly updated versions to Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877663

Dianne, you may want to ship these (or similar) upstream, but that's
obviously up to you. It doesn't make any difference to me personally,
since I use the downstream packaging.

[Dianne Skoll]

On Wed, 4 Oct 2017 18:58:39 -0500

Post by Richard Laager
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877663
Dianne, you may want to ship these (or similar) upstream, but that's
obviously up to you. It doesn't make any difference to me personally,
since I use the downstream packaging.

I might as well put it in upstream. Thanks for the work!

Regards,

Dianne.

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/li

[Richard Laager]

I ran into a problem with my systemd units for MIMEDefang. There is a
race condition between mimedefang-multiplexor creating the socket and
mimedefang trying to access it. If the multiplexor doesn't create the
socket in time, mimedefang fails on startup.

This is happening because Type=simple services are assumed to be running
immediately. This is documented in systemd.service(5), which says, "In
this mode, if the process offers functionality to other processes on the
system, its communication channels should be installed before the daemon
is started up (e.g. sockets set up by systemd, via socket activation),
as systemd will immediately proceed starting follow-up units."

I have attached updated versions to correct this. These change to
Type=forking and drop the -D flag. With the change to Type=forking, I
have also brought back pid files. Since systemd requires the pid file
path to be hardcoded (as the argument to PIDFile), I put it in /run
rather than the $SPOOLDIR (which the administrator can change). It can't
be directly in /run, as the daemon writes the pidfile after dropping
privileges, so we have to create a /run/mimedefang subdirectory owned by
$MX_USER. This requires some indirection, as $MX_USER can also be
customized by the administrator. I only create this directory for
mimedefang-multiplexor.service (and not mimedefang.service), as
mimedefang.service is already ordered after mimedefang-multiplexor.service.

Long term, the ideal answer is to use systemd's socket activation. This
would allow the services to startup in parallel. Another alternative
would be to use Type=notify. Either requires changes to the mimedefang C
code.

[Richard Laager]

I have attached updated systemd service files, with the PID file
directly in /run, which I have tested with 2.84. This change also
requires specifying a lock file in $SPOOLDIR.

If you prefer a diff rather than new files, I also attached a diff
against 2.84.

Post by Richard Laager
I ran into a problem with my systemd units for MIMEDefang. There is a
race condition between mimedefang-multiplexor creating the socket and
mimedefang trying to access it. If the multiplexor doesn't create the
socket in time, mimedefang fails on startup.
This is happening because Type=simple services are assumed to be running
immediately. This is documented in systemd.service(5), which says, "In
this mode, if the process offers functionality to other processes on the
system, its communication channels should be installed before the daemon
is started up (e.g. sockets set up by systemd, via socket activation),
as systemd will immediately proceed starting follow-up units."
I have attached updated versions to correct this. These change to
Type=forking and drop the -D flag. With the change to Type=forking, I
have also brought back pid files. Since systemd requires the pid file
path to be hardcoded (as the argument to PIDFile), I put it in /run
rather than the $SPOOLDIR (which the administrator can change).

This part no longer applies, as of MIMEDefang 2.83, because it writes

Post by Richard Laager
It can't
be directly in /run, as the daemon writes the pidfile after dropping
privileges, so we have to create a /run/mimedefang subdirectory owned by
$MX_USER. This requires some indirection, as $MX_USER can also be
customized by the administrator. I only create this directory for
mimedefang-multiplexor.service (and not mimedefang.service), as
mimedefang.service is already ordered after mimedefang-multiplexor.service.

[Dianne Skoll]

On Mon, 23 Jul 2018 03:31:06 -0500

Post by Richard Laager
I have attached updated systemd service files, with the PID file
directly in /run, which I have tested with 2.84. This change also
requires specifying a lock file in $SPOOLDIR.

Thanks! I'll put these in the next release.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http