Discussion:
[Mimedefang] utf-8 issue?
Mark Coetser
2017-12-12 13:43:14 UTC
Permalink
Hi

I have 4 different mail hubs, all running latest debian

ii perl 5.24.1-3+deb9u2 i386
Larry Wall's Practical Extraction and Report Language
ii mimedefang 2.79-2 i386
e-mail filter program for sendmail

Recently I am seeing alot of the following errors

Error from multiplexor: ERR No response from slave
Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED
UNEXPECTEDLY)

The above are logged in mail.err and are related to the following in
mail.log which rejects the email with a "try again later" message until
the email eveentually bounces after 5 days.

stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm
line 1878.

A google search shows this topic


https://lists.roaringpenguin.com/pipermail/mimedefang/2013-February/036880.html


But I am not doing anything with MIME::Entity; and the filter is pretty
much the stock microsoft mimedefang-filter and this does seem to be
related from the upgrade from Debian Jessie to Stretch, does anyone have
any pointers?
--
Thank you,

Mark Adrian Coetser
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/ma
Dianne Skoll
2017-12-12 18:03:38 UTC
Permalink
On Tue, 12 Dec 2017 15:43:14 +0200
Post by Mark Coetser
Error from multiplexor: ERR No response from slave
Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED
UNEXPECTEDLY)
I've never seen this before. I'm also not convinced it's related
to the UTF-8 issue. Could you post the exact filter you are using?

I'm also running on Stretch, btw.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailma
Mark Coetser
2017-12-13 07:25:57 UTC
Permalink
Post by Dianne Skoll
On Tue, 12 Dec 2017 15:43:14 +0200
Post by Mark Coetser
Error from multiplexor: ERR No response from slave
Reap: slave 1 (pid 15022) exited normally with status 22 (SLAVE DIED
UNEXPECTEDLY)
I've never seen this before. I'm also not convinced it's related
to the UTF-8 issue. Could you post the exact filter you are using?
I'm also running on Stretch, btw.
Regards,
Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
filter attached, I am doing some other stuff in there but as I said this
filter has been in place for many years and only recently have those
errors been occurring...


Thank you,

Mark Adrian Coetser
Bill Cole
2017-12-12 22:09:35 UTC
Permalink
Post by Mark Coetser
Hi
I have 4 different mail hubs, all running latest debian
ii perl 5.24.1-3+deb9u2 i386
Larry Wall's Practical Extraction and Report Language
ii mimedefang 2.79-2 i386
e-mail filter program for sendmail
[...]
Post by Mark Coetser
stderr: open body: Invalid argument at /usr/share/perl5/MIME/Entity.pm
line 1878.
A google search shows this topic
https://lists.roaringpenguin.com/pipermail/mimedefang/2013-February/036880.html
I doubt that's related.
Post by Mark Coetser
But I am not doing anything with MIME::Entity;
Not explicitly, but MD uses it (and the rest of MIME-Tools) extensively.

I'd bet this is the same as this bug:
https://rt.cpan.org/Public/Bug/Display.html?id=105377

Which remains open but maybe not unfixed.
Post by Mark Coetser
and the filter is pretty much the stock microsoft mimedefang-filter
and this does seem to be related from the upgrade from Debian Jessie
to Stretch, does anyone have any pointers?
It couldn't hurt to update MIME-Tools and it is certain that you're not
up-to-date with it, since line 1878 of the current Entity.pm is blank.
That error message would only be generated by line 1892 in the current
version. My bet is that the typo fix cited in the Changelog for 5.507
also fixed this...
--
Bill Cole
***@scconsult.com or ***@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpengu
Bill Cole
2017-12-13 21:28:58 UTC
Permalink
[...]
Post by Bill Cole
https://rt.cpan.org/Public/Bug/Display.html?id=105377
Which remains open but maybe not unfixed.
Post by Mark Coetser
and the filter is pretty much the stock microsoft mimedefang-filter
and this does seem to be related from the upgrade from Debian Jessie
to Stretch, does anyone have any pointers?
It couldn't hurt to update MIME-Tools and it is certain that you're
not up-to-date with it, since line 1878 of the current Entity.pm is
blank. That error message would only be generated by line 1892 in the
current version. My bet is that the typo fix cited in the Changelog
for 5.507 also fixed this...
line 1878 in my /usr/share/perl5/MIME/Entity.pm is the following
my $IO = $self->open("r") || die "open body: $!";
Indicating that you are using the same version of MIME-Tools as the
person who opened Bug #105377, which was 5.506.

Unfortunately, I tested a bit more and found that bug is still extant in
5.509, when tested with the one-liner in that bug report.

Hopefully Dianne will note that. This could be a different issue
generating the same error in the same place OR it could be that she has
fixed something since which mostly avoids triggering it. It would still
make sense to update to MIME-Tools 5.509 and MIMEDefang 2.83 to rule out
the possibility that this is a fixed issue with the current versions.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpe
Dianne Skoll
2017-12-13 21:39:09 UTC
Permalink
On Wed, 13 Dec 2017 16:28:58 -0500
Post by Bill Cole
Unfortunately, I tested a bit more and found that bug is still extant
in 5.509, when tested with the one-liner in that bug report.
That "bug" is a WONTFIX. You can NOT feed MIME::Entity->build()
data with raw characters > 0xFF. It doesn't make sense because
MIME messages are alway 8-bit messages; you need to encode everything
as UTF-8 first before passing to MIME::Entity->build(). I should
comment on the ticket. The correct way to build the entity
would be:

use Encode;
use MIME::Entity;

my $data = "\x{1f4a9}";
my $e = MIME::Entity->build(Data => Encode::encode('UTF-8', $data, Encode::FB_CROAK));
$e->print();

So the question is... how on Earth are characters > 0xFF getting passed
to MIME::Entity->build() from within mimedefang.pl?

I will close the bug on CPAN.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
htt
Bill Cole
2017-12-14 01:37:36 UTC
Permalink
Post by Dianne Skoll
On Wed, 13 Dec 2017 16:28:58 -0500
Post by Bill Cole
Unfortunately, I tested a bit more and found that bug is still extant
in 5.509, when tested with the one-liner in that bug report.
That "bug" is a WONTFIX.
Fair enough. I guess the answer to that specific bug report is in fact
"DON'T DO THAT!"

However, I was wondering how "Invalid Argument" was spitting out in that
particular place with 2 possibly different root causes, so I spent some
of my copious free time in the perl debugger and came up with this patch
(also added to the bug ticket):

--- Body.pm.old 2017-04-05 12:55:57.000000000 -0400
+++ Body.pm.new 2017-12-13 19:04:27.000000000 -0500
@@ -139,6 +139,7 @@
### System modules:
use Carp;
use IO::File;
+use IO::Scalar;

### The package version, both in 1.23 style *and* usable by MakeMaker:
$VERSION = "5.509";
@@ -522,7 +523,7 @@
die "bad mode: $mode";
}

- return IO::File->new(\ $self->{MBS_Data}, $mode);
+ return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
}

It is unclear to me how the original ever failed to fail, since
IO::File->new() really wants a filename or nothing and
MIME::Body->{MBS_Data} isn't a filename. With that patch and a slightly
modified one-liner (redirecting stderr because it emits a non-fatal
""Wide character in print"" warning) I get success:

bigsky:~ root# perl -MMIME::Entity \
Post by Dianne Skoll
-e 'MIME::Entity->build( Data => "\x{1f4a9}\n" )->print' \
2>/dev/null && echo "Success!"
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.509 (Entity 5.509)

💩
Success!

(I'm bracing for either a terse tutorial on how That's Not How This All
Works or a terser "Wrong!")

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/
Dianne Skoll
2017-12-14 14:14:12 UTC
Permalink
On Wed, 13 Dec 2017 20:37:36 -0500
Post by Bill Cole
- return IO::File->new(\ $self->{MBS_Data}, $mode);
+ return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
This will cause other problems down the line. I suggest you
study the section "Byte and Character Semantics" in the perlunicode
man page.

Modern Perl does let you open a "file" by passing a reference
to a scalar; it has built-in in-memory I/O.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpeng
Bill Cole
2017-12-14 22:26:38 UTC
Permalink
Post by Dianne Skoll
On Wed, 13 Dec 2017 20:37:36 -0500
Post by Bill Cole
- return IO::File->new(\ $self->{MBS_Data}, $mode);
+ return IO::Scalar->new(\ $self->{MBS_Data}, $mode);
This will cause other problems down the line. I suggest you
study the section "Byte and Character Semantics" in the perlunicode
man page.
Thanks for the reference and for your patience with my confusion. I
understand much better now: anything expressed as a string has to make
sense as a series of encoded characters, not a series of bytes.

And as `strings PerlIO/scalar/scalar.bundle` says: "Strings with code
points over 0xFF may not be mapped into in-memory file handles"
Post by Dianne Skoll
Modern Perl does let you open a "file" by passing a reference
to a scalar; it has built-in in-memory I/O.
Indeed. I was fooled by the fact that deep inside the 'open' call stack,
both $! and $^E get set to "No such file or directory" but that's
because XSLoader looks for a .bs while loading PerlIO::scalar...

I couldn't find exactly where the "Invalid Argument" error (and matching
numeric errno) is being generated, since it's in a debug-resistant
return that pops 4 stack layers in one step, but I'm perfectly happy to
believe that it's buried in PerlIO/scalar/scalar.bundle and is entirely
proper.
--
Bill Cole
***@scconsult.com or ***@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Currently Seeking Steady Work: https://linkedin.com/in/billcole
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpengu
Dianne Skoll
2017-12-15 14:09:28 UTC
Permalink
On Thu, 14 Dec 2017 17:26:38 -0500
Post by Bill Cole
Post by Dianne Skoll
This will cause other problems down the line. I suggest you
study the section "Byte and Character Semantics" in the perlunicode
man page.
Thanks for the reference and for your patience with my confusion. I
understand much better now: anything expressed as a string has to
make sense as a series of encoded characters, not a series of bytes.
Right. We completely overhauled our commercial CanIt software to
handle Unicode and it took me about four weeks to completely grok
how Perl handles Unicode and how the Encode module should be used.
It was quite painful. :)

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpengui

Bill Cole
2017-12-14 02:03:27 UTC
Permalink
Post by Dianne Skoll
It doesn't make sense because
MIME messages are alway 8-bit messages; you need to encode everything
as UTF-8 first before passing to MIME::Entity->build().
Re-reading that, I disagree.

It is entirely possible (as stated explicitly in RFC2045) for a MIME
entity to contain unencoded binary data: any arbitrary stream of bytes.
Email normally cannot, but that's a transport issue. Since the
BINARYMIME extension for SMTP exists, it is even possible to send mail
(with some MTAs) as an arbitrary stream of bytes.

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listi
Dianne Skoll
2017-12-14 14:10:30 UTC
Permalink
On Wed, 13 Dec 2017 21:03:27 -0500
Post by Bill Cole
It is entirely possible (as stated explicitly in RFC2045) for a MIME
entity to contain unencoded binary data: any arbitrary stream of bytes.
Stream of *bytes* yes. But Perl native characters > 0xFF are not bytes.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mi
Loading...