Discussion:
[Mimedefang] base64 to quoted-printable
Michael Fox
7 years ago
Permalink
I need to convert base64 to something readable. Per previous suggestions on
this list, I'm going to convert it to quoted-printable. So far, I'm using:



if ($bh = $entity->bodyhandle) ;

$qp_body = encode_qp($bh->as_string)

}



It seems to work (I use md_syslog() to show the first 20 chars). Is that
right? Or is there a better way?



Then, I see an action_change_header(), but no action_change_body(). So,
presume I need to build a new entity by copying headers from the old entity,
replacing the Content-Transfer-Encoding header, and replacing the body.
Then use replace_entire_message().



Does that sound about right?



Also, I've got to believe someone else has needed to do this before. But
I'm finding it tough to locate examples. I'm looking at docs for
mime::tools, mime::body, etc. but I'm hoping to locate examples of the
entire process. Any pointers?



Thanks,

Michael




_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/list
Joseph Brennan
7 years ago
Permalink
Post by Michael Fox
I need to convert base64 to something readable.
Also, I've got to believe someone else has needed to do this before.
To be honest I can't think of a reason to do this. Any client from
this century can read base64. Mimedefang and Spamassassin can read it
too, so searching content just works.

(You may have explained a special case, and if so, I'm sorry I forgot.)


Joseph Brennan

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/li
Dianne Skoll
7 years ago
Permalink
On Tue, 10 Oct 2017 16:11:07 -0400
Post by Joseph Brennan
Post by Michael Fox
I need to convert base64 to something readable.
To be honest I can't think of a reason to do this. Any client from
this century can read base64. Mimedefang and Spamassassin can read it
too, so searching content just works.
And QP is not necessarily any more readable than B64, depending
on the source language.

Is =D7=A2=D7=91=D7=A8=D7=99=D7=AA=E2=80=8E any more readable than
16LXkdeo15nXquKAjg== ?

:)

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.
Michael Fox
7 years ago
Permalink
Post by Dianne Skoll
On Tue, 10 Oct 2017 16:11:07 -0400
Post by Joseph Brennan
Post by Michael Fox
I need to convert base64 to something readable.
To be honest I can't think of a reason to do this.
Nevertheless, I need to for an old client.

So, back to my question:

if ($bh = $entity->bodyhandle) ;

$qp_body = encode_qp($bh->as_string)

}

It seems to work (I use md_syslog() to show the first 20 chars). Is that
right? Or is there a better way?

Then, I see an action_change_header(), but no action_change_body(). So,
presume I need to build a new entity by copying headers from the old entity,
replacing the Content-Transfer-Encoding header, and replacing the body.
Then use replace_entire_message().

Does that sound about right?

Michael


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.
Bill Cole
7 years ago
Permalink
Post by Michael Fox
Then, I see an action_change_header(), but no action_change_body().
So,
presume I need to build a new entity by copying headers from the old entity,
replacing the Content-Transfer-Encoding header, and replacing the body.
Then use replace_entire_message().
Does that sound about right?
Yes. For what you are determined to do, that would be the way. Note that
if the message body has been cryptographically signed in any way (DKIM,
OpenPGP, S/MIME, whatever) your replacement of the message will
invalidate the signature.
Post by Michael Fox
Also, I've got to believe someone else has needed to do this before.
But
I'm finding it tough to locate examples.
Faith is an amazing thing... :)

The irony is that this was more likely to be needed in the early 90's,
when there were many clients that could handle QP and not Base64, but
there wasn't any comprehensive toolkit for manipulating MIME messages.
By the time the tools got really good, it was no longer much needed or
generally safe to do the conversion to accommodate mail clients.

If the old client that you are accommodating is running on (or even just
displaying on) a reasonably modern OS, there's a strong chance that
you'll actually do better for readability by using UTF-8 as the
character set and "8bit" encoding instead of QP. Many (but not all)
older mail clients don't really know MIME but do happily pass through
all 8 bits of every byte to their output, whether it's a terminal driver
or a GUI API.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinf
Michael Fox
7 years ago
Permalink
-----Original Message-----
If the old client that you are accommodating is running on (or even just
displaying on) a reasonably modern OS, there's a strong chance that
you'll actually do better for readability by using UTF-8 as the
character set and "8bit" encoding instead of QP. Many (but not all)
older mail clients don't really know MIME but do happily pass through
all 8 bits of every byte to their output, whether it's a terminal driver
or a GUI API.
Thanks Bill.

Unfortunately, it's an amateur radio BBS written in the 80s that only understands plain text. It was pretty advanced in its time in that it uses SMTP internally. But no MIME, no UTF-8, etc. It's still extremely useful and widely used, especially in emergency comms, delivering mail over VHF radio where other connectivity isn't available.

Michael



_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefan
Paul Murphy
7 years ago
Permalink
Post by Michael Fox
Unfortunately, it's an amateur radio BBS written in the 80s that only understands plain text.
It's still extremely useful and widely used, especially in emergency comms
So worthwhile investing some cash and getting it updated so that it can cope with the modern world? And presumably the receiving equipment is 8-bit capable and not also something from the 80s?

As Dianne has already pointed out to you, converting everything to Quoted-Printable doesn't necessarily make it more readable. In the most extreme case, you'll encode spaces as well as extended characters, and especially when you're trying to pass addresses or telephone numbers then this could turn out to be problematic - try this one out:

We're=20off=20to=20the=20seaside,=20where=20the=20temperature=3D20=C2=B0C=20:-)=20=20El=20Ni=C3=B1o=20has=20some=20benefits!=20=20Call=20me=20on=20(+44)=20123=20456=207890

Paste it into http://www.webatic.com/run/convert/qp.php if you're struggling.

Now tell me that in an emergency situation you can extract the phone number quickly and without errors, and that you got the temperature from that message. If an address has Spanish, French, German, etc accented characters, they'll also be encoded rather than converted to the nearest unaccented character, so in the example above " El NiƱo" becomes "El=20Ni=C3=B1o" rather than "El Nino".

Eventually this will come back to bite you. Time to bring it at least into the 90s, if not into the 21st century...

Paul.


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/l
Michael Fox
7 years ago
Permalink
-----Original Message-----
Post by Michael Fox
Unfortunately, it's an amateur radio BBS written in the 80s that only
understands plain text.
Post by Michael Fox
It's still extremely useful and widely used, especially in emergency
comms
So worthwhile investing some cash and getting it updated so that it can
cope with the modern world? [clipped]
While I appreciate everyone's attempt to warn me that "there be dragons", the facts are that the client is what it is, it serves hundreds of people (in my location alone), and the problem needs solving.

And, while I may be a newbie to MIME and MIMEDefang (and Perl, for that matter), I'm definitely NOT a newbie in most other respects. I've been doing network engineering since the days of the NSFnet and I'm pretty certain I understand this particular application need better than anyone else here.

So, going forward, I'm hopeful I can get advice and help on MIME and MIMEDefang here. And warnings about dangers of this or that detail or approach are, of course, very helpful. But suggesting that the client be changed doesn't really answer the questions that I'm asking.

Paul, Dianne: regarding readability of quoted-printable: again, I'm a newbie to MIMEDefang, not the world. So I understand and appreciate the warning but I still have to solve the problem. The incoming messages were originated as plain ASCII text. So, the quoted-printable results I'm seeing aren't hard to read at all. But if worse comes to worse, I may end up squashing it to plain ASCII text. We'll see.

Michael



_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo
Dianne Skoll
7 years ago
Permalink
On Fri, 13 Oct 2017 06:53:33 -0700
Post by Michael Fox
While I appreciate everyone's attempt to warn me that "there be
dragons", the facts are that the client is what it is, it serves
hundreds of people (in my location alone), and the problem needs
solving.
OK. You will most likely have to do all your changes in
filter_end. You'll need to traverse the MIME tree, looking for
text/* parts. If any are Base-64 encoded, you'll need to load them in
and re-encode them using Quoted-Printable. As you do this, you need
to build up a *new* replacement MIME::Entity. Finally, call
replace_entire_message with your new entity as the argument.

I won't give the actual Perl code, but here's a basic sketch, completely
untested. I really have no idea whether or not this will work, but I
think the basic approach is correct. :)

sub re_encode
{
my ($in_entity, $out_entity, $did_something) = @_;

# If it's multipart, recurse
if ($in_entity->is_multipart) {
my $new_entity = MIME::Entity->build(Type => $in_entity->type);
$out_entity->add_part($new_entity);
foreach my $p (@{$in_entity->parts}) {
if (re_encode($p, $new_entity, $did_something)) {
$did_something = 1;
}
}
return $did_something;
}

# non-recursive case: Single part.
# If $in_entity is base-64 encoded, make a new $new_entity
# that is qp-encoded and call $out_entity->attach($new_entity)
# and return 1 to indicate that a change was made.
# Otherwise, call $out_entity->attach($in_entity)
# and return 0.
}

sub filter_end
{
my ($entity) = @_;
my $replacement_entity = MIME::Entity->build(Type => 'multipart/mixed');
my $changes_made = re_encode($entity, $replacement_entity, 0);
if ($changes_made) {
# Recursively remove single-part "multiparts" starting
# from top-level.
while ($replacement_entity->is_multipart && scalar(@{$replacement_entity->parts}) == 1) {
$replacement_entity = $replacement_entity->parts(0);
}
replace_entire_message($replacement_entity);
}
}

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com
Dianne Skoll
7 years ago
Permalink
On Fri, 13 Oct 2017 10:20:04 -0400
Post by Dianne Skoll
# non-recursive case: Single part.
# If $in_entity is base-64 encoded, make a new $new_entity
# that is qp-encoded and call $out_entity->attach($new_entity)
# and return 1 to indicate that a change was made.
# Otherwise, call $out_entity->attach($in_entity)
# and return 0.
Oh, sorry. ->attach should be ->add_part.

Regards,

Dianne.
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://li
Michael Fox
7 years ago
Permalink
-----Original Message-----
I won't give the actual Perl code, but here's a basic sketch, completely
untested. I really have no idea whether or not this will work, but I
think the basic approach is correct. :)
Thanks Dianne. That definitely points me in the right direction. Much appreciated.

Michael




_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.r
Joseph Brennan
7 years ago
Permalink
-----Original Message-----
Post by Michael Fox
Unfortunately, it's an amateur radio BBS written in the 80s that only
understands plain text.
Post by Michael Fox
It's still extremely useful and widely used, especially in emergency
comms
Interesting case. Most likely none of us have needed to do this type
of conversion. But you have a tool here that is flexible enough to do
it.

I suppose you want to take the text/plain MIME part and remove any
other parts. Others would be just an HTML alternative or something
that is not text.

If you happen to know that the original was always plain ascii with
normal line lengths (80 or less) then I'd tempted to say just go from
base64 to ascii.

It does make me wonder why it is base64 at all then, but I've been
seeing mail from Exchange that is always base64, so maybe that is what
you are dealing with.

Joe Brennan

_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/lis
Michael Fox
7 years ago
Permalink
-----Original Message-----
But you have a tool here that is flexible enough to do it.
Yes, indeed.
I suppose you want to take the text/plain MIME part and remove any
other parts. Others would be just an HTML alternative or something
that is not text.
Yup. And I'm doing that already, and adding a footer to tell the recipient what was removed. The case I'm trying to solve is when the text/plain or text/html part has Content-Transfer-Encoding of Base64.
If you happen to know that the original was always plain ascii with
normal line lengths (80 or less) then I'd tempted to say just go from
base64 to ascii.
Yup. And "groff -T ascii" has been suggested, just in case there are emoji's or other things to squash. But the quoted-printable suggestion would be less destructive so I'm trying that out first. Either way, 7bit or quoted-printable, I apparently need to more carefully parse the MIME structure and ultimately rebuild the message. Hence, my questions.
It does make me wonder why it is base64 at all then, but I've been
seeing mail from Exchange that is always base64, so maybe that is what
you are dealing with.
In most cases, it's the carrier or email service provider or sender's client app and it's often not something the sending user can control. So rejecting the message results in a complete loss of communications.

Example: send a simple text message like "Hello" from Sprint to an email address and they wrap it in HTML and encode it with Base64.

Sending from Verizon, a simple text message arrives as 7bit plain text. A short text, like "Hello", with one emoji arrives as quoted-printable. A short text message with multiple emoji's arrives as base64. A slightly longer text message, like "Me and the kids are safe at Mom's house. Call when you can.", with multiple emoji's arrives as an attachment. <sigh>

Different things also happen from smart phone email apps. So, "there be dragons". And ultimately, this may be beyond my ability to solve. But I'm not ready to give up yet. If I can convert at least some messages that otherwise would not be readable, it's a win.

Michael
a.k.a Don Quixote de La Mancha



_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpengu
John Nemeth
7 years ago
Permalink
On Oct 11, 11:54pm, "Michael Fox" wrote:
}
} > -----Original Message-----
} > If the old client that you are accommodating is running on (or even just
} > displaying on) a reasonably modern OS, there's a strong chance that
} > you'll actually do better for readability by using UTF-8 as the
} > character set and "8bit" encoding instead of QP. Many (but not all)
} > older mail clients don't really know MIME but do happily pass through
} > all 8 bits of every byte to their output, whether it's a terminal driver
} > or a GUI API.
}
} Thanks Bill.
}
} Unfortunately, it's an amateur radio BBS written in the 80s that

Which app is it?

} only understands plain text. It was pretty advanced in its time

Something this old may not understand quoted-printable.

} in that it uses SMTP internally. But no MIME, no UTF-8, etc.
} It's still extremely useful and widely used, especially in
} emergency comms, delivering mail over VHF radio where other
} connectivity isn't available.

I have a similar requirement to modify a message where I'm
using action_external_filter(). My filter() function looks like
this:

sub filter ($$$$) {
my($entity, $fname, $ext, $type) = @_;

return if message_rejected(); # Avoid unnecessary work

if ($fname eq "Cisco_fax.tif") {
# convert incoming fax to pdf
md_syslog('info', 'converting tiff to pdf');
action_external_filter($entity, "/usr/pkg/bin/tiff2pdf -o ./FILTEROUTPUT ./FILTERINPUT");
my $head = $entity->head;
$head->replace('content-type', 'application/pdf; name="Cisco_fax.pdf"');
}

return action_accept();
}

There should probably be some more error checking such as making
sure that action_external_filter() succeeds. Perhaps if you look
at the definition of action_external_filter() in mimedefang.pl it
will give you some ideas. It's a fairly simple function (most of
it is error checking).

VA7JMN

Note that I've only had licence for a couple of years and am still
getting my feet wet, but I'm definitely interested in data and
emergency comms. I've recently started playing with RMS Express
(over telnet). The next thing I want to do is get a TNC.

}-- End of excerpt from "Michael Fox"
_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinf
Michael Fox
7 years ago
Permalink
-----Original Message-----
} Unfortunately, it's an amateur radio BBS written in the 80s that
Which app is it?
JNOS
http://www.langelaar.net/projects/jnos2/
} only understands plain text. It was pretty advanced in its time
Something this old may not understand quoted-printable.
It doesn't really understand MIME. So it spits out quoted-printable as the text that it is.
I have a similar requirement to modify a message where I'm
using action_external_filter(). My filter() function looks like
Excellent. Examples of similar actions (even if they're not directly applicable) are a real help to a MIMEDefang and Perl newbie.
There should probably be some more error checking such as making
sure that action_external_filter() succeeds. Perhaps if you look
at the definition of action_external_filter() in mimedefang.pl it
will give you some ideas. It's a fairly simple function (most of
it is error checking).
VA7JMN
Thanks. I'll do that.

Michael
N6MEF


_______________________________________________
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.

Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list ***@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman

Continue reading on narkive:
Loading...