Excessive memory usage when calling new MimeMessage(Session, InputStream)

[David Erickson]

I sent this off to the dev team but I'm wondering if anyone else has noticed
this behavior, and if so has anyone done anything about it?

I have noticed a major problem with using the MimeMessage(Session,
InputStream) constructor. When you use this over 2x the size of the actual
message worth of memory will be allocated which can be a major problem if
you have large attachment sizes. The reason this is happening is down in
the ASCIIUtility.getBytes(InputStream) method. What happens is it reads the
inputstream fully into a ByteArrayOutputStream, which is fine. The problem
then comes is that when you call .getBytes() on that output stream it
creates a copy of the byte[], instead of just returning the internal byte[],
thus it is allocating 2x the memory. To fix this you have two options,
create your own OutputStream and give access to its internal byte[], or you
could subclass ByteArrayOutputStream and add an accessor to the buf byte
array since its protected, and return that.

Additionally I just noticed that javamail 1.3.2 source has been released, is
it legal to fix the above problem in the source, rebuild it, and use it in
your app? It didn't look like it according to the research liscense... but
if it's a problem with the code...

Thanks,
David Erickson

===========================================================================
To unsubscribe, send email to ***@java.sun.com and include in the body
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
***@java.sun.com and include in the body of the message "help".

[Ken Cline]

Post by David Erickson
I sent this off to the dev team but I'm wondering if anyone
else has noticed this behavior, and if so has anyone done
anything about it?

In my last job (where I was using JavaMail) we definitely
had problems with large messages and part of the issue was
the fact that the byte arrary IO streams caused you to have
multiple copies (at least 2) of the same bytes in memory.

Fortunately for us, we were popping the messages and hence
could stream the attachments to file and then replace
content-type and filename to indicate that the attachment
had been dumped and where it could be found. On the other
hand, it did mean that processing of the attachment (OCR, in
this case) had to run on the same server or we had to use
a file transfer protocol to move the file. Not a big deal
until you got into a production environment with lots of
redundancy and cross-wired of servers and etc.

Post by David Erickson
I have noticed a major problem with using the
MimeMessage(Session, InputStream) constructor. When you
use this over 2x the size of the actual message worth of
memory will be allocated which can be a major problem if
you have large attachment sizes. The reason this is
happening is down in the ASCIIUtility.getBytes(InputStream)
method. What happens is it reads the inputstream fully
into a ByteArrayOutputStream, which is fine. The problem
then comes is that when you call .getBytes() on that output
stream it creates a copy of the byte[], instead of just
returning the internal byte[], thus it is allocating 2x
the memory. To fix this you have two options, create your
own OutputStream and give access to its internal byte[],
or you could subclass ByteArrayOutputStream and add an
accessor to the buf byte array since its protected, and
return that.
Additionally I just noticed that javamail 1.3.2 source has
been released, is it legal to fix the above problem in the
source, rebuild it, and use it in your app? It didn't look
like it according to the research liscense... but if it's
a problem with the code...

Couldn't you create your own InputStream and then override
the protected MimeMessage.parse method so that it detects
when argument was an instance of your InputStream class?
You could then avoid calling ASCIIUtility.getBytes and
instead pull the byte[] directly out of your InputStream.

Note: Using your own InputStream with pre-read/loaded bytes
could have the added advantage of improving performance
_IF_ your application happens to be falling into the
else-block of ASCIIUtility.getBytes which is copying
the input stream in 1K blocks. A larger block size
means less memory allocations and array copies in the
ByteArrayOutputStream.

Even if that works, you should consider upgrading your
hardware (and of course increasing the heap size) so that
you have more than a 2x safety margin. There are probably
lots other places downstream that might create duplicate
allocations of your data and hence cause out-of-memory
errors.

Ken.



_________________________________________________________
Ken Cline W: [phone]
***@yahoo.com

===========================================================================
To unsubscribe, send email to ***@java.sun.com and include in the body
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
***@java.sun.com and include in the body of the message "help".

[David Erickson]

-----Original Message-----
Sent: Friday, September 02, 2005 11:35 AM
Subject: Re: Excessive memory usage when calling new MimeMessage(Session,
InputStream)

Post by David Erickson
I sent this off to the dev team but I'm wondering if anyone
else has noticed this behavior, and if so has anyone done
anything about it?

In my last job (where I was using JavaMail) we definitely
had problems with large messages and part of the issue was
the fact that the byte arrary IO streams caused you to have
multiple copies (at least 2) of the same bytes in memory.
Fortunately for us, we were popping the messages and hence
could stream the attachments to file and then replace
content-type and filename to indicate that the attachment
had been dumped and where it could be found. On the other
hand, it did mean that processing of the attachment (OCR, in
this case) had to run on the same server or we had to use
a file transfer protocol to move the file. Not a big deal
until you got into a production environment with lots of
redundancy and cross-wired of servers and etc.

Post by David Erickson
I have noticed a major problem with using the
MimeMessage(Session, InputStream) constructor. When you
use this over 2x the size of the actual message worth of
memory will be allocated which can be a major problem if
you have large attachment sizes. The reason this is
happening is down in the ASCIIUtility.getBytes(InputStream)
method. What happens is it reads the inputstream fully
into a ByteArrayOutputStream, which is fine. The problem
then comes is that when you call .getBytes() on that output
stream it creates a copy of the byte[], instead of just
returning the internal byte[], thus it is allocating 2x
the memory. To fix this you have two options, create your
own OutputStream and give access to its internal byte[],
or you could subclass ByteArrayOutputStream and add an
accessor to the buf byte array since its protected, and
return that.
Additionally I just noticed that javamail 1.3.2 source has
been released, is it legal to fix the above problem in the
source, rebuild it, and use it in your app? It didn't look
like it according to the research liscense... but if it's
a problem with the code...

Couldn't you create your own InputStream and then override
the protected MimeMessage.parse method so that it detects
when argument was an instance of your InputStream class?
You could then avoid calling ASCIIUtility.getBytes and
instead pull the byte[] directly out of your InputStream.
Note: Using your own InputStream with pre-read/loaded bytes
could have the added advantage of improving performance
_IF_ your application happens to be falling into the
else-block of ASCIIUtility.getBytes which is copying
the input stream in 1K blocks. A larger block size
means less memory allocations and array copies in the
ByteArrayOutputStream.
Even if that works, you should consider upgrading your
hardware (and of course increasing the heap size) so that
you have more than a 2x safety margin. There are probably
lots other places downstream that might create duplicate
allocations of your data and hence cause out-of-memory
errors.

Ya I definitely could override the parse method, my only concern is down the
road if they change the way things work in that method, my code could easily
break. And unfortunaetly my inputstream is a FileInputStream so I am
falling into the else block.

While I was profiling my app this is actually the only place that duplicates
the memory 2x... which is great. My hardware and memory is definitely up to
snuff that runs it, my only concern is this software will have the potential
to be receiving hundreds if not thousands of emails over a very short period
of time, so I need to ensure memory usage of each email is at a very minimum
amount.

-David

Ken.
_________________________________________________________
Ken Cline W: [phone]
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.10.18/89 - Release Date: 9/2/2005

===========================================================================
To unsubscribe, send email to ***@java.sun.com and include in the body
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
***@java.sun.com and include in the body of the message "help".

[Bill Shannon]

Post by David Erickson
I sent this off to the dev team but I'm wondering if anyone else has noticed
this behavior, and if so has anyone done anything about it?

Sorry, my spam filter trapped your message so I didn't see it at first.

Post by David Erickson
I have noticed a major problem with using the MimeMessage(Session,
InputStream) constructor. When you use this over 2x the size of the actual
message worth of memory will be allocated which can be a major problem if
you have large attachment sizes. The reason this is happening is down in
the ASCIIUtility.getBytes(InputStream) method. What happens is it reads the
inputstream fully into a ByteArrayOutputStream, which is fine. The problem
then comes is that when you call .getBytes() on that output stream it
creates a copy of the byte[], instead of just returning the internal byte[],
thus it is allocating 2x the memory. To fix this you have two options,
create your own OutputStream and give access to its internal byte[], or you
could subclass ByteArrayOutputStream and add an accessor to the buf byte
array since its protected, and return that.

See the javax.mail.internet.SharedInputStream interface. If your
InputStream implements this interface, it will be processed more
efficiently.

There are two implementations of that interface in the com.sun.mail.util
package - SharedByteArrayInputStream and SharedFileInputStream. Those
classes are being moved to the javax.mail.util package in JavaMail 1.4.

Try that and I think you'll see a significant improvement.

===========================================================================
To unsubscribe, send email to ***@java.sun.com and include in the body
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
***@java.sun.com and include in the body of the message "help".

[David Erickson]

This worked fantastic for reading in my message from a file (with a null
session). I did however run into a problem, before I send my message out I
want to add a footer onto the end, so I have a utility that parses through
the parts until it gets the text or html part and adds onto the end of it.
I am now getting an exception however using this sharedinputStream, here is
my code:

if (p.isMimeType("multipart/*")) {
MimeMultipart multiPart = (MimeMultipart) p.getContent();
boolean result = false;
for (int i = 0; i < multiPart.getCount(); ++i) {
result = addFooter(multiPart.getBodyPart(i), textFooter, htmlFooter)
|| result;
}

The code is now bombing on the multipart.getCount() when called on the root
messages content. Here is the exception:

javax.mail.MessagingException: IO Error;
nested exception is:
java.io.IOException: The handle is invalid
at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:424)
at
javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:159)
at org.tecas.util.EmailUtil.addFooter(EmailUtil.java:52)

Any ideas? Also is this going to be a problem when I call
message.saveChanges()?

Thanks,
David

-----Original Message-----
From: A mailing list for discussion of the JavaMail(tm) API
Sent: Friday, September 02, 2005 12:35 PM
Subject: Re: Excessive memory usage when calling new MimeMessage(Session,
InputStream)

Post by David Erickson
I sent this off to the dev team but I'm wondering if anyone else has

noticed

Post by David Erickson
this behavior, and if so has anyone done anything about it?

Sorry, my spam filter trapped your message so I didn't see it at first.

Post by David Erickson
I have noticed a major problem with using the MimeMessage(Session,
InputStream) constructor. When you use this over 2x the size of the

actual

Post by David Erickson
message worth of memory will be allocated which can be a major problem

if

Post by David Erickson
you have large attachment sizes. The reason this is happening is down

in

Post by David Erickson
the ASCIIUtility.getBytes(InputStream) method. What happens is it reads

the

Post by David Erickson
inputstream fully into a ByteArrayOutputStream, which is fine. The

problem

Post by David Erickson
then comes is that when you call .getBytes() on that output stream it
creates a copy of the byte[], instead of just returning the internal

byte[],

Post by David Erickson
thus it is allocating 2x the memory. To fix this you have two options,
create your own OutputStream and give access to its internal byte[], or

you

Post by David Erickson
could subclass ByteArrayOutputStream and add an accessor to the buf byte
array since its protected, and return that.

See the javax.mail.internet.SharedInputStream interface. If your
InputStream implements this interface, it will be processed more
efficiently.
There are two implementations of that interface in the com.sun.mail.util
package - SharedByteArrayInputStream and SharedFileInputStream. Those
classes are being moved to the javax.mail.util package in JavaMail 1.4.
Try that and I think you'll see a significant improvement.
==========================================================================
=
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
--
No virus found in this incoming message.
Checked by AVG Anti-Virus.
Version: 7.0.344 / Virus Database: 267.10.18/89 - Release Date: 9/2/2005

===========================================================================
To unsubscribe, send email to ***@java.sun.com and include in the body
of the message "signoff JAVAMAIL-INTEREST". For general help, send email to
***@java.sun.com and include in the body of the message "help".