Ask not what osTicket community can do for you - ask what you can do for osTicket community

Go Back   osTicket Forums > Project Tools > osTicket Bug Tracker > Bug

Issue Type Bug   Project osTicket Bug Tracker
International characters in eMails
Category Unknown
Affected Version 1.6 Stable
Priority 5 - Medium
Status Unconfirmed
Fixed Version (none)
Submitted 09-22-2010
Assigned Users (none) Tags (none)

issueid=227 09-22-2010 09:58 AM
Junior Member
International characters in eMails

Hi,

Currently implementing osTicket, i'm facing some problems with french (and other languages) characters, like é , è , à , ...

It seems to work correctly with the web interface, but not with the eMails... here's a small summary :

Sending EMails
The header (subject) of the email is announced as ISO but coded using UTF-8. Example : "Subject: =?ISO-8859-1?Q?[#121115]=20Message=20ajout=C3=A9?="

Fix : OST allways use UTF-8, but the pear Mail_mime default charset is ISO-8859-1 ... OST define the charset for body but not for the header...

to fix, add the "head_charset" settings in class.email.php like that :

Quote:
$options=array('head_encoding' => 'quoted-printable',
'text_encoding' => 'quoted-printable',
'html_encoding' => 'base64',
'html_charset' => 'utf-8',
'text_charset' => 'utf-8',
'head_charset' => 'utf-8');
(This settings appear twice in the file).


Receiving EMails
Using POP3 collector
For a message coded using ISO-8859-1 charset, the subject is truncated at the first accentuated character. The body seems to be correct.

For a message coded using UTF-8 charset, apparently no problem.

Using PIPE
For a message coded using ISO-8859-1 charset, apparently no problem

For a message coded using UTF-8, the message seems to be interpreted first using ISO-8859-1 charset, meaning that every accentuated char is displayed as two differents chars, example, "é" is replaced by "é" (the UTF-8 representation of "é" is =C3=A9 )


Of course, we cannot know in advance which charset will be used when someone send us a mail, and we cannot force anyone to use the same charset :)



I've fixed the problem for the mails OST send, but i need some help to fix the incoming mails, preferably using the PIPE method... did someone know how to first decode the mail using the right charset (defined by the content-type header) before recoding it using UUTF-8 ?
Reply

09-24-2010 06:44 PM
Junior Member
 
I confirm this e-mail thing. Piping e-mail, which text is not in ASCII set leads to broken text received by the system. Text, which is entered through web form and then sent by e-mail is OK.
The problem is in pear/Mail/mimeDecode.php which pay absolutely no attention to the character set notes in headers. I've found a patch in their bug tracker. When I applied, it become to convert mails, but in api/pipe.php there is calls to utf8_encode, which brokes that other way. When I removed calls to utf8_encode, text from emails is OK now.

Here is patch (hope it will be ok embedded here):
Code:
diff -ru a/api/pipe.php b/api/pipe.php
--- a/api/pipe.php      2010-02-10 03:43:54.000000000 +0300
+++ b/api/pipe.php      2010-09-25 02:14:26.235597524 +0400
@@ -1,4 +1,3 @@
-#!/usr/bin/php -q
 <?php
 /*********************************************************************
     pipe.php
@@ -77,16 +76,16 @@
 $name=trim($from->personal,'"');
 if($from->comment && $from->comment[0])
     $name.=' ('.$from->comment[0].')';
-$subj=utf8_encode($parser->getSubject());
+$subj=$parser->getSubject();
 if(!($body=Format::stripEmptyLines($parser->getBody())) && $subj)
     $body=$subj;
 
 $var['mid']=$parser->getMessageId();
 $var['email']=$from->mailbox.'@'.$from->host;
-$var['name']=$name?utf8_encode($name):$var['email'];
+$var['name']=$name?$name:$var['email'];
 $var['emailId']=$emailId?$emailId:$cfg->getDefaultEmailId();
 $var['subject']=$subj?$subj:'[No Subject]';
-$var['message']=utf8_encode(Format::stripEmptyLines($body));
+$var['message']=Format::stripEmptyLines($body);
 $var['header']=$parser->getHeader();
 $var['pri']=$cfg->useEmailPriority()?$parser->getPriority():0;
 
diff -ru a/include/pear/Mail/mimeDecode.php b/include/pear/Mail/mimeDecode.php
--- a/include/pear/Mail/mimeDecode.php  2010-02-10 09:23:04.000000000 +0300
+++ b/include/pear/Mail/mimeDecode.php  2010-09-25 02:05:51.248094321 +0400
@@ -298,12 +300,12 @@
             switch (strtolower($content_type['value'])) {
                 case 'text/plain':
                     $encoding = isset($content_transfer_encoding) ? $content_transfer_encoding['value'] : '7bit';
-                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $encoding) : $body) : null;
+                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $encoding, $return->ctype_parameters) : $body) : null;
                     break;
 
                 case 'text/html':
                     $encoding = isset($content_transfer_encoding) ? $content_transfer_encoding['value'] : '7bit';
-                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $encoding) : $body) : null;
+                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $encoding, $return->ctype_parameters) : $body) : null;
                     break;
 
                 case 'multipart/parallel':
@@ -342,7 +344,7 @@
                 default:
                     if(!isset($content_transfer_encoding['value']))
                         $content_transfer_encoding['value'] = '7bit';
-                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $content_transfer_encoding['value']) : $body) : null;
+                    $this->_include_bodies ? $return->body = ($this->_decode_bodies ? $this->_decodeBody($body, $content_transfer_encoding['value'], $return->ctype_parameters) : $body) : null;
                     break;
             }
 
@@ -543,34 +545,41 @@
      */
     function _decodeHeader($input)
     {
-        // Remove white space between encoded-words
-        $input = preg_replace('/(=\?[^?]+\?(q|b)\?[^?]*\?=)(\s)+=\?/i', '\1=?', $input);
 
-        // For each encoded-word...
-        while (preg_match('/(=\?([^?]+)\?(q|b)\?([^?]*)\?=)/i', $input, $matches)) {
+        if (extension_loaded('iconv')) {
+            return iconv_mime_decode($input,2);
+        } else {
+            // Remove white space between encoded-words
+            $input = preg_replace('/(=\?[^?]+\?(q|b)\?[^?]*\?=)(\s)+=\?/i', '\1=?', $input);
 
-            $encoded  = $matches[1];
-            $charset  = $matches[2];
-            $encoding = $matches[3];
-            $text     = $matches[4];
-
-            switch (strtolower($encoding)) {
-                case 'b':
-                    $text = base64_decode($text);
-                    break;
+            // For each encoded-word...
+            while (preg_match('/=\?([^?]+)\?(q|b)\?([^?]*)\?=/i', $input, $matches)) {
+
+                $encoded  = $matches[0];
+                $charset  = $matches[1];
+                $encoding = $matches[2];
+                $text     = $matches[3];
 
-                case 'q':
-                    $text = str_replace('_', ' ', $text);
-                    preg_match_all('/=([a-f0-9]{2})/i', $text, $matches);
-                    foreach($matches[1] as $value)
+
+
+                switch (strtolower($encoding)) {
+                    case 'b':
+                        $text = iconv_mime_decode($text,2);
+                        break;
+
+                    case 'q':
+                        $text = str_replace('_', ' ', $text);
+                        preg_match_all('/=([a-f0-9]{2})/i', $text, $matches);
+                        foreach($matches[1] as $value)
                         $text = str_replace('='.$value, chr(hexdec($value)), $text);
-                    break;
+                        break;
+                }
+
+                $input = str_replace($encoded, $text, $input);
             }
 
-            $input = str_replace($encoded, $text, $input);
+            return $input;
         }
-
-        return $input;
     }
 
     /**
@@ -579,26 +588,40 @@
      *
      * @param  string Input body to decode
      * @param  string Encoding type to use.
+     * @param  string Character set
      * @return string Decoded body
      * @access private
      */
-    function _decodeBody($input, $encoding = '7bit')
+    function _decodeBody($input, $encoding = '7bit', $ctype_params = null)
     {
+
+        $bytes = $input;
         switch (strtolower($encoding)) {
-            case '7bit':
-                return $input;
+            case '7bit':                
                 break;
 
             case 'quoted-printable':
-                return $this->_quotedPrintableDecode($input);
+                $bytes = $this->_quotedPrintableDecode($input);
                 break;
 
             case 'base64':
-                return base64_decode($input);
+                $bytes = base64_decode($input);
                 break;
 
             default:
-                return $input;
+                break;
+        }
+
+        
+        if (extension_loaded('iconv') &&
+            is_array($ctype_params) &&
+            array_key_exists('charset', $ctype_params)) {
+            
+            $charset = $ctype_params['charset'];
+                
+            return iconv($charset, iconv_get_encoding('internal_encoding'), $bytes);
+        } else {
+            return $bytes;
         }
     }
Reply
09-25-2010 03:33 AM
Junior Member
 
Thanks for your help.

Could you tell me what utf8_encode in pipe.php broke ?

For me, if i apply only your patch on mimeDecode.php it seems to work fine, but if i remove the utf8_encode then every accentuated char is stripped...

I will leave the utf8_encode and see further, but for now it seems to work correctly for me.
Reply
10-04-2010 03:37 AM
Junior Member
 
Hmm. If it works for You, may be its OK. But on my server utf8_encode leads to utf8-text converted as iso8859-1 to utf8. May be it depends on some php options.
Reply
10-04-2010 03:54 AM
Junior Member
 
It's working, but i'm still facing a problem with some emails :

With EMail containing the ' character encoded like =92 (URL encoding ?)

The message is truncated at the first occurrence of =92

I've not yet checked in detail, i think that the SQL security filter replacing every ' character could be executed before the decoding of the mail, and thus after the decoding the ' could be misinterpreted by sql ... if it's the problem, it could be a security problem, allowing SQL injections...
Reply
10-04-2010 03:55 PM
Junior Member
 
It seems that the ' problem occur after the patch of mimeDecode... thus i reverted back, i've installed the latest official Pear mimeDecode (1.5.4)... but of course, the problem of UTF-8 encoded messages occur again... utf8_encode does not work correctly with allready UTF-8 encoded strings...

I've then modified pipe.php to replace the utf8_encode function with mb_convert_encoding and mb_detect_encoding:

Quote:
// $var['message']=utf8_encode(Format::stripEmptyLines($body));
$string=Format::stripEmptyLines($body);
$var['message']=mb_convert_encoding($string, "UTF-8", mb_detect_encoding($string, "UTF-8, ISO-8859-1, ISO-8859-15", true));
I've done the same with other occurrences of utf8_encode in pipe.php...

It's not perfect yet, but it's much better... accentuated characters are correctly handled, the only detail is that the ' character is now stripped... i prefer that than a whole mail truncated...
Reply
02-03-2011 12:48 PM
Junior Member
 
Thanks antarex - this fix worked for me as well.
I also don't mind the missing quotes if the whole message comes through either. Cheers.
Reply
03-17-2011 10:57 AM
Banned
 
thanks for ur help it worked
technology
Reply
01-12-2012 04:21 PM
Junior Member
 
Hi
Please help me with russian charset sent to script via pipe

i sent
HTML Code:
йцукенгшщз
but receive to the ticket this
HTML Code:
ЪУеЫХЮЧлнк
has you see this is not same

before i have problem with Russian totally, and i found solution here
http://www.osticket.com/forums/showp...7&postcount=11

but now i receive different text to ticket
Reply
02-29-2012 04:52 PM
Junior Member
 
Green,
if i try and use an é in a filename to be added as an attachment, that also fails.
i guess a similar bug but with the symptom in a different place?
Reply

Issue Tools
Subscribe to this issue

All times are GMT -4. The time now is 06:38 PM.