Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

In this Discussion

osTicket v1.10 (stable) and Maintenance Release v1.9.15 are now available! Go get it now

Email Attachments - non-English characters

Hi guys,
First, sorry for such a long post, this is an issue that has existed in osTicket for 3 years from what I can see and still no real solution so it justifies plenty of information to help.

There are a number of posts regarding the issues that exist with email attachments which contain characters like Ç, ã, é, etc...

The reason for this separate post is to get community feedback on an insight regarding the issue.

Over the past 3 1/2 months I have tried around 20 different potential solutions but nothing has worked, then today I had an thought, maybe these files aren't actually being looked at by osTicket?

the preg_replace() function is working perfectly (I have tested in a separate folder and parsed a large number of strings with every special character and name I could think of) but files with the (in my case) Brazilian Portuguese characters just are not being parsed. I have used a number of variations of preg_replace() with the most successful & easiest being in include/class.format.php

function file_name($filename) {
$filename = preg_replace(\"/[^a-zA-Z0-9.]/\", \"_\", $filename);
return $filename;
}


This doesn't replace a "ç" with a "c" for example, but honestly for the purposes of this it doesn't actually matter to us, we just need the files.

To test the strings I have 4 PDF files, osTicket is set to allow PDF files and does so for us daily, the names are
  • chrome.pdf (control file, if any file parses this one will
  • special characters.pdf (basic test file, this has a single replacement of a space to ensure the preg_replace() function is working)
  • comçeÇ.pdf (character replacement file, to test if the basic ç and Ç are rpelaced)
  • spéĉíãl chârãçtôrs.pdf (standard file, being used to test multiple character and space replacements
In every test only chrome.pdf and special characters.pdf are parsed with the others that include the Brazilian Portuguese characters being canned.

The only thing that is now making sense after 2 days complete straight looking at and testing modifications to the core code is, this is maybe osTicket isn't actually looking at the attachments.

Does anyone else have any insights to this possibility?

I appreciate any help you guys can provide!

Cheers,
Ryan

Comments

  • File not parsing

    Hi guys,
    I'm need of some help here.

    I have created some logs to record output to try and locate the issue, this includes an sql log (when the information is being inserted to the database) and a log of all file names being parsed through each step of the system.

    What I have found is there is no logged record of these files coming into the system at all. What I need to know is what code is actually picking up the file at the source email.

    For example, there are functions like file_name() but this is only amending the file name once it has already been picked up from the email. Using grep I have found functions that are looking at the files earlier than the file_name() function but none that appear to be actually detecting the files at source.

    The reason I need to know this is so I can create a series of logs and isolate the code that is ignoring these files, this will allow me to put in place a fix for writing the files to the server and inserting the records to the database.

    Thanks in advance for your help!
  • SOLVED: naughty mime

    The first thing was to discover why the attachments were not being parsed through the system. To work this out I started logging each function being called that deals with attachments. Initially when I would send 2 attachements, 1 with English characters only (I will call this att1) and 1 with Portuguese characters also (to be referred to as att2) I would only see att1 parsing in the log, this told me that att2 was not being detected by osTicket. This was not actually correct as I found out.

    The following code in include/class.mailfetch.php simply hid the problem:

    function saveAttachments($ticket,$mid,$part,$index=0) {
    global $cfg;
    if($part && $part->ifdparameters && ($filename=$part->dparameters[0]->value)){ //attachment
    $index=$index?$index:1;


    When I started logging every action in the function I found att1 and att2, when I logged before the $index variable was defined I found att2 but after it disappeared. Why? Because att2's file name was not decoded from mime as att1's file name was with the rest of the headers.

    To fix this I have put in place the following fallback

    function saveAttachments($ticket,$mid,$part,$index=0) {
    global $cfg;
    if($part && $part->ifdparameters && ($filename=$part->dparameters[0]->value)){ //attachment
    // ISO-8859-1 FALLBACK BY Ryan from Ahgora Sistemas (www.ahgora.com.br) START
    // Tell me sir, what am I looking for?
    $pm_pattern = '/ISO-8859-1/';
    // Now where do I find it?
    $pm_go = preg_match($pm_pattern,$filename);
    if($pm_go) {
    // Ok mister, I found it but what should I do with it?
    $decoder = imap_mime_header_decode($filename);
    $filename = $decoder[0]->text;
    // She is delivered safe and sound sir
    }
    // ISO-8859-1 FALLBACK BY Ryan from Ahgora Sistemas (www.ahgora.com.br) END
    $index=$index?$index:1;


    Where there will be another reason for this situation I will not keep trying to work it out for the time being, the code above makes anyone dealing with the ISO-8859-1 charset safe to get all attachments you should.

    To the developers of osTicket, for your next release you're welcome to include the fix.

    I would appreciate any feedback about the above.

    Cheers,
    Ryan - Ahgora Sistemas
    www.ahgora.com.br
  • Character replacement for attachments

    Hi guys,
    I just received a support ticket from another osTicket user with a related but not the same problem.
    This problem is with the change of the file names from the Portuguese character set to the English counter-part. What happens is the file name is truncated, most likely the Portuguese characters are replaced with an underscore.

    The function that replaces the letters like "ç" with an underscore is located (on the default 1.6ST distribution) in /include/class.format.php on line 34. The standard PHP function is called preg_replace, documentation can be found here.

    To correct this issue to replace for example a "ç" with a "c" find the following code in include/class.format.php:

    $search = array('/ß/','/ä/','/Ä/','/ö/','/Ö/','/ü/','/Ü/','([^[:alnum:]._])');
    $replace = array('ss','ae','Ae','oe','Oe','ue','Ue','_');
    return preg_replace($search,$replace,$filename);


    and replace it with

     $search = array('/Á/', '/Â/', '/Ã/', '/À/', '/Ç/', '/É/', '/Ê/', '/Í/', '/Ó/', '/Ô/', '/Õ/', '/Ú/', '/á/', '/â/', '/ã/', '/à/', '/ç/', '/é/', '/ê/', '/í/', '/ó/', '/ô/', '/õ/', '/ú/','/ /','([^[:alnum:]._])');
    $replace = array('A', 'A', 'A', 'A', 'C', 'E', 'E', 'I', 'O', 'O', 'O', 'U', 'a', 'a', 'a', 'a', 'c', 'e', 'e', 'i', 'o', 'o', 'o', 'u','_','');
    return preg_replace($search,$replace,$filename);


    What does this do?
    The $search variable defines the characters that requires replacing, you will notice the ([^[:alnum:]._]) at the end, this tells the preg_replace function to replace anything that is not a standard English character or number.
    The $replace variable defines what the replacement character is.
    preg_replace will basically count what item in the $search array that it has found and replace it with the equivalent in the $replace array, so for the example of "Ç", it is the 5th item int he $search array so it will replace it with the 5th item in the $replace array being "C".

    Note: you will see that each character is entered twice, once in upper case and once in lower case. This is because preg_replace is strict on character casing, this is because some applications using preg_replace may change the character case as part of the application, an example would be if you have a user defined input, some users may enter something like "HeLlO wOrLd", with the right string preg_replace can change this to "Hello World", "hello world", "HELLO WORLD" or any even "H***o W***d".

    You can make the $search and replace arrays as long as you need to, so if you really want to you can include all non-English characters as shown in this post on PHP.net. Keep in mind that with preg_replace being a regular expression, in the $search array you need the following format
    '/ITEM/',

    a search string with the '/ or the /' will return an error and the function will fail. So if you're going to copy and paste the example, make sure for the $search array you format the expression correctly before trying it.

    Cheers,
    Ryan - Ahgora Sistemas
    www.ahgora.com.br
  • Æ, Ø, Å and spaces in attachment files.

    Dear all,

    I have tried the suggested solution here, to add the following to class.format.php :

        function file_name($filename) {

    $search = array('/Á/', '/Â/', '/Ã/', '/À/', '/Ç/', '/É/', '/Ê/', '/Í/', '/Ó/', '/Ô/', '/Õ/', '/Ú/', '/á/', '/â/', '/ã/', '/à/', '/ç/', '/é/', '/ê/', '/í/', '/ó/', '/ô/', '/õ/', '/ú/','/ /','([^[:alnum:]._])');
    $replace = array('A', 'A', 'A', 'A', 'C', 'E', 'E', 'I', 'O', 'O', 'O', 'U', 'a', 'a', 'a', 'a', 'c', 'e', 'e', 'i', 'o', 'o', 'o', 'u','_','');
    return preg_replace($search,$replace,$filename);
    }


    However; whenever an attachment contains the characters Æ, Ø, Å (æøå) or a blank space, they will not display in osTicket (the ticket doesn't even show that the attachments exist).

    I have also tried to implement ryanp's ISO-8859-1 fallback solution to class.mailfetch.php , but this did not solve the problem....

    Hope for feedback, this is urgent for us ;-)
  • My solution

    I solved this issue in this way:


    function saveAttachments($ticket,$mid,$part,$index=0) {
    global $cfg;

    if($part && $part->ifdparameters && ($filename=$part->dparameters[0]->value)){ //attachment
    // Encoded http://greenbytes.de/tech/webdav/rfc5987.html
    if($part->dparameters[0]->attribute == 'FILENAME*') {
    $filename = urldecode($filename);

    if (substr($filename, 0, 5) != 'UTF-8')
    $filename = utf8_encode($filename);

    $filename = preg_replace(\"/.*'/\", '', $filename);
    }

    $index=$index?$index:1;
    if($ticket && $cfg->canUploadFileType($filename) && $cfg->getMaxFileSize()>=$part->bytes) {
    //extract the attachments...and do the magic.
    $data=$this->decode($part->encoding, imap_fetchbody($this->mbox,$mid,$index));
    $ticket->saveAttachment($filename,$data,$ticket->getLastMsgId(),'M');
    return;
    }
    //TODO: Log failure??
    }


    I also set function file_name() to do nothing, because actual file systems support special characters in filenames.

    Greetings,
    Juan Miguel Sosso
    www.binfactory.com
  • Long file names

    This morning we get a problem with long file names attachments with international characters.

    It's necesary to modify my last solution in this way:


    function getFilename($part)
    {
    if (isset($part->dparameters[0]->value)) {
    $filename = $part->dparameters[0]->value;

    // Encoded http://greenbytes.de/tech/webdav/rfc5987.html
    if(substr($part->dparameters[0]->attribute, 0, 9) == 'FILENAME*') {
    $i = 1;
    while (isset($part->dparameters[$i]))
    $filename .= $part->dparameters[$i++]->value;

    $filename = urldecode($filename);

    if (substr($filename, 0, 5) != 'UTF-8')
    $filename = utf8_encode($filename);

    $filename = preg_replace(\"/.*'/\", '', $filename);
    }
    }
    else {
    $filename = FALSE;
    }

    return $filename;
    }


    function saveAttachments($ticket,$mid,$part,$index=0) {
    global $cfg;

    $filename = $this->getFilename($part);

    if($part && $part->ifdparameters && $filename) { //attachment
    $index=$index?$index:1;
    if($ticket && $cfg->canUploadFileType($filename) && $cfg->getMaxFileSize()>=$part->bytes) {
    ...
  • thanks for sharing this fix. works like a charm.
  • need help with your solution

    @jmsosso
    Could you please write in which files you have implemented this code? I have similar problem with polish letters.

    Thanks in advance!
    Oskar
  • I'd like to apologize for this long standing issue. I was able to fix a bug yesterday concerning detection of attachments and filenames of inline attachments in mails that did not specify the Content-Disposition. Coincidently, I also added rfc5987 support to the filenames of piped mail attachments.

    Today, I was able to port the fix to the mail fetching side. The fix is posted here, https://github.com/osTicket/osTicket-1.7/pull/738, and will be released into osTicket 1.7.1.5. It probably wouldn't be difficult to get a patch from git and apply it to osTicket 1.6 if upgrading to 1.7 isn't a viable option.

    Thanks for posting the details of the issue. They were helpful in addressing the root cause. Please post software coding bugs on our GitHub page so we have a centralized place to track and correct software defects.

    Cheers,
This discussion has been closed.