Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

In this Discussion

osTicket v1.10 (stable) and Maintenance Release v1.9.15 are now available! Go get it now

Can't import existing e-mails

Hello!

I have a fresh installation of osticket. I also have a mailbox of ~65,000 e-mails that I wish to import. However, it seems that I have a problem with the import.

Around ~17,000 e-mails get imported, but no more. I saw an error about an e-mail loop, or something similar, but I accidentally deleted that log entry and it won't re-appear again (it was dumb of me, now I know that). However, aside from "there's an e-mail loop" there was no information about the message itself. 

To import I'm using the PHP cron job, running the script every minute. Therefore, I have a couple of questions:

* What are the steps to debug this problem?
* What could possibly be wrong, that's causing my import to fail?

Now I assume that the same "bug" can happen to me on new e-mails, and the last thing I want is to stop receiving new e-mails to osticket. It's either a bug in osticket, or my missconfiguration - I assume it's the latter, as I expect that if it was a bug in osticket more people would notice it.

Bonus question: Has anybody imported a database of EXISTING e-mails while persisting the threads? I couldn't find any information on that, and it would be great if I could retain thread information in form of tickets. The general idea is to import last couple of years of e-mails to ostickets so that agents would have a single point where they could search through old e-mails history. 


Comments

  • Please help us to help you by reading and following the posting guidelines located in this thread: Please read before requesting assistance.  The more information you give us the better we will be able to assist you. Thank you.

    How many emails are you trying to process at once?  If you are running cron every minute then maybe you have it set way to high?
  • Hi,

    I tried various settings, and it never does anything. 

    Basically I have Fetching Email via IMAP or POP enabled, through IMAP, credentials are OK and SMTP server is not complaining (I have access to logs). It's set to import up to 100 e-mails per 1 minute right now. 

    I try to do the "mass import" using Recurring Tasks Scheduler, but I also tried other ways to make sure it's not related to this particular method - it looks like it isn't.

    I tried looking through PHP error logs, but there's nothing from osticket. Like I mentioned before, there's also nothing in dovecot logs 

    Some basic information about the machine:

    PHP Version: PHP Version 5.6.24 (behind nginx, via fastcgi, php-fpm)
    OS: FreeBSD 10.3-RELEASE-p3, amd64
    IMAP server: dovecot 2.2.25 (7be1766)
    Basically all crucial PHP modules are installed. I'd like not to post whole phpinfo file for security reasons, although if there's something crucial you need from it I can most likely post it.

    So, just tu summarize: around 17k emails from 65k total went through just fine (it took some time, but it worked as expected). At some point, probably, osticket encountered an e-mail it couldn't process for some reason - however, I don't know what that e-mail is or why it doesn't work. If I knew that I could probably come up with some workaround.

    Then again, I'm not sure if the way I try to import old e-mail threads to osticket is the right way to go?
  • What have you configured in the "Fetched Emails" section?

    I would expect when you configure it to "Move to folder" (is recommended) the import process will break again and you could see by the content of the inbox folder which email is making problems.

    Processed emails are in the import folder, unprocessed stay in the inbox folder. 
    When the import has stopped, look in the inbox folder and move the latest or first mail (don't know which order the importer uses) temporarily to another folder to see if the import proceeds again. 
    Then you know which mail is the problem.
  • I do have moving enabled, so I'll try your solution. How can I, however, know what is wrong with a particular e-mail? Is there any debug log that I can look into? The default "debug log" visible in osticket admin dashboard doesn't say anything.
  • I'm not aware that osTicket has a debug log for this. I could imagine that you find this information in a debug or transaction log of the mailserver you are using (if you have the possibility to access those ressources)..
  • edited September 7
    I do have access, but I only see information about logging in and out, so there's nothing really helpful there - only information about osticket logging in and out.

    So I did some debugging and it looks like I'm hitting this condition: https://github.com/osTicket/osTicket/blob/develop/include/class.mailfetch.php#L778

    Does this mean that it's trying to process again an e-mail, that has already been processed, for some reason? Since I have my osticket set to move e-mails to a separate folder, this would mean that the move has failed, am I correct? 
  • Also, I modified code a bit to make it print the message id + subject of e-mail it's trying to process. I can see the script tries to re-process the same messages over and over again.
  • I'm not a developer, so I can't assist you with understanding of the code. 

    But I can imagine that your setting with fetching 100 Mails per fetch is making it harder to find the "black sheep".
    I would think it is processing this 100 mails again and again because the problematic mail (black sheep) is under this mails.
    Try to reduce it temporarily  to "1" to find the problematic mail. I guess it will proceed with fetching till the problematic mail is reached and then only the problematic mail will be processed again and again. That could be a way to find it.

    As a side note: nginx is not officially supported (but I don't think this is the reason for your problem).


  • Unless you have osTicket move the emails to a sub folder or delete them it will in fact have to process all the emails again and again and again and again.  Your best bet would be to have it move them to a sub folder or delete them and make the batch a lot smaller.  Some times emails with malformed headers can cause a fetch error.


Sign In or Register to comment.