CentOS 7.x Zimbra mail queue management

<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>

CentOS 7.x Zimbra mail queue management

Graphical access to Zimbra email queue

For graphical access to mail queue login as admin user and go to Monitor -> Mail queues. Here Deferred queue is important and shown first. This queue contains emails for which delivery was attempted and failed. The option is to filter emails by Receiver domain, Origin IP, Sender domain, Receiver address, Sender address and Error. We can click on a particular item (eg particular receiver domain) and in the messages pane below only deffered emails related to that particular item are shown. If we hover mouse over any such deferred message row in messages list then error is displayed properly.

Reading particular email using command-line

If it is required to read email then we should note hexadecimal ID of the message in the below list. Then go to folder:

cd /opt/zimbra/data/postfix/spool/ 

to see various folders related to various queues. For deferred emails visit deferred/<first-hex-letter> sub-folder inside above folder. Then use either:

less <message-ID>

while accepting warning to display binary message or use less-readable

xxd <message-ID>

Avoid use of cat as it might change terminal settings due to binary data. In case terminal settings are changed exit and reconnect. Other option is to use 'reset' command to reset terminal settings.

There is older article on mail queue management at Zimbra mail queue management

Clearing long queue after attack or when the current active or incoming or deferred etc. queue are too big

If you have recently faced attack (Often due to a particular user-ID / password getting leaked to attacker) then it is possible that Zimbra mail queue might become very big. In such cases to clear the queue so that normal email functioning starts without loosing any of the important emails use:

1. Login into Zimbra admin dashboard and look at email queue. If there are many emails from same source IP, deny incoming emails from that IP using:

   firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='185.235.15.224' reject"
   #firewall-cmd --add-rich-rule='rule family="ipv4" source address="185.235.15.224" reject'
   firewall-cmd --reload

   Refer:

   • https://fedoraproject.org/wiki/Features/FirewalldRichLanguage#Handle_rich_rules_with_the_command_line_client
   • https://access.redhat.com/discussions/1342573
2. Validate that you have not blocked one or your own office IPs using: https://www.whatismyipaddress.com/ip/<blocked-ip>/ after replacing <blocked-ip> appropriately.
3. Restart zimbra on another shell for server incoming connections to drop

   su - zimbra
   zmcontrol restart

4. Go to various folders and sub-folders inside postfix/spool and move emails to hold folder

   cd /opt/zimbra/data/postfix/spool/active
   mv * ../hold
   
   cd /opt/zimbra/data/postfix/spool/defer
   #defer has 0 to f sub-folders inside
   for A in *; do mv -f $A/* ../hold; done
   
   cd /opt/zimbra/data/postfix/spool/deferred
   #deferred has 0 to f sub-folders inside
   for A in *; do mv -f $A/* ../hold; done
   
   cd /opt/zimbra/data/postfix/spool/incoming
   mv * ../hold

5. From hold folder move SPAM emails to another folder for deletion. Note that use grep on * may not work if the no. of SPAM is too high with below output:

   mkdir /root/2019-04-30-to-be-deleted-spam/
   [root@mail hold]# grep -l 185.235.15.224 * | xargs -I mv {} /root/2019-04-30-to-be-deleted-spam/
   -bash: /usr/bin/grep: Argument list too long

   Hence choose a smaller subset based on prefix such as:

   grep -l 185.235.15.224 00* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/

   Look at email count in to-be-deleted folder

   ls -1 /root/2019-04-30-to-be-deleted-spam/ | wc -l

   Also look at email count in hold queue:

   ls -1 | wc -l

6. Ideally no. of emails to be deleted in /root/2019-04-30-to-be-deleted-spam folder should increase and emails in hold folder should decrease over time.
7. Move emails with all prefixes 0 to f using:

   grep -l 185.235.15.224 0* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 1* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 2* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 3* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 4* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 5* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 6* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 7* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 8* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 9* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 A* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 B* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 C* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 D* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 E* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/
   grep -l 185.235.15.224 F* | xargs -I{} mv {} /root/2019-04-30-to-be-deleted-spam/

8. Again check "wc -l" on important folders and "Refresh" mail queue on Zimbra Admin web dashboard. Now again search for bad origin or source IPs and filter emails based on them:

   grep -l 105.112.33.150 * | wc -l
   mv $(grep -l 105.112.33.150 *) /root/2019-04-30-to-be-deleted-spam/

9. Again ensure that your own office IP is not blocked using: https://www.whatismyipaddress.com/ip/<blocked-ip>
10. If web browser queue refresh is not working use:

    su - zimbra
    zmmtactl restart

    and try to refresh mail queue in admin dashboard again.

11. Often email responses with 'timed out' or 'refused to talk to me' can be part of hold queue since we moved everything. These can again be moved to deleted folder using:

    mkdir /root/2019-04-30-to-be-deleted-error-responses
    mv $(grep -l '^status=[0-9]\.[0-9]\.[0-9]' *) /root/2019-04-30-to-be-deleted-error-responses/

    If you search for sasl_username in most of these messages and it is the same user id then most likely that ID is compromised. We should reset password of that ID immediately and inform user about it using other communication channels.

12. Other option is to grep for different types of error messages and move emails containing those messages. Ideally the status option suggested above should capture all the below types of messages automatically.

    mv $(grep -l 'timed out' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'refused to talk to me' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'lost connection' * ) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'no relay available' * ) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'temporarily deferred' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'the message has been blocked' * ) /root/2019-04-30-to-be-deleted-error-responses/
    
    mv $(grep -l 'Temporarily unable to process your email' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Could not complete sender verify callout' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'does not have an MX record' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Connection refused' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Host or domain name not found' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'No route to host' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Client host rejected' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'resource temporarily unavailable' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Relay access denied' *) /root/2019-04-30-to-be-deleted-error-responses/
    mv $(grep -l 'Transient reject by behaviour spam' *) /root/2019-04-30-to-be-deleted-error-responses/

13. Move domain specific emails to spam folder:

    mv $(grep -l emailsecurity.com *) /root/2019-04-30-to-be-deleted-spam/

14. Requeue emails from held queue which are legitimate
15. Create folder for emails that could be important but are not readable by graphical wizard (ID shown in messages pane below):

    /root/2019-04-30-could-be-important
    mv 6C6C3F10AFBD /root/2019-04-30-could-be-important/

16. Ideally hold graphical pane should show details of sender, receiver, etc. and we should be able to requeue based on that information.

<yambe:breadcrumb self="Zimbra mail queue management">CentOS_7.x_Zimbra_configuration|Zimbra configuration</yambe:breadcrumb>