Email notifications stopped working

  • 0
  • 1
  • Question
  • Updated 2 months ago
  • Acknowledged
Email notifications stopped working after QB switch data center last night.  Does anyone has the same problem?
Photo of Hing Chan

Hing Chan

  • 180 Points 100 badge 2x thumb
  • Not very happy

Posted 2 months ago

  • 0
  • 1
Photo of Kevin O'Boyle

Kevin O'Boyle

  • 602 Points 500 badge 2x thumb
We have the same issue. Just one of our apps that Quickbase has in quarantine
Photo of Slider

Slider

  • 1,326 Points 1k badge 2x thumb
Are they set as Active? 
Please keep us informed if this has been resolved.


Photo of Kevin O'Boyle

Kevin O'Boyle

  • 602 Points 500 badge 2x thumb
Our notifications fired at 11:30pm last night so it appears it was something on the QB side
Photo of Glenn

Glenn

  • 60 Points
Seems emails for password reset and add new users invitations are not firing either. G

Photo of Evan Martinez

Evan Martinez, Community Manager

  • 8,774 Points 5k badge 2x thumb
Hi Hing and Kevin,

If you are experiencing any issues with notifications after the data center change we do encourage you to put in a case with our Care Team so they can help to diagnose the issues. I would also suggest making sure all of the email server ip addresses listed in our Knowledge Base article are whitelisted by your email provider, those ip addresses can be found here

You can put in a case with our Care Team by clicking the question mark icon that appears at the top, right hand corner of the page when signed into a Quick Base application by clicking Manage Support case or you can also put in a case here. This way we can help you address any issues with notifications not firing or system emails (like password resets) not going through. I'm very sorry to hear you had this issue over the weekend. 
Photo of Hing Chan

Hing Chan

  • 180 Points 100 badge 2x thumb
I did put in a support ticket when our users noticed the email problem last Saturday.  We have mission critical processes that run 24/7 and we rely heavily on QB for our communications.  Submitting a supporting ticket was more of a formality and CYA I guessed since there is no weekend support from QB and no one was monitoring QB technical problems.  We had to resort to find QB CTO in LinkedIn to escalate the problem.
Photo of Hing Chan

Hing Chan

  • 180 Points 100 badge 2x thumb

FYI from Kirk Trachy

Root Cause:

A configuration error prevented about 10% of customer e-mails from being routed to our e-mail service provider.  We had already automated the implementation of that particular configuration to avoid issues like this but some legacy mis-configuration existed that we had missed.  That mis-configuration has been corrected.  We will also be implementing improved monitoring, both to detect this specific case and to refine the monitoring and alerting we have in place to detect when anomalous e-mail volume occurs, either too much or too little.