tag:status.leanplum.com,2005:/historyLeanplum Status - Incident History2024-03-28T05:51:26-07:00Leanplumtag:status.leanplum.com,2005:Incident/177277302023-06-30T07:00:22-07:002023-06-30T07:00:22-07:00Partner Integrations and Data Feed maintenance<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>07:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>01:48</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance for all Partner Integrations. During this time there will be no audience syncs and exports to partners (including the data feeds) would not work. Data imports and session exports to AWS will not be impacted.<br />When the maintenance finishes, there will be no data loss - all exports will be sent and audiences will continue to sync. We expect the maintenance to continue for up to 5 hours and will provide an update once finished.</p>tag:status.leanplum.com,2005:Incident/176967082023-06-27T11:32:39-07:002023-06-27T11:32:39-07:00Data Freshness Delay<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:32</var> PDT</small><br><strong>Resolved</strong> - The data freshness is back to normal levels as of 2:30pm UTC.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>05:16</var> PDT</small><br><strong>Update</strong> - Correction - the data was last updated at 07:00 UTC on Tuesday 27th June.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>01:03</var> PDT</small><br><strong>Identified</strong> - We are currently experiencing data freshness delays in our Analytics dashboard. Data was last updated at 23:00 UTC on Monday 26th June. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/176547732023-06-26T05:00:14-07:002023-06-26T05:00:14-07:00Scheduled infrastructure upgrade<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>08:11</var> PDT</small><br><strong>Scheduled</strong> - We are upgrading pieces of Leanplum infrastructure and some services of secondary importance MAY not be available for short intervals during the maintenance window.</p>tag:status.leanplum.com,2005:Incident/176211522023-06-20T05:00:21-07:002023-06-20T05:00:21-07:00Scheduled infrastructure upgrade<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>07:47</var> PDT</small><br><strong>Scheduled</strong> - We are upgrading pieces of Leanplum infrastructure and some services of secondary importance MAY not be available for short intervals during the maintenance window.</p>tag:status.leanplum.com,2005:Incident/175934362023-06-19T05:00:17-07:002023-06-19T05:00:17-07:00Scheduled infrastructure upgrade<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>05:46</var> PDT</small><br><strong>Scheduled</strong> - We are upgrading pieces of Leanplum infrastructure and some services of secondary importance MAY not be available for short intervals during the maintenance window.</p>tag:status.leanplum.com,2005:Incident/174732732023-06-04T19:14:35-07:002023-06-04T19:14:35-07:00API unavailable<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>19:14</var> PDT</small><br><strong>Resolved</strong> - The API issue has been resolved and the platform is back to normal API performance.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>16:25</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>16:16</var> PDT</small><br><strong>Update</strong> - We are working on a resolution with help from Google Cloud support</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>13:30</var> PDT</small><br><strong>Monitoring</strong> - We are facing operational issues with Google Kubernetes Engine (GKE).</p>tag:status.leanplum.com,2005:Incident/169656092023-04-27T07:41:28-07:002023-04-27T07:41:28-07:00Campaign results maintenance<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>07:41</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>25</var>, <var data-var='time'>13:00</var> PDT</small><br><strong>Scheduled</strong> - We are planning a maintenance of campaign results to upgrade some of our infrastructure and increase reliability. During this time our data processing infrastructure will become unavailable, which will cause our campaign result, campaigns and messaging overview pages to show stale data. No data will be lost and once we are back online, all data accumulated during the period of the maintenance will be ingested and available on the dashboard. Please contact support@leanplum.com if you have any questions.</p>tag:status.leanplum.com,2005:Incident/161366602023-02-14T13:21:16-08:002023-02-14T13:21:16-08:00Degraded Dashboard Performance<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>13:21</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>09:13</var> PST</small><br><strong>Monitoring</strong> - We have identified and mitigated the root cause for the dashboard performance issues. We expect to be back to normal dashboard performance by 09:30 pm UTC</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>06:18</var> PST</small><br><strong>Investigating</strong> - We are investigating reports of decreased dashboard performance on the Analytics page. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/160739742023-02-09T04:55:02-08:002023-02-09T04:55:02-08:00Issues with Audience Insights Page<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>04:55</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>01:35</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing a problem with Audience Insights Page. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/135473532022-11-21T06:22:45-08:002022-11-21T06:22:45-08:00Campaign result and overview page delays<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>06:22</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>01:21</var> PST</small><br><strong>Identified</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p>tag:status.leanplum.com,2005:Incident/112255172022-09-22T15:30:11-07:002022-09-22T15:30:11-07:00Data Freshness Delay<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Resolved</strong> - The data freshness issue has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>14:20</var> PDT</small><br><strong>Monitoring</strong> - We have identified and mitigated the root cause for the data freshness delay. We are monitoring the data ingestion progress and expect to be fully caught up by 15:30 PST.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>14:19</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing data freshness delays in our audience targeting for scheduled messages. Audience page freshness is also affected. Data was last updated at 10:20 PST.</p>tag:status.leanplum.com,2005:Incident/112082342022-09-21T08:21:57-07:002022-09-21T08:22:24-07:00Segmentation outage<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>08:21</var> PDT</small><br><strong>Resolved</strong> - From 3AM UTC on 20 Sep till 2PM UTC on 21 Sep all new unique event names, state names and attribute names were not ingested properly.</p>tag:status.leanplum.com,2005:Incident/111956932022-09-20T14:57:57-07:002022-09-20T14:57:57-07:00Segmentation Outage<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>14:57</var> PDT</small><br><strong>Resolved</strong> - The incident has been resolved and Segmentation freshness is back to normal.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>12:23</var> PDT</small><br><strong>Monitoring</strong> - We have implemented and deployed a fix addressing the root cause. Segmentation freshness will remain high in the next hours as we continue to monitor the system.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>10:26</var> PDT</small><br><strong>Identified</strong> - We are currently experiencing an outage related to Segmentation. Scheduled messages and campaigns are impacted as well as Audiences estimates. In order to recover, you will experience increased Segmentation freshness over the next few hours. We apologize for the inconvenience and will update this status page as soon as we have resolved the root cause.</p>tag:status.leanplum.com,2005:Incident/107596922022-08-09T02:37:42-07:002022-08-09T02:37:42-07:00Campaign results maintenance<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>02:37</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>06:03</var> PDT</small><br><strong>Scheduled</strong> - Campaign results will undergo a scheduled maintenance during the specified window. During the maintenance event we will not be ingesting new data, but any data that was already available before that time will remain available. The maintenance aims to improve the reliability of this component by upgrading some of the services it uses.</p>tag:status.leanplum.com,2005:Incident/106758662022-07-23T20:23:09-07:002022-07-23T20:23:09-07:00Campaign result and overview page delays<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>20:23</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>08:24</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring it.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>07:48</var> PDT</small><br><strong>Investigating</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p>tag:status.leanplum.com,2005:Incident/102095402022-06-09T20:11:12-07:002022-06-09T20:11:12-07:00Degraded System Performance<p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>20:11</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>20:01</var> PDT</small><br><strong>Update</strong> - Push Notifications and Email are operational, however we are continuing to investigate the issue</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>17:04</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing a full platform outage. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/97115952022-04-05T02:34:26-07:002022-04-05T02:34:26-07:00Campaign result and overview page delays<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>02:34</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>02:28</var> PDT</small><br><strong>Investigating</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>Identified</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p>tag:status.leanplum.com,2005:Incident/95935332022-03-21T05:05:10-07:002022-03-21T05:05:10-07:00Message Delivery Delay<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>05:05</var> PDT</small><br><strong>Resolved</strong> - The message delivery delays have been resolved and the platform is back to normal delivery performance.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>04:46</var> PDT</small><br><strong>Monitoring</strong> - We have identified and mitigated the root cause for the message delivery delays. The delay is decreasing and we expect to be back to normal delivery performance within 1 hour.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>04:38</var> PDT</small><br><strong>Investigating</strong> - We are investigating reports of delivery delays of up to 2.5 hours with scheduled messages and campaigns. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/95581162022-03-16T13:32:05-07:002022-03-16T13:32:05-07:00Campaign result and overview page delays<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:32</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:54</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring it.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:12</var> PDT</small><br><strong>Identified</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p>tag:status.leanplum.com,2005:Incident/92917242022-02-11T11:41:29-08:002022-02-11T11:41:29-08:00Problem with Segmentation<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>11:41</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved and freshness is up to date.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>07:49</var> PST</small><br><strong>Monitoring</strong> - Issue has been addressed and fixed. There is currently a freshness delay which will be resolved soon.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>06:52</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>06:51</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing a platform wide problem with Segmentation. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/91104662022-01-31T10:42:39-08:002022-01-31T10:42:39-08:00Data Freshness Delay<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:42</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>04:10</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>09:36</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented. We have huge backlog of data to be computed. We are monitoring it.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>08:01</var> PST</small><br><strong>Identified</strong> - The issue has been identified and we are working on a fix.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>07:27</var> PST</small><br><strong>Update</strong> - We are actively working on the issue and we are continuing the investigation.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>03:49</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing data freshness delays in our Analytics dashboard. The data shown might be inaccurate. The data is not lost. We have an issue with our reporting service and we are working on fixing the issue ASAP.</p>tag:status.leanplum.com,2005:Incident/92011962022-01-31T02:15:41-08:002022-01-31T02:15:41-08:00Audiences Degraded System Performance<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>02:15</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>01:00</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>01:00</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:49</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing a degraded performance on Audience related functionality. We apologise for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/91673142022-01-26T07:50:17-08:002022-01-26T07:50:17-08:00Triggered campaigns and messages delay<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:50</var> PST</small><br><strong>Resolved</strong> - A fix has been deployed. Delayed messages have been sent and the system seem to be back to normal.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>04:09</var> PST</small><br><strong>Investigating</strong> - There is a delay in the triggered campaigns and single messages, 95th percentile - 8.5h, 50th percentile - ~3h. It has started yesterday 01-25-2022 at around 14:00 UTC. We are currently working on a fix.</p>tag:status.leanplum.com,2005:Incident/90409632022-01-09T21:34:48-08:002022-01-09T21:34:48-08:00Data Freshness Delay<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>21:34</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>13:07</var> PST</small><br><strong>Identified</strong> - We are currently experiencing data freshness delays in our Analytics dashboard. Data was last updated at 8PM UTC. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.</p>tag:status.leanplum.com,2005:Incident/90254732022-01-08T13:12:02-08:002022-01-08T13:12:02-08:00Campaign result and overview page delays<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>13:12</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>04:34</var> PST</small><br><strong>Identified</strong> - We have resolved the root cause for this issue and are now catching up on the backlog accumulated while the issue lasted. We expect to be back to normal freshness within the next 6 hours</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>05:17</var> PST</small><br><strong>Investigating</strong> - We are experiencing delays with our data processing infrastructure, which are causing our campaign result and overview page to be behind. The engineering team is actively working on this. Please contact support@leanplum.com if you have any questions.</p>