tag:status.findify.io,2005:/historyFindify by Maropost Status - Incident History2024-03-29T06:23:14+01:00Findify by Maroposttag:status.findify.io,2005:Incident/198151552024-01-24T13:33:36+01:002024-01-24T13:33:36+01:00Pulsar experiencing issues causing delays in Sync & Analytics<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>13:33</var> CET</small><br><strong>Resolved</strong> - The incident have now been fully resolved. <br />Actions are taken to avoid similar issues.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:31</var> CET</small><br><strong>Monitoring</strong> - Sync running as normal</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:19</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.findify.io,2005:Incident/196435122024-01-08T10:00:51+01:002024-01-08T10:00:51+01:00Hyperdrive issue affects Norce Product Sync<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - Norce has resolved the issue and product sync is now running again.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>09:34</var> CET</small><br><strong>Identified</strong> - Norce Support team are working on identifying the issue and restoring the Hyperdrive access</p>tag:status.findify.io,2005:Incident/193743372023-12-08T09:49:15+01:002023-12-08T09:49:29+01:00Temporary Delay in Sync<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>09:49</var> CET</small><br><strong>Resolved</strong> - Some clients experience delay in sync updates. <br />The problem has been identified to a memory profile in the sync infrastructure & resolved. <br />The memory has been increased & alarm implemented to prevent future problems. <br /><br />The search API has been unaffected and search, collections & recommendation results has been continuously served.</p>tag:status.findify.io,2005:Incident/191252782023-11-15T09:33:29+01:002023-11-15T09:33:29+01:00Norce Hyperdrive Error<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:33</var> CET</small><br><strong>Resolved</strong> - The issue has been resolved by Norce Team.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>09:16</var> CET</small><br><strong>Investigating</strong> - All Norce hyperdrive feeds are returning errors and can hence not be downloaded/synced. <br />The Norce team is investigating</p>tag:status.findify.io,2005:Incident/186776552023-10-03T15:41:40+02:002023-10-03T15:42:04+02:00Partial issues with custom Sync<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>15:41</var> CEST</small><br><strong>Resolved</strong> - An update has been released and the issue is resolved</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>14:53</var> CEST</small><br><strong>Identified</strong> - A few merchants running on custom feed sync is currently experience problems to update products. <br />The root cause has been identified and the fix is being tested.</p>tag:status.findify.io,2005:Incident/179109132023-07-20T05:00:00+02:002023-08-03T13:49:26+02:00Product Sync Affected Partially<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>05:00</var> CEST</small><br><strong>Resolved</strong> - A subset of our merchants experienced interruptions in product sync. All other components of Findify were working as intended. The issue was identified and resolved.</p>tag:status.findify.io,2005:Incident/176871182023-06-15T17:00:00+02:002023-08-03T13:49:26+02:00Partial outages in the US region<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>17:00</var> CEST</small><br><strong>Resolved</strong> - In the afternoon of the 15th June 2023, we had an incident which caused partial outages in the US region. This was caused by an outburst of search requests which overloaded our Redis Connections. The issue was automatically detected by our internal alarming & immediately resolved where full recovery took 30-60 minutes to fully apply.<br />No components were marked as affected by this incident. (.)</p>tag:status.findify.io,2005:Incident/176871682023-05-29T10:46:00+02:002023-08-03T13:49:26+02:00Cluster Degraded Performance<p><small>May <var data-var='date'>29</var>, <var data-var='time'>10:46</var> CEST</small><br><strong>Resolved</strong> - We are experiencing performance issues with Nodes in one cluster and a few clients are affected. <br />Clients have been moved to functional clusters and the team are identifying the root cause.</p>tag:status.findify.io,2005:Incident/176871862023-05-11T10:48:27+02:002023-08-03T13:49:26+02:00API Disrupted<p><small>May <var data-var='date'>11</var>, <var data-var='time'>10:48</var> CEST</small><br><strong>Resolved</strong> - At 10:47 we published an updated internal URL containing additional data for new functionality. This change was reviewed, tested & validated in our staging environment without any problems. Once released, our internal alarming immediately notified that clients were affected. The change was immediately rolled back but took a few minutes to take effect. <br /><br />At 11:03 service was confirmed to be back up running.<br /><br /><br />We take this very seriously and are investigating why the initial tests didn't correctly reflect the outcome. Until that is resolved we will not publish any updates at all.</p>tag:status.findify.io,2005:Incident/176872202022-12-08T12:49:15+01:002023-08-03T13:49:26+02:00Redis Error affecting authentication<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:49</var> CET</small><br><strong>Resolved</strong> - 22:09 An error in Redis are affecting authentication.<br /><br />22:14 The issue has been identified and a fix is being implemented.<br /><br />22:19 80% restored<br /><br />22:23 100 % restored.<br /><br />22:28 The incident is validated to be fully restored. Monitoring is continued.</p>tag:status.findify.io,2005:Incident/176872472022-11-25T17:30:00+01:002023-08-03T13:49:26+02:00Autoscaling Deficiency<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>17:30</var> CET</small><br><strong>Resolved</strong> - During high volume inflow we experienced delays in the autoscaler which lead to outage of our main load balancer. <br />This resulted in 25min of index disruptance and failure to readily serve product results. <br /><br />The team has scaled all systems manually and all functions are operational. <br />We are further investigating why the autoscaler failed.</p>tag:status.findify.io,2005:Incident/176872572022-09-14T16:42:00+02:002023-08-03T13:49:26+02:00Index not fully updated<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>16:42</var> CEST</small><br><strong>Resolved</strong> - 16:42 Since a few minutes back we have an issue with indexing. Some clients are affected where indexing is not happening for the whole product catalogue. We are investigating this issue<br /><br />16:54 The issue has been identified and a fix is being implemented.<br /><br />16:57 A fix has been implemented and we are monitoring the results.<br /><br />17:32 This incident has been validated to be resolved.</p>tag:status.findify.io,2005:Incident/176872632022-06-28T14:02:00+02:002023-08-03T13:49:26+02:00Search API - Access control layer<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>14:02</var> CEST</small><br><strong>Resolved</strong> - Today in the European morning (North American night) we had an issue in the access control layer, which affected the search api causing it to lose access to the configuration service. The team jumped on the issue immediately and it was solved within 20 minutes during which the service was partially unavailable. We are changing the access control method, to make it less error prone and more resilient in the future.</p>tag:status.findify.io,2005:Incident/176873132022-05-31T20:28:00+02:002023-08-03T13:49:26+02:00Depreciated search service<p><small>May <var data-var='date'>31</var>, <var data-var='time'>20:28</var> CEST</small><br><strong>Resolved</strong> - We just experienced a huge traffic spike (larger than we get on black Friday) that for a few minutes resulted in an increased error rate. We have increased capacity in the affected components, fine tuned the autoscaling and will now look into the source of the traffic to understand if we need to prepare for similar or even larger spikes going forward.</p>tag:status.findify.io,2005:Incident/176873462022-05-16T15:50:00+02:002023-08-03T13:49:26+02:00Increased Error rate Search<p><small>May <var data-var='date'>16</var>, <var data-var='time'>15:50</var> CEST</small><br><strong>Resolved</strong> - 15:50 Some clients are experiencing an increase error rate. <br />We are looking into the root cause<br /><br />15:53 This incident has been resolved.</p>tag:status.findify.io,2005:Incident/176873592022-05-10T11:12:00+02:002023-08-03T13:49:26+02:00Access to Admin API<p><small>May <var data-var='date'>10</var>, <var data-var='time'>11:12</var> CEST</small><br><strong>Resolved</strong> - 11:12 We are currently experiencing an issue with the admin API. This may prevent users to log into the dashboard.<br /><br />11:19 The issue has been identified and a fix is being implemented.<br /><br />12:42 A fix has been implemented and we are monitoring the results.<br /><br />13:06 This incident has been resolved.</p>tag:status.findify.io,2005:Incident/176873682022-01-18T21:16:00+01:002023-08-03T13:49:26+02:00Service degradation search<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>21:16</var> CET</small><br><strong>Resolved</strong> - 21:16 We have a service degradation on a number of clients. <br />We are working to identify the root cause.<br /><br />21:29 This incident has been resolved.<br />We will continue to monitor the performance.</p>tag:status.findify.io,2005:Incident/176873812021-12-23T10:02:00+01:002023-08-03T13:49:27+02:00Analytics are not loaded for some of the merchant in the Merchant Dashboard<p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>10:02</var> CET</small><br><strong>Resolved</strong> - 10:02 We currently have an issue with the service that handles Findify analytics. For now Findify is unable to display some analytics in the merchant dashboard. The data is still available and we are working to have it restored. This does not affect the core functionality of search, collections and recommendations on site.<br /><br />10:36 The issue has been resolved.</p>tag:status.findify.io,2005:Incident/176873852021-12-13T15:54:00+01:002023-08-03T13:49:27+02:00Analytics in merchant dashboard are not loading<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>15:54</var> CET</small><br><strong>Resolved</strong> - We currently have an issue with the service that handles Findify analytics. For now Findify is unable to display any analytics in the merchant dashboard. There data is still available and we are working to have it restored. This does not affect the core functionality of search, collections and recommendations on site.<br /><br />Update: The issue was resolved over the weekend and the data is now restored.</p>