tag:status.crossref.org,2005:/historyCrossref Status - Incident History2024-03-28T18:40:28ZCrossreftag:status.crossref.org,2005:Incident/203742022024-03-27T21:26:04Z2024-03-27T21:26:04ZSome metadata submissions missed reference matching<p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>21:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>18:35</var> UTC</small><br><strong>Monitoring</strong> - We have rerun the reference matching for those affected submissions from yesterday and are monitoring the results.</p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>09:42</var> UTC</small><br><strong>Identified</strong> - Around 21:30 UTC yesterday, we had problems processing reference matching for some submissions in our metadata deposit system. We will be digging into the details today and will update here when we have re-run and processed the affected records. Other deposits are coming through as normal now.<br />-GH</p>tag:status.crossref.org,2005:Incident/203521312024-03-26T16:00:02Z2024-03-26T16:00:02Z26 March - weekly maintenance, release v0.194.0<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>15:57</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.194.0 - on Tuesday, 26 March from 1400 to 1600 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/202138252024-03-13T17:14:36Z2024-03-13T17:14:36ZBug found in Cited-by counts<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>17:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Identified</strong> - We've identified the cause and are implementing a fix.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:08</var> UTC</small><br><strong>Investigating</strong> - Since Friday 8th March, we have received some reports of inconsistent or unexpected Cited-by counts. The /works endpoint of our API is intermittently returning incorrect Cited-by counts in a small number of cases (in the fields is-referenced-by in JSON and crm-item name="citedby-count" in XML). The incorrect values are very large, e.g. 842,611,252. Sorry for the inconvenience; we are currently investigating and trying to replicate the problem and will fix or update here ASAP.</p>tag:status.crossref.org,2005:Incident/202138822024-03-12T17:37:08Z2024-03-12T17:37:08ZMetadata Search outage for 24 hours<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Resolved</strong> - We found the issue which was related to an instability with DNS. We applied a fix yesterday and the service is stable. Resolving this issue. Thanks for your patience!<br /><br />-GH</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Monitoring</strong> - Metadata Search (search.crossref.org) suffered a complete outage from about 10.15 AM UTC on Sunday 10 March until 10.30 AM UTC on Monday 11 March. We have deployed a fix for this issue and are continuing to monitor the situation.</p>tag:status.crossref.org,2005:Incident/202164082024-03-12T15:11:33Z2024-03-12T15:11:33Z12 March - weekly maintenance, release v0.193.0<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>15:11</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.193.0 - on Tuesday, xx MONTH from 1400 to 1600 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/201592712024-03-05T16:44:34Z2024-03-05T16:44:34Z5 March - weekly maintenance, release v0.192.0<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>16:44</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.192.0 - on Tuesday, 5 March from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/198167852024-02-28T09:22:17Z2024-02-28T09:22:17ZIntermittent network issues<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>09:22</var> UTC</small><br><strong>Resolved</strong> - Resolving this for now as the network has been stable for over three weeks.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>21:22</var> UTC</small><br><strong>Update</strong> - Our main website - https://www.crossref.org/ - was down from 1835 to 1843 UTC today. We believe this downtime was related to the intermittent network issues we've been investigating.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>15:09</var> UTC</small><br><strong>Update</strong> - Everything is now stable, but it is possible that this issue may recur in the near future. Because of that, we're going to leave this incident notification up for the time being while we continue to investigate the cause and work on a permanent fix.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>14:07</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>11:32</var> UTC</small><br><strong>Investigating</strong> - We seem to be experiencing intermittent network issues in our data centre which has caused brief interruptions to all services. Our infrastructure team are working to understand the cause and to resolve the problem.</p>tag:status.crossref.org,2005:Incident/200800582024-02-27T17:00:56Z2024-02-27T17:00:56Z27 February - weekly maintenance, release v0.191.0<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>16:12</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.191.0 - on Tuesday, 27 February from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/200768632024-02-26T21:57:23Z2024-02-26T21:57:23ZProblems with search.crossref.org<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>21:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>10:55</var> UTC</small><br><strong>Monitoring</strong> - search.crossref.org seems to be stable again. We think this problem was caused by heavy traffic to our APIs, so we have added extra resources and will continue to monitor.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>09:10</var> UTC</small><br><strong>Investigating</strong> - search.crossref.org is not currently working well - our team is investigating and we'll post more as we learn more.</p>tag:status.crossref.org,2005:Incident/200007012024-02-20T15:47:22Z2024-02-20T15:47:22Z20 February - weekly maintenance, release v0.190.0<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>15:22</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.190.0 - on Tuesday, 20 February from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/199696682024-02-13T17:00:56Z2024-02-13T17:00:56Z13 February - weekly maintenance, release v0.189.0<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:51</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.189.0 - on Tuesday, 13 February from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/192529272024-02-08T21:25:20Z2024-02-08T21:25:20ZSmall portion of metadata records not being indexed by REST API<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>21:25</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:06</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jan <var data-var='date'>12</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Update</strong> - We are continuing to work on this issue. Because we're taking caution to not overwhelm our system with indexing this portion of previously unindexed records, the indexing is taking longer than anticipated. We continue to index these records, but it will take us additional time to get through them all.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>03:58</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>13:58</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>21:12</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>20:59</var> UTC</small><br><strong>Investigating</strong> - Our team has identified a bug that is intermittently preventing REST API metadata records from being indexed. We estimate that 1-2% of records, including new and updated records, may be impacted by this bug. We are currently working on a fix.</p>tag:status.crossref.org,2005:Incident/199131472024-02-06T17:00:56Z2024-02-06T17:00:56Z6 February - weekly maintenance, release v0.188.0<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Update</strong> - The scheduled maintenance is still in progress. We may need an additional 30 minutes today to get the release completed; we should have the work completed by 1730 UTC.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>15:08</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.188.0 - on Tuesday, 6 February from 1500 to 1730 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/198857522024-02-01T16:46:45Z2024-02-01T16:46:45Z1 February - unplanned maintenance, addressing a potential vulnerability<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>15:46</var> UTC</small><br><strong>Investigating</strong> - We will be undergoing unplanned maintenance, today, 1 February, as soon as possible to address a recently discovered vulnerability. Our members may experience delays with our Grant Registration Form, Key Manager UI, and Plus REST API access.</p>tag:status.crossref.org,2005:Incident/198592902024-01-30T17:00:31Z2024-01-30T17:00:31Z30 January - weekly maintenance, release v0.187.0<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>16:22</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.187.0 - on Tuesday, 30 January from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/198363662024-01-26T07:30:00Z2024-01-26T09:31:47ZWebsite unavailable ~ an hour and a half<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>07:30</var> UTC</small><br><strong>Resolved</strong> - Our website (which includes https://search.crossref.org/) was unavailable from 0740 - 0910 UTC. This was due to a known error with our load balancer that our team is investigating. Sorry for the inconvenience!</p>tag:status.crossref.org,2005:Incident/197962572024-01-23T16:00:40Z2024-01-23T16:00:40Z23 January - weekly maintenance, release v0.186.0<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.186.0 - on Tuesday, 23 January from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/197800442024-01-22T16:56:31Z2024-01-22T16:56:31ZNetwork instability earlier today<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>01:15</var> UTC</small><br><strong>Monitoring</strong> - We had some network issues earlier today that caused instability in a number of our products and services. Everything is stable now; we're continuing to monitor.</p>tag:status.crossref.org,2005:Incident/197508112024-01-18T17:20:15Z2024-01-18T17:20:15ZIntermittent service disruptions.<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>12:27</var> UTC</small><br><strong>Update</strong> - We have completed the rollback deployment and the submission queue is back up and processing deposits. We will be monitor systems and looking to resolve any remaining issues.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:58</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:58</var> UTC</small><br><strong>Update</strong> - We will be undergoing an unscheduled maintenance rollback on Thursday, from 1100 to 1300 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:47</var> UTC</small><br><strong>Investigating</strong> - We're looking into balancing some traffic-related issues causing intermittent disruption of these tools for the last 5+ hours. This might also affect other services like the simple text query form.<br /><br />We will keep this updated as we know more.</p>tag:status.crossref.org,2005:Incident/197256122024-01-17T17:01:46Z2024-01-17T17:01:46Z17 January - weekly maintenance, release v0.185.0<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.185.0 - on Wednesday, 17 January from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/197109442024-01-15T10:30:00Z2024-01-15T13:29:17ZShort outage Sunday 14th and Monday 15th January.<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>10:30</var> UTC</small><br><strong>Resolved</strong> - On Sunday 14th January 00:01 UTC and Monday 15th January at 6:00 UTC, we had two outages that lasted approximately 7 minutes. We might not have been able to accept new deposits during those times. Any deposits that were already in the queue were not affected.<br />We are investigating what caused these outages.</p>tag:status.crossref.org,2005:Incident/196480752024-01-09T17:00:36Z2024-01-09T17:00:36Z9 January - weekly maintenance, release v0.184.0<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.184.0 - on Tuesday, 9 January from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>tag:status.crossref.org,2005:Incident/196578632024-01-09T08:00:00Z2024-01-09T15:36:05ZCrossref Metadata Search was down yesterday and earlier today<p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Resolved</strong> - Crossref Metadata Search - search.crossref.org - was down yesterday from 2300 UTC until 0800 UTC today, 9 January, due to a DNS caching error, the same error that caused an outage to our main website on 2023 December 20. The error was fixed by our technical team earlier today.</p>tag:status.crossref.org,2005:Incident/194853912023-12-20T08:47:30Z2023-12-20T08:47:30ZMain website is down<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>08:47</var> UTC</small><br><strong>Resolved</strong> - This problem is resolved and our website is available again. This was caused by a problem with our haproxy server. The team is investigating the cause of the problem to ensure it doesn’t happen again.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>02:45</var> UTC</small><br><strong>Investigating</strong> - Our main website has been down since 02:05 UTC. Our team is a bit thin at the moment given the holidays, but we're investigating.</p>tag:status.crossref.org,2005:Incident/194024032023-12-12T17:00:53Z2023-12-12T17:00:53Z12 December - weekly maintenance, release v0.183.0<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance - release v0.183.0 - on Tuesday, 12 December from 1500 to 1700 UTC. Our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the maintenance has been completed.</p>