User:Badmachine/wikimedia-operations-2013-07-31
MyWikiBiz, Author Your Legacy — Monday December 30, 2024
Jump to navigationJump to search[00:18:31] <grrrit-wm> (�03PS2�) �10�Dzahn��: RT #5464 - apply etherpad-lite live hack fix by apergos [operations/debs/etherpad-lite] - �10https://gerrit.wikimedia.org/r/76661�
[00:22:38] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:23:28] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[00:36:38] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:37:28] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time
[00:58:00] <icinga-wm> PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours
[01:07:23] <springle> !log restarting db43 slave threads
[01:07:37] <morebots> Logged the message, Master
[01:08:27] <springle> !log restarting db45 slave threads
[01:08:37] <morebots> Logged the message, Master
[01:43:06] <grrrit-wm> (�03PS1�) �10�Yurik��: Do not mark HTTPS requests with X-CS header [operations/puppet] - �10https://gerrit.wikimedia.org/r/76873�
[01:43:44] <yurik> paravoid, there is a problem with mark's patch - see ^
[01:53:45] <icinga-wm> PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours
[02:15:57] <logmsgbot> !log LocalisationUpdate completed (1.22wmf12) at Wed Jul 31 02:15:56 UTC 2013
[02:16:08] <morebots> Logged the message, Master
[02:21:59] <fr3djuicy> justice
[02:22:01] <fr3djuicy> on wikipedia
[02:22:02] <fr3djuicy> must be served
[02:22:03] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:03] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:03] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:04] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:04] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:04] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:05] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:05] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte legoktm LeslieCarr logmsgbot
[02:22:05] <fr3djuicy> Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur Vito vvv wm-bot woosters yurik Yu
[02:22:06] <fr3djuicy> viPanda_zz Zidonuke
[02:22:06] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:07] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:07] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:08] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:15] * awjr punches fr3djuicy in the face�
[02:22:15] <Krenair> !ops
[02:22:25] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte legoktm LeslieCarr logmsgbot
[02:22:25] <fr3djuicy> Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur Vito vvv wm-bot woosters yurik Yu
[02:22:25] <fr3djuicy> viPanda_zz Zidonuke
[02:22:26] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:26] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:26] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:26] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:26] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:27] <fr3djuicy> justice
[02:22:28] <fr3djuicy> justice
[02:22:29] <fr3djuicy> justice
[02:22:29] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte legoktm LeslieCarr logmsgbot
[02:22:29] <fr3djuicy> Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur Vito vvv wm-bot woosters yurik Yu
[02:22:30] <fr3djuicy> viPanda_zz Zidonuke
[02:22:41] <luke1_> !ops
[02:22:41] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte legoktm LeslieCarr logmsgbot
[02:22:42] <fr3djuicy> luke1_ Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur Vito vvv wm-bot woosters y
[02:22:42] <fr3djuicy> urik YuviPanda_zz Zidonuke
[02:22:42] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte legoktm LeslieCarr logmsgbot
[02:22:42] <fr3djuicy> luke1_ Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur Vito vvv wm-bot woosters y
[02:22:42] <fr3djuicy> urik YuviPanda_zz Zidonuke
[02:22:43] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:43] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:44] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:44] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:44] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:45] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:46] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:46] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:58] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:59] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:59] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:59] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:22:59] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:01] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:01] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:02] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:02] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:04] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack bburhans boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte ldunn legoktm Leslie
[02:23:05] <fr3djuicy> Carr logmsgbot luke1_ Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Th0masR0ss Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur
[02:23:05] <fr3djuicy> Vito vvv wm-bot woosters yurik YuviPanda_zz Zidonuke
[02:23:05] <Jyothis> oh boy
[02:23:05] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack bburhans boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte ldunn legoktm Leslie
[02:23:05] <fr3djuicy> Carr logmsgbot luke1_ Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Th0masR0ss Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur
[02:23:06] <fr3djuicy> Vito vvv wm-bot woosters yurik YuviPanda_zz Zidonuke
[02:23:06] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:07] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:07] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:08] <fr3djuicy> justice
[02:23:09] <fr3djuicy> justice
[02:23:12] <luke1_> fr3djuicy:STOP!!!!!!!!!!1
[02:23:21] <fr3djuicy> AaronSchulz addshore AFBorchert andrewbogott_afk apergos aude awjr AzaToth basile bblack bburhans boklm Bsadowski1 ccook ChanServ closedmouth cndiv_ Coren csteipp_afk Danny_B dbbot-wm Dereckson dtm ecmabot-wm Elsie fr3djuicy godog greg-g grrrit-wm gwicke_away hexmode icinga-wm Isarra James_F|Away Jamesofur jarry1250 JasonDC Jasper_Deng_away jeblair jeremyb jorm Jyothis jzerebec1i j^ kma500 Krenair Krinkle Laaknor Lcawte ldunn legoktm Leslie
[02:23:21] <fr3djuicy> Carr logmsgbot luke1_ Lydia_WMDE manybubbles|away mark marktraceur matanya mavhc mdale1 michi_cc morebots MustaidGhanta mutante mwalker Nemo_bis Nikerabbit Nischayn22|Away notpeter ori-l paravoid paulproteus petan pgehres|away phuzion PinkAmpersand ragesoss RD Reedy relrod retsreklawts RoanKattouw_away RobH roblaAWAY Ryan_Lane se4598_2 siebrand Snaps Snowolf springle Th0masR0ss Thehelpfulone TimStarling ToAruShiroiNeko twkozlowski Venusaur
[02:23:21] <fr3djuicy> Vito vvv wm-bot woosters yurik YuviPanda_zz Zidonuke
[02:23:24] <jorm> no one cares.
[02:23:25] <Th0masR0ss> !ops not sure if ASM is here
[02:23:32] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:33] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:34] <luke1_> !ops
[02:23:34] <fr3djuicy> https://encyclopediadramatica.se/Ironholds#Wikipedia_vandalism
[02:23:39] <luke1_> vandalism
[02:23:44] <Ryan_Lane> ops is alreadyhere...
[02:23:49] <Th0masR0ss> ASM isn't here I guess.
[02:24:08] <Ryan_Lane> luke1_: yelling stop at trolls never helps
[02:24:12] <Ryan_Lane> ignore them
[02:24:17] <luke1_> ok
[02:24:20] <Th0masR0ss> Doesn't really matter cause they pinged the entire nick list lol
[02:24:27] <Jamesofur> though I appreciate the sentiment :)
[02:25:00] <jorm> wikipediocracy idiots.
[02:25:48] <Jamesofur> jorm: at least they could come up with something new
[02:26:06] <yurik> could this be some form of a browser attack? Like that URL exploiting some new or maybe even 0-day browser vulnerability?
[02:26:07] <Jamesofur> if I had NEW dirt on Oliver it would be much more palatable :P
[02:26:12] <jorm> no.
[02:26:16] <jorm> people trying to get oliver fired.
[02:26:42] <jorm> now he's spamming me via /msg
[02:26:58] <Krenair> yurik, no, this is some idiot flooding IRC
[02:27:30] <jorm> fr3djuicy has left IRC (K-Lined)
[02:28:08] <AzaToth> jorm: he was smart to enter #freenode to spam
[02:28:31] <mwalker> hah; seriously?
[02:28:35] <jorm> i was about to report him in there.
[02:28:36] <AzaToth> yes
[02:28:57] <mwalker> I'm not sure if I know a faster way to get kickbanned
[02:29:16] <AzaToth> two lines, then he got autoklined I think
[02:29:48] <logmsgbot> !log LocalisationUpdate completed (1.22wmf11) at Wed Jul 31 02:29:48 UTC 2013
[02:30:00] <morebots> Logged the message, Master
[02:31:44] <jeremyb> bburhans: help in #wikimedia-commons ?
[02:31:57] <jeremyb> ah, someone got it
[02:46:49] <logmsgbot> !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 31 02:46:48 UTC 2013
[02:47:01] <morebots> Logged the message, Master
[03:04:52] <icinga-wm> PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours
[03:30:59] <icinga-wm> PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours
[03:36:00] <icinga-wm> PROBLEM - Puppet freshness on mw1018 is CRITICAL: No successful Puppet run in the last 10 hours
[03:36:00] <icinga-wm> PROBLEM - Puppet freshness on mw1095 is CRITICAL: No successful Puppet run in the last 10 hours
[03:36:00] <icinga-wm> PROBLEM - Puppet freshness on srv238 is CRITICAL: No successful Puppet run in the last 10 hours
[03:36:00] <icinga-wm> PROBLEM - Puppet freshness on mw1103 is CRITICAL: No successful Puppet run in the last 10 hours
[03:37:59] <icinga-wm> PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours
[03:37:59] <icinga-wm> PROBLEM - Puppet freshness on cp1069 is CRITICAL: No successful Puppet run in the last 10 hours
[03:37:59] <icinga-wm> PROBLEM - Puppet freshness on pc1002 is CRITICAL: No successful Puppet run in the last 10 hours
[03:37:59] <icinga-wm> PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours
[03:39:08] <icinga-wm> PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours
[03:39:08] <icinga-wm> PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours
[03:39:08] <icinga-wm> PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 10 hours
[03:39:08] <icinga-wm> PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours
[03:39:08] <icinga-wm> PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours
[03:41:08] <icinga-wm> PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours
[03:41:08] <icinga-wm> PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours
[03:41:08] <icinga-wm> PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours
[03:41:08] <icinga-wm> PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours
[03:41:08] <icinga-wm> PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:08] <icinga-wm> PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:08] <icinga-wm> PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:08] <icinga-wm> PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:08] <icinga-wm> PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:09] <icinga-wm> PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:09] <icinga-wm> PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:10] <icinga-wm> PROBLEM - Puppet freshness on db63 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:10] <icinga-wm> PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:11] <icinga-wm> PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:11] <icinga-wm> PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:12] <icinga-wm> PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:12] <icinga-wm> PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:13] <icinga-wm> PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:13] <icinga-wm> PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:14] <icinga-wm> PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:14] <icinga-wm> PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:15] <icinga-wm> PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:15] <icinga-wm> PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours
[03:42:16] <icinga-wm> PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours
[03:43:08] <icinga-wm> PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours
[03:43:08] <icinga-wm> PROBLEM - Puppet freshness on db1040 is CRITICAL: No successful Puppet run in the last 10 hours
[03:43:08] <icinga-wm> PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours
[03:43:08] <icinga-wm> PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours
[03:43:08] <icinga-wm> PROBLEM - Puppet freshness on ms-be11 is CRITICAL: No successful Puppet run in the last 10 hours
[03:45:08] <icinga-wm> PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours
[03:45:08] <icinga-wm> PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours
[03:45:08] <icinga-wm> PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours
[03:45:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1003 is CRITICAL: No successful Puppet run in the last 10 hours
[03:45:08] <icinga-wm> PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:08] <icinga-wm> PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:08] <icinga-wm> PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:08] <icinga-wm> PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:08] <icinga-wm> PROBLEM - Puppet freshness on db1026 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:08] <icinga-wm> PROBLEM - Puppet freshness on cp1017 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:09] <icinga-wm> PROBLEM - Puppet freshness on cp1060 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:09] <icinga-wm> PROBLEM - Puppet freshness on db1027 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:10] <icinga-wm> PROBLEM - Puppet freshness on db66 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:10] <icinga-wm> PROBLEM - Puppet freshness on mw1011 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:11] <icinga-wm> PROBLEM - Puppet freshness on mw1021 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:11] <icinga-wm> PROBLEM - Puppet freshness on db9 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:12] <icinga-wm> PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:12] <icinga-wm> PROBLEM - Puppet freshness on es1004 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:13] <icinga-wm> PROBLEM - Puppet freshness on mw1131 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:13] <icinga-wm> PROBLEM - Puppet freshness on mw1154 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:14] <icinga-wm> PROBLEM - Puppet freshness on mw1155 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:14] <icinga-wm> PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:15] <icinga-wm> PROBLEM - Puppet freshness on mw60 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:15] <icinga-wm> PROBLEM - Puppet freshness on mw93 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:16] <icinga-wm> PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:16] <icinga-wm> PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:17] <icinga-wm> PROBLEM - Puppet freshness on sq84 is CRITICAL: No successful Puppet run in the last 10 hours
[03:46:17] <icinga-wm> PROBLEM - Puppet freshness on ssl1003 is CRITICAL: No successful Puppet run in the last 10 hours
[03:47:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours
[03:47:08] <icinga-wm> PROBLEM - Puppet freshness on cp1018 is CRITICAL: No successful Puppet run in the last 10 hours
[03:47:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours
[03:47:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours
[03:47:08] <icinga-wm> PROBLEM - Puppet freshness on cp1062 is CRITICAL: No successful Puppet run in the last 10 hours
[03:49:08] <icinga-wm> PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours
[03:49:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours
[03:49:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours
[03:49:08] <icinga-wm> PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours
[03:49:08] <icinga-wm> PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours
[03:51:08] <icinga-wm> PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours
[03:51:08] <icinga-wm> PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours
[03:51:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:51:08] <icinga-wm> PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours
[03:51:08] <icinga-wm> PROBLEM - Puppet freshness on amssq40 is CRITICAL: No successful Puppet run in the last 10 hours
[03:53:08] <icinga-wm> PROBLEM - Puppet freshness on db1045 is CRITICAL: No successful Puppet run in the last 10 hours
[03:53:08] <icinga-wm> PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours
[03:53:08] <icinga-wm> PROBLEM - Puppet freshness on db73 is CRITICAL: No successful Puppet run in the last 10 hours
[03:53:08] <icinga-wm> PROBLEM - Puppet freshness on db60 is CRITICAL: No successful Puppet run in the last 10 hours
[03:53:08] <icinga-wm> PROBLEM - Puppet freshness on chromium is CRITICAL: No successful Puppet run in the last 10 hours
[03:54:08] <icinga-wm> PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours
[03:54:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours
[03:54:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours
[03:54:08] <icinga-wm> PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:54:08] <icinga-wm> PROBLEM - Puppet freshness on cp1053 is CRITICAL: No successful Puppet run in the last 10 hours
[03:55:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours
[03:55:08] <icinga-wm> PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours
[03:55:08] <icinga-wm> PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours
[03:55:08] <icinga-wm> PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours
[03:55:08] <icinga-wm> PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours
[03:56:08] <icinga-wm> PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours
[03:56:08] <icinga-wm> PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours
[03:56:08] <icinga-wm> PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours
[03:56:08] <icinga-wm> PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours
[03:56:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <icinga-wm> PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <icinga-wm> PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <icinga-wm> PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <icinga-wm> PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:08] <icinga-wm> PROBLEM - Puppet freshness on cp3022 is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:08] <icinga-wm> PROBLEM - Puppet freshness on db1039 is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:08] <icinga-wm> PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours
[03:58:08] <icinga-wm> PROBLEM - Puppet freshness on es1007 is CRITICAL: No successful Puppet run in the last 10 hours
[03:59:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours
[03:59:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours
[03:59:08] <icinga-wm> PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours
[03:59:08] <icinga-wm> PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours
[03:59:08] <icinga-wm> PROBLEM - Puppet freshness on db1007 is CRITICAL: No successful Puppet run in the last 10 hours
[04:00:08] <icinga-wm> PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours
[04:00:08] <icinga-wm> PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours
[04:00:08] <icinga-wm> PROBLEM - Puppet freshness on cp1048 is CRITICAL: No successful Puppet run in the last 10 hours
[04:00:08] <icinga-wm> PROBLEM - Puppet freshness on db1030 is CRITICAL: No successful Puppet run in the last 10 hours
[04:00:08] <icinga-wm> PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours
[04:01:08] <icinga-wm> PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours
[04:01:08] <icinga-wm> PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours
[04:01:08] <icinga-wm> PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours
[04:01:08] <icinga-wm> PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours
[04:01:08] <icinga-wm> PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:08] <icinga-wm> PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:08] <icinga-wm> PROBLEM - Puppet freshness on ms-be6 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:08] <icinga-wm> PROBLEM - Puppet freshness on es1010 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:08] <icinga-wm> PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:08] <icinga-wm> PROBLEM - Puppet freshness on mw1067 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:09] <icinga-wm> PROBLEM - Puppet freshness on mw1016 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:09] <icinga-wm> PROBLEM - Puppet freshness on mw1152 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:10] <icinga-wm> PROBLEM - Puppet freshness on mw32 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:10] <icinga-wm> PROBLEM - Puppet freshness on search1020 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:11] <icinga-wm> PROBLEM - Puppet freshness on mw1028 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:11] <icinga-wm> PROBLEM - Puppet freshness on pc1001 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:12] <icinga-wm> PROBLEM - Puppet freshness on mw1209 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:12] <icinga-wm> PROBLEM - Puppet freshness on mw1005 is CRITICAL: No successful Puppet run in the last 10 hours
[04:02:13] <icinga-wm> PROBLEM - Puppet freshness on mw21 is CRITICAL: No successful Puppet run in the last 10 hours
[04:03:08] <icinga-wm> PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours
[04:03:08] <icinga-wm> PROBLEM - Puppet freshness on amssq45 is CRITICAL: No successful Puppet run in the last 10 hours
[04:03:08] <icinga-wm> PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours
[04:03:08] <icinga-wm> PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours
[04:03:08] <icinga-wm> PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours
[04:05:08] <icinga-wm> PROBLEM - Puppet freshness on db1054 is CRITICAL: No successful Puppet run in the last 10 hours
[04:05:08] <icinga-wm> PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours
[04:05:08] <icinga-wm> PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours
[04:05:08] <icinga-wm> PROBLEM - Puppet freshness on es1 is CRITICAL: No successful Puppet run in the last 10 hours
[04:05:08] <icinga-wm> PROBLEM - Puppet freshness on mw1088 is CRITICAL: No successful Puppet run in the last 10 hours
[04:06:43] <icinga-wm> PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours
[04:06:43] <icinga-wm> PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours
[04:06:43] <icinga-wm> PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours
[04:06:43] <icinga-wm> PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours
[04:06:43] <icinga-wm> PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours
[04:13:43] <icinga-wm> PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours
[04:51:46] <icinga-wm> PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours
[05:29:17] <grrrit-wm> (�03PS1�) �10�Demon��: Remove some unused packages [operations/puppet] - �10https://gerrit.wikimedia.org/r/76884�
[05:29:18] <grrrit-wm> (�03PS1�) �10�Demon��: Remove obsolete backup stuff [operations/puppet] - �10https://gerrit.wikimedia.org/r/76885�
[05:44:28] <grrrit-wm> (�03CR�) �10�Demon��: "So when this is merged, all of the existing svn backups (dunno how long they're retained) can be nuked." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76885� (owner: �10�Demon��)
[06:01:48] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:03:38] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[06:26:59] <icinga-wm> PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours
[07:22:40] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:23:30] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[07:57:08] <hashar> grmblbl
[07:57:11] <hashar> I broke puppet doc
[08:12:42] <grrrit-wm> (�03PS2�) �10�Hashar��: define ishmael_config lacked dollars for paramaeters [operations/puppet] - �10https://gerrit.wikimedia.org/r/76280�
[08:12:48] <grrrit-wm> (�03PS3�) �10�Hashar��: define ishmael_config() parameters lacked dollar signs [operations/puppet] - �10https://gerrit.wikimedia.org/r/76280�
[08:13:40] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:14:30] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[08:18:41] <grrrit-wm> (�03PS2�) �10�Hashar��: cleanup rspec fixtures [operations/puppet] - �10https://gerrit.wikimedia.org/r/76887�
[08:22:40] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:24:30] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.564 second response time
[08:29:40] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:30:30] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.091 second response time
[09:05:11] <mark> so bad token messages
[09:05:20] <mark> that's kinda weird since varnish doesn't do anything with those cookies
[09:05:52] <paravoid> yes
[09:06:42] <mark> do we have any more info?
[09:06:49] <paravoid> a bit, yes
[09:07:20] <paravoid> see query
[09:10:36] <paravoid> unrelated, but also see https://gerrit.wikimedia.org/r/#/c/76873/
[09:10:41] <paravoid> blergh
[09:10:46] <mark> yeah saw that
[09:12:00] <paravoid> where's azatoth
[09:12:03] <paravoid> https://lists.debian.org/debian-devel/2013/07/msg00825.html
[09:12:16] <paravoid> someone ITPed buck, referencing his work at gerrit.wm.org
[09:15:35] <MaxSem> we can ping the reporter
[09:15:41] <MaxSem> addshore, yt?
[09:16:36] <paravoid> morning max
[09:17:19] <MaxSem> hey
[09:17:22] <paravoid> https://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox/
[09:17:25] <mark> so edits are POSTs
[09:17:33] <mark> and get passed by varnish, not cached in any way
[09:17:40] <addshore> ping MaxSem
[09:17:43] <mark> and requests with session or token cookies don't see any cookie header rewriting
[09:17:44] <mark> hi
[09:18:17] <mark> it's weird
[09:18:25] <addshore> weird indeed!
[09:18:42] <mark> addshore: so you were getting "bad token" errors on edits, correct?
[09:19:12] <paravoid> btw, aude mentioned issues as well
[09:19:21] <mark> aude: ping
[09:19:30] <paravoid> :)
[09:19:37] <addshore> mark: yup
[09:19:47] <mark> addshore: those edits were POST requests I presume?
[09:19:51] <addshore> aude seemed to be getting cached / old data from the api
[09:19:56] <addshore> and yes they were post :)
[09:20:08] <mark> weird
[09:20:15] <mark> perhaps some caching header is wrong
[09:20:19] <MaxSem> paravoid, aude's requests were coming through squid
[09:20:21] <addshore> aude had to add a random extra parameter to a Request to get fresh data
[09:20:22] <mark> and squid happens to not cache it for some reason
[09:20:46] <mark> nonfresh data could also be a purge issue
[09:20:58] <mark> but bad token errors are really weird
[09:23:25] <paravoid> we confirmed that it got fixed as soon as I turned it back to squid
[09:30:10] <mark> wikidata is closed for editing by anons?
[09:33:59] <MaxSem> it's not
[09:35:18] <mark> might also be related to the new SUL code
[09:36:48] <MaxSem> I checked his cookies - they had CentralAuth among them
[09:36:53] <mark> yeah
[09:37:12] <mark> addshore: do you have a full url or even better, a full HTTP request that isn't working?
[09:37:52] <mark> maybe the page setting the edit token gets cached somehow
[09:50:01] <mark> or maybe the session/edit token were retrieved via squid and somehow not valid when sent through varnish?
[09:50:04] <mark> i don't see how though :)
[09:55:04] <icinga-wm> PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[09:55:04] <icinga-wm> PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[09:55:04] <icinga-wm> PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[09:55:04] <icinga-wm> PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours
[09:55:04] <icinga-wm> PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[09:55:05] <icinga-wm> PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours
[09:57:27] <MaxSem> mark, I've prepared my own bot in case you need experimenting
[09:57:35] <mark> awesome
[09:57:55] <mark> how does it get its session/edit cookies/tokens?
[09:58:22] <MaxSem> api.php?action=login, saves cookies
[09:58:32] <mark> ok
[10:00:00] <mark> i think we're just going to reenable it in a bit and then experiment
[10:00:26] <mark> this is also relevant to the mobile changes
[10:00:29] <mark> as they're almost the same
[10:02:06] <mark> as far as I can tell, addshore's bot doesn't really automatically retrieve cookies does it
[10:02:16] <mark> it just uses the same cookies always from a static file?
[10:02:29] <mark> that makes it extra weird that it wouldn't work on varnish
[10:02:41] <MaxSem> yeah, but he should be getting them from logging in
[10:03:04] <mark> i don't see how it would work through squid but not varnish
[10:03:53] <grrrit-wm> (�03CR�) �10�Demon��: "I spoke too soon before it was done dumping. Actually about 14G. And in /svnroot/final-backup, due to disk space issues on /" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76885� (owner: �10�Demon��)
[10:04:06] <mark> let's reenable it in 30 mins or so
[10:04:10] <mark> and then see if your bot works
[10:04:16] <mark> ok?
[10:04:22] * MaxSem installs Wireshark�
[10:04:26] <MaxSem> sure
[10:04:28] <mark> that would be very helpful
[10:04:42] <mark> i can try to catch the requests on the servers
[10:05:13] <mark> brb
[10:06:32] <paravoid> we don't have to reenable for everyone, do we now?
[10:06:53] <paravoid> we can just modify maxsem's bot to hit varnish
[10:07:08] <MaxSem> eh, it's AWB
[10:07:36] <MaxSem> and I don't have Visual Studio anymore:P
[10:07:49] <MaxSem> I can set up a different bot FW though:)
[10:08:34] <paravoid> you could also /etc/hosts it :)
[10:10:12] <icinga-wm> PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[10:10:26] <MaxSem> bleh
[10:10:55] <MaxSem> you mean C:\Windows\System32\Drivers\etc\hosts ?:p
[10:11:12] <icinga-wm> RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)
[10:17:43] <mark> paravoid: no, internal
[10:17:54] <mark> i was thinking maybe we should setup a test lvs cluster
[10:17:58] <mark> that all realservers accept
[10:18:30] <aude> woah, being pinged
[10:18:46] <MaxSem> bleh, all my network communications go through L2TP
[10:19:40] <mark> aude: you also had problems that may have been related to varnish yesterday?
[10:19:49] <mark> or was it confirmed to go through squid?
[10:19:56] <MaxSem> mark, she used https
[10:20:04] <aude> MaxSem: i used everythign :)
[10:20:21] <aude> my problem was with api http get requests, getting cached data
[10:20:36] <mark> via vpn?
[10:20:41] <mark> as in, were you hitting esams or eqiad?
[10:20:45] <aude> from labs and from linode (both US)
[10:20:51] <mark> eqiad then
[10:20:53] <aude> yes
[10:21:21] <mark> most likely varnish is working fine in that regard but purging isn't working (well enough?)
[10:21:22] <aude> when i tried https, it worked, tried adding random bogus extra param or xml format then http worked
[10:21:29] <mark> and squid caches far less wikidata as it contends with other wikis
[10:21:35] <mark> but that's just my guess
[10:21:41] <aude> ok
[10:22:28] <aude> the consequence was my bot adding duplicate data
[10:22:36] <aude> since it thought certain properties were missing
[10:22:57] <aude> when wikidata was switched back to squid, my bot operated fine
[10:22:59] <mark> right
[10:23:07] <mark> i do see purges for wikidata
[10:23:19] <aude> i shouldn't have to add bogus params to ensure frech data :)
[10:23:23] <aude> fresh
[10:23:40] <mark> no
[10:24:07] <mark> how can purging even work with the hostname doubled...
[10:24:19] <mark> i only see misses
[10:24:23] <aude> no idea
[10:26:05] <mark> /* Allow purging */
[10:26:05] <mark> call vcl_recv_purge;
[10:26:05] <mark> /* make absolute URIs relative as to not have to rewrite them too;
[10:26:05] <mark> * do this after vcl_recv_purge, as it operates on a full URI */
[10:26:05] <mark> call rewrite_proxy_urls;
[10:26:16] <mark> wasn't it you who added that paravoid?
[10:26:32] <paravoid> I... don't remember?
[10:26:33] <paravoid> :)
[10:26:42] <mark> i don't see how this can work anyway
[10:26:50] <mark> vhtcpd sends both a host header and an absolute uri
[10:26:57] <mark> both get hashed
[10:27:19] <mark> as normal requests don't have absolute uris, this can't work
[10:27:41] <paravoid> what can work?
[10:28:01] <mark> rewriting first?
[10:29:06] * mark looks up the email conversation between you and asher�
[10:30:17] <paravoid> there were "normal" requests coming with absolute URIs
[10:30:44] <paravoid> from carrier or corporate forward proxies or whatever
[10:30:59] <mark> yes
[10:31:01] <mark> so we should continue to rewrite
[10:31:09] <paravoid> right
[10:31:17] <mark> but there's no reason we even need absolute URIs for purging right
[10:31:27] <mark> i think that was done due to the stupid http perl module
[10:31:34] <paravoid> right
[10:31:39] <paravoid> varnishhtcpd didn't set Host
[10:31:41] <paravoid> at all
[10:31:54] <mark> yeah
[10:31:57] <mark> and vhtcpd has an option
[10:32:03] <mark> I think it's time to change that option ;)
[10:32:05] <paravoid> and you had that purge_regex guard
[10:32:09] <mark> -F -- Use full absolute URL in PURGE request
[10:32:11] <mark> yeah
[10:36:40] <paravoid> but why does it break now?
[10:36:57] <mark> because it hashes the host header and then the absolute url
[10:37:04] <paravoid> oh
[10:37:20] <paravoid> how did it work all this time?!
[10:37:49] <mark> i don't think it did
[10:40:13] <grrrit-wm> (�03PS1�) �10�Mark Bergsma��: Don't use absolute URIs for purging [operations/puppet] - �10https://gerrit.wikimedia.org/r/76895�
[10:42:25] <grrrit-wm> (�03CR�) �10�Faidon��: [C: �03�1��] "I guess this will do until the regexp is moved to vhtcpd itself." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76895� (owner: �10�Mark Bergsma��)
[10:42:34] <paravoid> (or the groups split)
[10:44:14] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] Don't use absolute URIs for purging [operations/puppet] - �10https://gerrit.wikimedia.org/r/76895� (owner: �10�Mark Bergsma��)
[10:46:57] <mark> i guess it did work because rewrite_proxy_urls was called
[10:47:09] <mark> but let's clean this mess up
[10:48:59] <mark> yay, much better
[10:51:27] <grrrit-wm> (�03PS1�) �10�Hashar��: rewrite beta autoupdater in python [operations/puppet] - �10https://gerrit.wikimedia.org/r/76897�
[10:51:33] <hashar> morning mark :-]Â Can I have some python script merged in please? https://gerrit.wikimedia.org/r/76897
[10:52:00] <hashar> I had a lame shell script to handle the code update on beta, python makes it a bit nicer :-]
[10:52:09] <mark> where's the backdoor
[10:52:18] <hashar> oh man, I forgot to add one
[10:52:19] <mark> ah it's beta
[10:52:22] <mark> you don't need one there
[10:53:00] <hashar> and parser validation is happy :-]
[10:55:22] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:56:43] <grrrit-wm> (�03PS1�) �10�Petr Onderka��: added README [operations/dumps/incremental] (gsoc) - �10https://gerrit.wikimedia.org/r/76900�
[10:57:02] <grrrit-wm> (�03CR�) �10�Petr Onderka��: [C: �03��2��� V: �03��2���] added README [operations/dumps/incremental] (gsoc) - �10https://gerrit.wikimedia.org/r/76900� (owner: �10�Petr Onderka��)
[10:58:12] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[10:58:32] <icinga-wm> PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours
[11:03:09] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] rewrite beta autoupdater in python [operations/puppet] - �10https://gerrit.wikimedia.org/r/76897� (owner: �10�Hashar��)
[11:03:16] <hashar> \O/
[11:03:30] <hashar> and I found out I am missing an option in git :(
[11:06:42] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:06:56] <mark> aude: MaxSem: shall we put wikidata on varnish again?
[11:07:01] <mark> best way to figure out what's wrong ;)
[11:07:04] <MaxSem> I\m ready
[11:07:17] <aude> alright, my bot is ready to try again
[11:07:26] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time
[11:07:34] <MaxSem> I even have squid cookies;)
[11:07:39] <mark> cool hehe
[11:07:47] <mark> hmm my purging change hasn't applied on all varnish boxes yet
[11:07:51] <mark> maybe should wait for that
[11:07:55] <mark> not sure it helps anything
[11:08:39] <mark> and perhaps I should wipe the caches just to be sure
[11:08:42] <mark> meh
[11:10:41] <mark> done...
[11:13:26] <aude> alright, time to run my bot?
[11:14:17] <mark> it's still on squid
[11:14:22] <aude> ok
[11:14:24] <mark> waiting two more minutes for puppet to run
[11:14:24] * aude waits�
[11:15:09] <MaxSem> mark, I guess I should specify the US IP explicitly?
[11:15:16] <mark> yes
[11:15:21] <mark> europe remains on squid
[11:15:29] * aude is on labs�
[11:15:33] <MaxSem> what's it?
[11:15:36] <mark> labs should be eqiad
[11:15:43] <aude> yep
[11:15:55] <mark> malus-3:~ mark$ host wikidata-lb.eqiad.wikimedia.org
[11:15:55] <mark> wikidata-lb.eqiad.wikimedia.org has address 208.80.154.242
[11:16:22] <mark> ok, added two varnish boxes into the pool...
[11:16:26] <mark> letting cache warm up a bit
[11:16:26] <paravoid> this won't just work though
[11:16:35] <aude> ok
[11:16:39] <paravoid> unless you add it to /etc/hosts, or modify the bot
[11:16:55] <mark> you do need to send the right Host header yes
[11:17:30] <aude> you mean request the ip instead of www.wikidata.org?
[11:17:43] <aude> i am on toollabs so can't exactly change hosts
[11:17:59] <mark> you don't need to
[11:18:01] <aude> ok
[11:18:06] <mark> maxsem does, as he requests from europe
[11:18:12] <aude> ok
[11:18:14] <mark> ok, gonna remove the squids from the pool now
[11:18:26] <MaxSem> changed my hosts
[11:18:55] <mark> so sounds like aude's bot worked yesterday
[11:18:58] <mark> except for nonfresh data
[11:19:02] <mark> so it can't ALL be broken
[11:19:13] <aude> tokens were okay
[11:19:30] <grrrit-wm> (�03PS1�) �10�Hashar��: beta: phase out shell autoupdater [operations/puppet] - �10https://gerrit.wikimedia.org/r/76905�
[11:19:33] <aude> only issue was adding duplicate data
[11:19:48] <mark> alright it's fully on varnish now
[11:19:52] <aude> ok :)
[11:19:55] <mark> aude: i don't think that issue is fixed
[11:20:02] <mark> but let's see what we get
[11:20:05] <aude> hmmm,
[11:20:37] <MaxSem> wee, I got stale data
[11:20:56] <aude> yep its adding duplicate data
[11:21:01] <mark> ok
[11:21:05] <mark> can we test with a single request?
[11:21:08] <mark> as in, see if the purge comes in
[11:21:32] <aude> what do i do?
[11:21:37] <MaxSem> basically, I made an edit, then loaded page content again, it returned the previous state of the page
[11:21:45] <mark> just do a single request, tell me what the url is before you do it
[11:21:53] <aude> http://www.wikidata.org/w/index.php?title=Q2134240&action=history
[11:21:54] <mark> so I can see what happens
[11:22:36] <aude> i'll try another page
[11:22:50] <mark> as long as you tell me upfront
[11:22:58] <mark> so i can adjust my filter
[11:23:26] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:23:47] <aude> https://www.wikidata.org/w/api.php?action=wbgetclaims&entity=q2614969&format=json is empty
[11:24:10] <MaxSem> mark, https://dpaste.de/QjC25/
[11:24:26] <MaxSem> X-Cache: cp1065 hit (1), cp1065 frontend hit (1)
[11:24:59] <mark> yes, now choose another page, tell me first, so I can see if there's a purge coming in
[11:25:11] * aude editing the item�
[11:25:16] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[11:25:44] <mark> are you editing q2614969 now aude?
[11:25:54] <aude> i did
[11:25:57] <aude> now i get '{"claims":[]}',
[11:25:59] <aude> still
[11:26:05] <aude> i can do another page
[11:26:12] <mark> please do, and DONT EDIT until i'm ready
[11:26:15] <aude> ok
[11:26:57] <aude> https://www.wikidata.org/w/api.php?action=wbgetclaims&entity=q2615206&format=json
[11:27:22] <mark> ok
[11:27:23] * aude waiting�
[11:27:24] <mark> now you can edit
[11:27:28] <aude> k
[11:27:45] <aude> doig
[11:27:47] <aude> doing
[11:28:02] <aude> done
[11:28:02] <mark> seeing purges
[11:28:04] <mark> like 10+ of them
[11:28:12] <aude> 12 edits
[11:28:13] <mark> cp1065.eqiad.wmnet 1 2013-07-31T11:27:41 0.000062227 127.0.0.1 -/204 0 PURGE http://www.wikidata.org/wiki/Q2615206 - - - - vhtcpd
[11:28:13] <mark> cp1065.eqiad.wmnet 2 2013-07-31T11:27:41 0.000041962 127.0.0.1 -/204 0 PURGE http://www.wikidata.org/w/index.php?title=Q2615206&action=history - - - - vhtcpd
[11:28:24] <mark> but not for that api url
[11:28:29] <mark> so purges are not sent for that
[11:28:35] <aude> oh
[11:28:38] <mark> and varnish can't know that is not fresh
[11:28:50] <aude> hmmm
[11:29:00] <aude> '{"claims":[]}',
[11:29:09] <mark> yes
[11:29:16] <mark> that's a wikidata bug, not varnish
[11:29:24] <aude> huh?
[11:29:27] <mark> you're probably not seeing it on squid much, as squid simply doesn't cache this much
[11:29:39] <aude> so, what exactly are we doing wrong?
[11:29:53] <mark> wikidata, on edit of an item, needs to send a purge for any API URLs related to that item
[11:30:11] <aude> ok?
[11:30:12] <mark> varnish doesn't know it needs to purge http://www.wikidata.org/w/api.php?action=wbgetclaims&entity=q2615206&format=json
[11:30:14] <aude> header?
[11:30:16] <mark> how would it know?
[11:30:20] <aude> ok
[11:30:28] <mark> so mediawiki sends two purges currently
[11:30:30] <mark> the two i pasted above
[11:30:35] <mark> the normal wiki page, and the history page
[11:30:40] <mark> and those will be up to date probably
[11:30:46] <aude> right
[11:30:52] <mark> but the corresponding api requests don't get purged
[11:31:00] <aude> ok
[11:31:09] <mark> not in squid either
[11:31:11] <aude> i suppose action=edit purges
[11:31:16] <mark> yes
[11:31:23] <aude> i can see what/how it does
[11:32:03] <mark> and i don't know how many different API calls exist for that entity
[11:32:07] <mark> but tis can quickly become a big problem
[11:32:09] <aude> a bunch
[11:32:20] <mark> even reordering of parameters will break it
[11:32:39] <aude> you mean bypass the cache?
[11:32:50] <mark> no, will not purge and thus you'll get stale data
[11:32:59] <aude> oh
[11:33:18] <MaxSem> mark, what about my pastebin? it got cached even with cookies and cache-control: private
[11:33:27] <mark> MaxSem: checking
[11:34:24] <mark> MaxSem: cache-control: private is for clients, not varnish
[11:34:30] <mark> the frontend varnish sends that to the client
[11:34:34] <mark> it's not what varnish itself uses
[11:34:43] <mark> so mediawiki most likely DID NOT send cache-control: private
[11:35:27] <mark> hmm no
[11:35:29] <mark> varnish sets:
[11:35:40] <aude> i see $response->header( 'Cache-Control: private' ); in api main
[11:35:40] <mark> set resp.http.Cache-Control = "private, s-maxage=0, max-age=0, must-revalidate";
[11:35:44] <mark> and also doesn't do that in the api
[11:35:54] <MaxSem> so how the hell it worked with squid?
[11:36:20] <aude> setCacheMode( $mode )
[11:36:30] <mark> right
[11:36:31] <mark> that is weird
[11:36:36] <icinga-wm> PROBLEM - Disk space on wtp1014 is CRITICAL: DISK CRITICAL - free space: / 262 MB (2% inode=78%):
[11:36:41] <mark> ok, i'm going to put traffic back on squid
[11:36:43] <mark> and then investigate that
[11:36:47] <aude> ok
[11:36:52] <mark> that could also explain token problems
[11:37:03] <mark> if mediawiki sends cache-control: private and varnish caches anyway
[11:37:46] <mark> back on squid
[11:37:58] <aude> hmmm "Private caching is now the default"
[11:39:00] <mark> note the case of Cache-control:
[11:39:05] <mark> instead of Cache-Control:
[11:39:08] <mark> i really hope that's not it ;)
[11:39:47] <aude> mark: really? it's case sensitive?
[11:39:52] <mark> i hope not
[11:40:02] <aude> ok
[11:40:07] <mark> it' just something I noticed, i'm gonna look at the varnish code now
[11:40:13] <aude> ok
[11:40:29] <aude> if there's anything particular we need to do for wikibase, please put it in a bug report
[11:40:37] <icinga-wm> PROBLEM - Disk space on wtp1017 is CRITICAL: DISK CRITICAL - free space: / 307 MB (3% inode=78%):
[11:40:41] * aude slightly confused :)�
[11:40:42] <mark> ok
[11:40:46] <mark> thanks for your help!
[11:40:49] <aude> sure
[11:40:50] <MaxSem> looks like it's not WB-specific
[11:41:04] <aude> i can't imagine we're doing anything that special for caching
[11:41:13] <aude> we derive from the core api classes
[11:41:20] <icinga-wm> PROBLEM - Parsoid on wtp1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:41:27] <paravoid> I see nothing about Cache-Control: private in varnish's source
[11:41:42] <paravoid> it does seem to obey s-maxage & maxage though
[11:42:40] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:43:03] <mark> that's what I found too
[11:43:18] <paravoid> nice
[11:43:43] <mark> wth
[11:43:57] <aude> yep, squid sends me fresh data
[11:44:25] <mark> yep, squid respects cache-control: private
[11:44:28] <aude> ok
[11:44:30] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time
[11:44:43] <paravoid> mark: 81006eafd6
[11:44:44] <aude> sounds like the issue, although i really am not qualified
[11:44:49] <aude> to say
[11:44:51] <paravoid> Handle Cache-control: (private|no-cache|no-store) by default
[11:44:51] <paravoid>
[11:44:51] <paravoid> As per discussion at the Varnish dev meeting, seeing Cache-control:
[11:44:54] <paravoid> no-cache (and friends) now disable caching by default. As a twist, we
[11:44:57] <paravoid> respect Surrogate-control if that is present, in the default VCL.
[11:45:00] <paravoid>
[11:45:03] <paravoid> Fixes: 1124
[11:45:05] <paravoid> References: 477
[11:45:08] <paravoid> https://www.varnish-cache.org/trac/ticket/1124 & https://www.varnish-cache.org/trac/ticket/477
[11:45:46] <paravoid> the fix is a default VCL change
[11:45:50] <icinga-wm> PROBLEM - Parsoid on wtp1017 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:45:51] <paravoid> it's in master, not in 3.0.4 though
[11:45:59] <MaxSem> bleh
[11:46:06] <MaxSem> so we need to patch up MW...
[11:46:11] <paravoid> not necessarily
[11:46:14] <mark> patching up MW would be good
[11:46:15] <paravoid> we can just change our VCL
[11:46:26] <paravoid> I think in this case it's a Varnish bug actually
[11:46:35] <mark> it is
[11:46:43] <mark> i'm going to put this in wikimedia.vcl
[11:46:57] <paravoid> well, it's debatable
[11:47:00] <paravoid> phk said it wasn't
[11:47:33] <paravoid> "Varnish is not a cache in the RFC2616 sense" :)
[11:47:46] <paravoid> but it was fixed in varnish nevertheless
[11:47:59] <paravoid> I'm guessing that's why they put it in the default VCL instead of in the code though
[11:49:06] <paravoid> mark: also note Surrogate-Control in that commit, I've mentioned the header in https://bugzilla.wikimedia.org/show_bug.cgi?id=48835
[11:49:18] <mark> i know
[11:49:25] <paravoid> ok :)
[11:50:30] <paravoid> #1124 is a pretty good bug report
[11:50:37] <paravoid> explains the situation and recommends solutions
[11:51:40] <paravoid> brb food
[11:53:50] <icinga-wm> PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours
[11:56:16] <mark> so it would be good to make mediawiki send max-age=0 and s-maxage=0 just in case
[11:56:40] <icinga-wm> RECOVERY - Disk space on wtp1014 is OK: DISK OK
[11:57:12] <grrrit-wm> (�03PS1�) �10�Mark Bergsma��: Add max-age/s-maxage on CC: private/no-cache responses [operations/puppet] - �10https://gerrit.wikimedia.org/r/76906�
[11:58:09] <mark> hmm
[11:58:11] <mark> this won't work
[11:58:13] <mark> ttl is already set then
[11:59:03] <grrrit-wm> (�03PS2�) �10�Mark Bergsma��: Add max-age/s-maxage on CC: private/no-cache responses [operations/puppet] - �10https://gerrit.wikimedia.org/r/76906�
[11:59:21] <grrrit-wm> (�03CR�) �10�Hashar��: "That is apparently breaking the puppet automatic documentation." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76280� (owner: �10�Hashar��)
[11:59:45] <grrrit-wm> (�03PS3�) �10�Hashar��: style for ipv6relay.pp [operations/puppet] - �10https://gerrit.wikimedia.org/r/76285�
[12:00:22] <grrrit-wm> (�03CR�) �10�Jeroen De Dauw��: [C: �03�1��] Add DataTypes extension [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76481� (owner: �10�Aude��)
[12:01:25] <grrrit-wm> (�03PS2�) �10�Hashar��: Revert "try setting some variables in my.cnf for fundraising db" . . . well that was full of cryptic fail. [operations/puppet] - �10https://gerrit.wikimedia.org/r/72157� (owner: �10�Jgreen��)
[12:01:31] <grrrit-wm> (�03CR�) �10�jenkins-bot��: [V: �04�-1��] Revert "try setting some variables in my.cnf for fundraising db" [operations/puppet] - �10https://gerrit.wikimedia.org/r/72157� (owner: �10�Jgreen��)
[12:01:40] <icinga-wm> RECOVERY - Disk space on wtp1017 is OK: DISK OK
[12:02:21] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:02:41] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] Add max-age/s-maxage on CC: private/no-cache responses [operations/puppet] - �10https://gerrit.wikimedia.org/r/76906� (owner: �10�Mark Bergsma��)
[12:03:55] <grrrit-wm> (�03PS5�) �10�TTO��: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76342�
[12:04:10] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[12:07:20] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:08:37] <mark> actually that's wrong isn't it
[12:08:46] <mark> if it's private, max-age could be > 0
[12:09:07] <mark> then again we don't know what that value would be
[12:09:12] <mark> so best to leave it at 0 I guess
[12:10:10] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[12:10:44] <grrrit-wm> (�03PS1�) �10�Mark Bergsma��: Missing brace [operations/puppet] - �10https://gerrit.wikimedia.org/r/76907�
[12:11:26] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] Missing brace [operations/puppet] - �10https://gerrit.wikimedia.org/r/76907� (owner: �10�Mark Bergsma��)
[12:23:24] <MaxSem> will we make another attempt?
[12:23:36] <aude> huh?
[12:23:50] <icinga-wm> PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:24:40] <icinga-wm> RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[12:25:36] <MaxSem> with VCL fixes...
[12:25:40] <grrrit-wm> (�03PS1�) �10�Mark Bergsma��: req.hash_ignore_busy can't be set from vcl_hash for some reason [operations/puppet] - �10https://gerrit.wikimedia.org/r/76908�
[12:26:00] <mark> yes
[12:26:08] <mark> in > 30mins though
[12:26:10] <aude> tell me when to try editing
[12:26:11] <aude> ok
[12:26:11] <mark> this needs to go out first
[12:26:37] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] req.hash_ignore_busy can't be set from vcl_hash for some reason [operations/puppet] - �10https://gerrit.wikimedia.org/r/76908� (owner: �10�Mark Bergsma��)
[12:36:22] <paravoid> AzaToth: ping?
[12:57:41] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:59:17] <MaxSem> oh lol - my PC almost ended up in a swapdeath after I left Wireshark logging and started downloading a torrent:P
[12:59:31] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time
[13:05:23] <mark> purging all varnish boxes again...
[13:05:31] <icinga-wm> PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours
[13:06:43] <aude> ok?
[13:07:16] <mark> going to put traffic on varnish again
[13:07:24] <aude> alright
[13:07:40] <aude> just let me know when to edit :)
[13:10:37] <mark> ok, go ahead
[13:11:04] <mark> cache hit rate is absolutely 0 now
[13:11:10] <mark> so it's not possible for this not to work ;-)
[13:11:18] <MaxSem> X-Cache: cp1065 miss (0), cp1052 frontend miss (0)
[13:11:28] <MaxSem> mark, seems to work:)
[13:11:34] <aude> https://www.wikidata.org/w/api.php?action=wbgetclaims&entity=Q1809285&format=json is the item
[13:11:46] <mark> go ahead, i won't even monitor now ;)
[13:11:52] <aude> alright
[13:12:43] <aude> toollabs seems momentarliy frozen
[13:13:03] <aude> it does that occassionally (maybe puppet is running)
[13:13:23] <aude> it should be responsive again in a minute or 2
[13:13:55] <grrrit-wm> (�03CR�) �10�Akosiaris��: [C: �03��2���] "LGTM. Thanks for spotting this out" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76280� (owner: �10�Hashar��)
[13:14:00] <aude> it's back
[13:14:55] <aude> yay! :)
[13:15:00] <aude> mark: works for me
[13:15:04] <mark> good
[13:15:15] <mark> now I wonder if addshore's problem is fixed too
[13:15:17] <mark> addshore: ping
[13:15:27] <aude> i'm poking him
[13:15:57] <addshore> well, its still editing :)
[13:16:02] <aude> good!
[13:16:07] <mark> nice
[13:16:20] <addshore> and aude toollabs freezing is an nfs issue :<
[13:16:25] <aude> oh, ok
[13:16:48] <MaxSem> what, NFS is still broken?
[13:17:38] * aude shall add coordinates for all the nyc subway stations :)�
[13:18:54] <mark> hit rate average: 0.0003
[13:19:18] <mark> there's absolutely 0 point in having varnish in front of wikidata atm ;)
[13:19:36] <aude> huh
[13:19:37] <aude> hah
[13:19:56] <aude> all those people coming to read wikidata.....
[13:20:27] <mark> it's just bots, editing via the api, which doesn't allow caching
[13:20:36] <aude> pretty much at this point
[13:20:49] <mark> but it's good that we did this
[13:20:50] <paravoid> didn't we disable caching for everyone a while back because of issues with multiple languages?
[13:21:04] <paravoid> the interface being internationalized via a cookie or something
[13:21:09] <aude> paravoid: no
[13:21:12] <aude> i don't think so
[13:21:19] <aude> we disabled uls lang switching for anons
[13:21:40] <aude> something that would be nice to solve someday :)
[13:23:13] <MaxSem> okay, WD.o works, what's next?:P
[13:23:42] <mark> wikivoyage ;-)
[13:23:56] <MaxSem> RIP
[13:30:55] <AzaToth> paravoid: pong
[13:31:27] <mark> MaxSem: let's deploy that mobile cookies change
[13:31:47] <icinga-wm> PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours
[13:31:58] <MaxSem> ok, I'm ready to test
[13:32:21] <grrrit-wm> (�03PS9�) �10�Mark Bergsma��: Mobile Cookie Vary caching optimizations [operations/puppet] - �10https://gerrit.wikimedia.org/r/75316�
[13:32:28] <mark> just a rebase
[13:32:37] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:33:27] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[13:33:47] <mark> hmm
[13:33:51] <mark> let's test on beta
[13:33:56] <hashar> \O/
[13:35:14] <mark> deployed on deployment-staging-cache-mobile01
[13:35:20] <paravoid> AzaToth: https://lists.debian.org/debian-devel/2013/07/msg00825.html
[13:35:41] <MaxSem> isthere a *02?
[13:36:47] <icinga-wm> PROBLEM - Puppet freshness on mw1018 is CRITICAL: No successful Puppet run in the last 10 hours
[13:36:47] <icinga-wm> PROBLEM - Puppet freshness on mw1095 is CRITICAL: No successful Puppet run in the last 10 hours
[13:36:47] <icinga-wm> PROBLEM - Puppet freshness on mw1103 is CRITICAL: No successful Puppet run in the last 10 hours
[13:36:47] <icinga-wm> PROBLEM - Puppet freshness on srv238 is CRITICAL: No successful Puppet run in the last 10 hours
[13:37:11] <mark> no
[13:37:12] <mark> 13 TxHeader b X-Orig-Cookie: optin=1; blabla=0;
[13:37:12] <mark> 13 TxHeader b Cookie: optin=1
[13:38:10] <mark> 13 TxHeader b Cookie: optin=1; blabla=0; enwiki_Session=01234
[13:38:20] <MaxSem> X-Cache: deployment-cache-mobile01 miss (0), deployment-cache-mobile01 frontend miss (0)
[13:38:47] <icinga-wm> PROBLEM - Puppet freshness on cp1069 is CRITICAL: No successful Puppet run in the last 10 hours
[13:38:47] <icinga-wm> PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours
[13:38:47] <icinga-wm> PROBLEM - Puppet freshness on pc1002 is CRITICAL: No successful Puppet run in the last 10 hours
[13:38:47] <icinga-wm> PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours
[13:39:15] <mark> if we split that regsuball into multiple regsubs
[13:39:21] <mark> that has the added advantage that the cookies get sorted
[13:39:29] <mark> or, well, get a consistent order
[13:40:03] <icinga-wm> PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours
[13:40:03] <icinga-wm> PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours
[13:40:03] <icinga-wm> PROBLEM - Puppet freshness on cp1006 is CRITICAL: No successful Puppet run in the last 10 hours
[13:40:03] <icinga-wm> PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours
[13:40:03] <icinga-wm> PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours
[13:40:40] <AzaToth> paravoid: good luck :-P
[13:40:57] <paravoid> it's not me
[13:41:05] <paravoid> it's some random person who I've never talked with
[13:42:03] <icinga-wm> PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours
[13:42:03] <icinga-wm> PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours
[13:42:03] <icinga-wm> PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours
[13:42:03] <icinga-wm> PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours
[13:42:03] <icinga-wm> PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:03] <icinga-wm> PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:03] <icinga-wm> PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:03] <icinga-wm> PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:03] <icinga-wm> PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:04] <icinga-wm> PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:04] <icinga-wm> PROBLEM - Puppet freshness on db63 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:05] <icinga-wm> PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:05] <icinga-wm> PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:06] <icinga-wm> PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:06] <icinga-wm> PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:07] <icinga-wm> PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:07] <icinga-wm> PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:08] <icinga-wm> PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:08] <icinga-wm> PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:09] <icinga-wm> PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:09] <icinga-wm> PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:10] <icinga-wm> PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:10] <icinga-wm> PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 10 hours
[13:43:11] <icinga-wm> PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours
[13:44:03] <icinga-wm> PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours
[13:44:03] <icinga-wm> PROBLEM - Puppet freshness on db1040 is CRITICAL: No successful Puppet run in the last 10 hours
[13:44:03] <icinga-wm> PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours
[13:44:03] <icinga-wm> PROBLEM - Puppet freshness on ms-be11 is CRITICAL: No successful Puppet run in the last 10 hours
[13:44:03] <icinga-wm> PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:45:47] <MaxSem> Error: 503, Service Unavailable
[13:46:02] <MaxSem> labs NFS again?
[13:46:03] <icinga-wm> PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours
[13:46:03] <icinga-wm> PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours
[13:46:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:46:03] <icinga-wm> PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours
[13:46:03] <icinga-wm> PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on cp1017 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on cp1060 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:03] <icinga-wm> PROBLEM - Puppet freshness on db1026 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:04] <icinga-wm> PROBLEM - Puppet freshness on db1027 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:04] <icinga-wm> PROBLEM - Puppet freshness on db66 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:04] <icinga-wm> PROBLEM - Puppet freshness on db9 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:05] <icinga-wm> PROBLEM - Puppet freshness on es1004 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:06] <icinga-wm> PROBLEM - Puppet freshness on mw1011 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:06] <icinga-wm> PROBLEM - Puppet freshness on mw1021 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:07] <icinga-wm> PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:07] <icinga-wm> PROBLEM - Puppet freshness on mw1131 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:08] <icinga-wm> PROBLEM - Puppet freshness on mw1154 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:08] <icinga-wm> PROBLEM - Puppet freshness on mw1155 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:09] <icinga-wm> PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:09] <icinga-wm> PROBLEM - Puppet freshness on mw60 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:10] <icinga-wm> PROBLEM - Puppet freshness on mw93 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:10] <icinga-wm> PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:11] <icinga-wm> PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:11] <icinga-wm> PROBLEM - Puppet freshness on sq84 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:12] <icinga-wm> PROBLEM - Puppet freshness on ssl1003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:58] <MaxSem> mark, is that 503 related to the VCL change?
[13:48:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours
[13:48:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours
[13:48:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours
[13:48:03] <icinga-wm> PROBLEM - Puppet freshness on cp1016 is CRITICAL: No successful Puppet run in the last 10 hours
[13:48:03] <icinga-wm> PROBLEM - Puppet freshness on cp1018 is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:03] <icinga-wm> PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:03] <icinga-wm> PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:03] <icinga-wm> PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours
[13:50:18] <hashar> one day all those server could use a puppet fix :D
[13:51:03] <icinga-wm> PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:03] <icinga-wm> PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:03] <icinga-wm> PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:03] <icinga-wm> PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:04] <icinga-wm> PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:04] <icinga-wm> PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:05] <icinga-wm> PROBLEM - Puppet freshness on mw1083 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:05] <icinga-wm> PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:06] <icinga-wm> PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:06] <icinga-wm> PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:07] <icinga-wm> PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:07] <icinga-wm> PROBLEM - Puppet freshness on mw134 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:08] <icinga-wm> PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:08] <icinga-wm> PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:09] <icinga-wm> PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:09] <icinga-wm> PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:10] <icinga-wm> PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:10] <icinga-wm> PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:11] <icinga-wm> PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:11] <icinga-wm> PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours
[13:51:12] <icinga-wm> PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours
[13:52:03] <icinga-wm> PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours
[13:52:03] <icinga-wm> PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours
[13:52:03] <icinga-wm> PROBLEM - Puppet freshness on amssq40 is CRITICAL: No successful Puppet run in the last 10 hours
[13:52:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:52:03] <icinga-wm> PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours
[13:54:03] <icinga-wm> PROBLEM - Puppet freshness on chromium is CRITICAL: No successful Puppet run in the last 10 hours
[13:54:03] <icinga-wm> PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours
[13:54:03] <icinga-wm> PROBLEM - Puppet freshness on db60 is CRITICAL: No successful Puppet run in the last 10 hours
[13:54:03] <icinga-wm> PROBLEM - Puppet freshness on db1045 is CRITICAL: No successful Puppet run in the last 10 hours
[13:54:03] <icinga-wm> PROBLEM - Puppet freshness on linne is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:02] <AzaToth> paravoid: I see
[13:55:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:03] <icinga-wm> PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:03] <icinga-wm> PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours
[13:55:10] <AzaToth> cmjohnson1: puppet again?
[13:55:14] <grrrit-wm> (�03CR�) �10�Akosiaris��: [C: �03��2���] "LGTM" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76887� (owner: �10�Hashar��)
[13:56:00] <cmjohnson1> AzaToth: looks that way
[13:56:04] <icinga-wm> PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours
[13:56:04] <icinga-wm> PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours
[13:56:04] <icinga-wm> PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours
[13:56:04] <icinga-wm> PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours
[13:56:04] <icinga-wm> PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours
[13:57:05] <cmjohnson1> not sure if it's puppet or icinga
[13:57:45] <hashar> on mw60 puppet ran a few minutes ago
[13:58:03] <icinga-wm> PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours
[13:58:03] <icinga-wm> PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours
[13:58:03] <icinga-wm> PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours
[13:58:03] <icinga-wm> PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours
[13:58:03] <icinga-wm> PROBLEM - Puppet freshness on hooper is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:03] <icinga-wm> PROBLEM - Puppet freshness on cp3022 is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:03] <icinga-wm> PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:03] <icinga-wm> PROBLEM - Puppet freshness on db1039 is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:03] <icinga-wm> PROBLEM - Puppet freshness on db37 is CRITICAL: No successful Puppet run in the last 10 hours
[13:59:28] <hashar> mw60: notice: /Stage[main]/Base::Puppet/Exec[neon puppet snmp trap]/returns: executed successfully
[13:59:29] <hashar>
[13:59:42] <hashar> maybe snmptt on icinga is dead :D
[14:00:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours
[14:00:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours
[14:00:03] <icinga-wm> PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours
[14:00:03] <icinga-wm> PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours
[14:00:03] <icinga-wm> PROBLEM - Puppet freshness on db1007 is CRITICAL: No successful Puppet run in the last 10 hours
[14:01:03] <icinga-wm> PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours
[14:01:03] <icinga-wm> PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours
[14:01:03] <icinga-wm> PROBLEM - Puppet freshness on cp1048 is CRITICAL: No successful Puppet run in the last 10 hours
[14:01:03] <icinga-wm> PROBLEM - Puppet freshness on db1030 is CRITICAL: No successful Puppet run in the last 10 hours
[14:01:03] <icinga-wm> PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:03] <icinga-wm> PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:04] <icinga-wm> PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:04] <icinga-wm> PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:04] <icinga-wm> PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:04] <icinga-wm> PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:04] <icinga-wm> PROBLEM - Puppet freshness on db1038 is CRITICAL: No successful Puppet run in the last 10 hours
[14:02:38] <akosiaris> Warning: The results of service 'Puppet freshness' on host 'mw1163' are stale by 0d 0h 0m 55s (threshold=0d 10h 0m 0s). I'm forcing an immediate check of the service.
[14:02:45] <akosiaris> huh ?
[14:03:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:03] <icinga-wm> PROBLEM - Puppet freshness on es1010 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:03] <icinga-wm> PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:03] <icinga-wm> PROBLEM - Puppet freshness on mw1005 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:03] <icinga-wm> PROBLEM - Puppet freshness on mw1028 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:04] <icinga-wm> PROBLEM - Puppet freshness on mw1067 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:04] <icinga-wm> PROBLEM - Puppet freshness on ms-be6 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:05] <icinga-wm> PROBLEM - Puppet freshness on mw1152 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:05] <icinga-wm> PROBLEM - Puppet freshness on mw1016 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:06] <icinga-wm> PROBLEM - Puppet freshness on mw21 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:06] <icinga-wm> PROBLEM - Puppet freshness on pc1001 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:07] <icinga-wm> PROBLEM - Puppet freshness on mw32 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:07] <icinga-wm> PROBLEM - Puppet freshness on search1020 is CRITICAL: No successful Puppet run in the last 10 hours
[14:03:07] <akosiaris> 55s last time i checked is definetely less than 10 hours...
[14:03:08] <icinga-wm> PROBLEM - Puppet freshness on mw1209 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:03] <icinga-wm> PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:03] <icinga-wm> PROBLEM - Puppet freshness on amssq45 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:03] <icinga-wm> PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:03] <icinga-wm> PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:03] <icinga-wm> PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours
[14:04:21] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �04��-2���] "This preserves only one cookie" [operations/puppet] - �10https://gerrit.wikimedia.org/r/75316� (owner: �10�Mark Bergsma��)
[14:06:03] <icinga-wm> PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours
[14:06:03] <icinga-wm> PROBLEM - Puppet freshness on db1054 is CRITICAL: No successful Puppet run in the last 10 hours
[14:06:03] <icinga-wm> PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours
[14:06:03] <icinga-wm> PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours
[14:06:03] <icinga-wm> PROBLEM - Puppet freshness on labsdb1002 is CRITICAL: No successful Puppet run in the last 10 hours
[14:07:53] <hashar> akosiaris: you can have a look at the snmptt receiver on icinga (neon.wikimedia.org)
[14:07:59] <hashar> maybe it is dead / no more processing traps
[14:09:06] <akosiaris> hashar: i am already looking into it
[14:09:41] <icinga-wm> RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Wed Jul 31 14:09:35 UTC 2013
[14:09:51] <icinga-wm> RECOVERY - Puppet freshness on mw76 is OK: puppet ran at Wed Jul 31 14:09:40 UTC 2013
[14:09:51] <icinga-wm> RECOVERY - Puppet freshness on mw1146 is OK: puppet ran at Wed Jul 31 14:09:41 UTC 2013
[14:09:51] <icinga-wm> RECOVERY - Puppet freshness on mw1038 is OK: puppet ran at Wed Jul 31 14:09:46 UTC 2013
[14:09:51] <icinga-wm> RECOVERY - Puppet freshness on ms-be1012 is OK: puppet ran at Wed Jul 31 14:09:46 UTC 2013
[14:09:52] <icinga-wm> RECOVERY - Puppet freshness on mw1056 is OK: puppet ran at Wed Jul 31 14:09:46 UTC 2013
[14:09:52] <icinga-wm> RECOVERY - Puppet freshness on mw1057 is OK: puppet ran at Wed Jul 31 14:09:46 UTC 2013
[14:09:57] <hashar> spamm :-]
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Wed Jul 31 14:09:51 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Wed Jul 31 14:09:51 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on search19 is OK: puppet ran at Wed Jul 31 14:09:51 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on db1004 is OK: puppet ran at Wed Jul 31 14:09:51 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on mw1159 is OK: puppet ran at Wed Jul 31 14:09:51 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on sq43 is OK: puppet ran at Wed Jul 31 14:09:56 UTC 2013
[14:10:01] <icinga-wm> RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Wed Jul 31 14:09:56 UTC 2013
[14:10:02] <icinga-wm> RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:02] <icinga-wm> RECOVERY - Puppet freshness on ms-be9 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:03] <icinga-wm> RECOVERY - Puppet freshness on sq85 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:03] <icinga-wm> RECOVERY - Puppet freshness on sq42 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:04] <icinga-wm> RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:04] <icinga-wm> RECOVERY - Puppet freshness on mc1001 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:05] <icinga-wm> RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:05] <icinga-wm> RECOVERY - Puppet freshness on cp1013 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:06] <icinga-wm> RECOVERY - Puppet freshness on db58 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:06] <icinga-wm> RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:07] <icinga-wm> RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:07] <icinga-wm> RECOVERY - Puppet freshness on manganese is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:08] <icinga-wm> RECOVERY - Puppet freshness on sq48 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:08] <icinga-wm> RECOVERY - Puppet freshness on mw8 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:09] <icinga-wm> RECOVERY - Puppet freshness on cp1049 is OK: puppet ran at Wed Jul 31 14:09:57 UTC 2013
[14:10:11] <icinga-wm> RECOVERY - Puppet freshness on search18 is OK: puppet ran at Wed Jul 31 14:10:02 UTC 2013
[14:10:11] <icinga-wm> RECOVERY - Puppet freshness on mw121 is OK: puppet ran at Wed Jul 31 14:10:02 UTC 2013
[14:10:11] <icinga-wm> RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Wed Jul 31 14:10:02 UTC 2013
[14:10:11] <icinga-wm> RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Wed Jul 31 14:10:02 UTC 2013
[14:10:11] <icinga-wm> RECOVERY - Puppet freshness on mw1188 is OK: puppet ran at Wed Jul 31 14:10:02 UTC 2013
[14:10:12] <icinga-wm> RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Wed Jul 31 14:10:07 UTC 2013
[14:10:12] <icinga-wm> RECOVERY - Puppet freshness on mw1007 is OK: puppet ran at Wed Jul 31 14:10:07 UTC 2013
[14:10:13] <icinga-wm> RECOVERY - Puppet freshness on mw1197 is OK: puppet ran at Wed Jul 31 14:10:07 UTC 2013
[14:10:13] <icinga-wm> RECOVERY - Puppet freshness on mw1140 is OK: puppet ran at Wed Jul 31 14:10:07 UTC 2013
[14:10:14] <akosiaris> damn i hate this bot
[14:10:14] <icinga-wm> RECOVERY - Puppet freshness on mw1087 is OK: puppet ran at Wed Jul 31 14:10:07 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on srv276 is OK: puppet ran at Wed Jul 31 14:10:12 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on mw1106 is OK: puppet ran at Wed Jul 31 14:10:12 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on mw1043 is OK: puppet ran at Wed Jul 31 14:10:12 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Wed Jul 31 14:10:12 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Wed Jul 31 14:10:12 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on mw1041 is OK: puppet ran at Wed Jul 31 14:10:17 UTC 2013
[14:10:21] <icinga-wm> RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Wed Jul 31 14:10:17 UTC 2013
[14:10:22] <icinga-wm> RECOVERY - Puppet freshness on srv292 is OK: puppet ran at Wed Jul 31 14:10:17 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Wed Jul 31 14:10:22 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on mw1171 is OK: puppet ran at Wed Jul 31 14:10:22 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on mw1210 is OK: puppet ran at Wed Jul 31 14:10:22 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on ekrem is OK: puppet ran at Wed Jul 31 14:10:22 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on mw1186 is OK: puppet ran at Wed Jul 31 14:10:27 UTC 2013
[14:10:31] <icinga-wm> RECOVERY - Puppet freshness on mw1063 is OK: puppet ran at Wed Jul 31 14:10:27 UTC 2013
[14:10:41] <icinga-wm> PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours
[14:10:41] <icinga-wm> PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours
[14:10:41] <icinga-wm> PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours
[14:10:41] <icinga-wm> PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours
[14:10:41] <icinga-wm> PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours
[14:11:51] <icinga-wm> RECOVERY - Puppet freshness on rdb1003 is OK: puppet ran at Wed Jul 31 14:11:43 UTC 2013
[14:11:51] <icinga-wm> RECOVERY - Puppet freshness on es6 is OK: puppet ran at Wed Jul 31 14:11:43 UTC 2013
[14:11:51] <icinga-wm> RECOVERY - Puppet freshness on db63 is OK: puppet ran at Wed Jul 31 14:11:43 UTC 2013
[14:11:52] <icinga-wm> RECOVERY - Puppet freshness on wtp1007 is OK: puppet ran at Wed Jul 31 14:11:48 UTC 2013
[14:12:01] <icinga-wm> RECOVERY - Puppet freshness on mw2 is OK: puppet ran at Wed Jul 31 14:11:53 UTC 2013
[14:12:01] <icinga-wm> RECOVERY - Puppet freshness on cp1010 is OK: puppet ran at Wed Jul 31 14:11:53 UTC 2013
[14:12:01] <icinga-wm> RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Wed Jul 31 14:11:53 UTC 2013
[14:12:01] <icinga-wm> RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Wed Jul 31 14:11:58 UTC 2013
[14:12:01] <icinga-wm> RECOVERY - Puppet freshness on mw98 is OK: puppet ran at Wed Jul 31 14:11:58 UTC 2013
[14:12:11] <icinga-wm> RECOVERY - Puppet freshness on mw1189 is OK: puppet ran at Wed Jul 31 14:12:03 UTC 2013
[14:12:11] <icinga-wm> RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Wed Jul 31 14:12:03 UTC 2013
[14:12:11] <icinga-wm> RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Wed Jul 31 14:12:03 UTC 2013
[14:12:11] <icinga-wm> RECOVERY - Puppet freshness on mw1003 is OK: puppet ran at Wed Jul 31 14:12:08 UTC 2013
[14:12:11] <icinga-wm> RECOVERY - Puppet freshness on mw1033 is OK: puppet ran at Wed Jul 31 14:12:08 UTC 2013
[14:12:12] <icinga-wm> RECOVERY - Puppet freshness on mw1201 is OK: puppet ran at Wed Jul 31 14:12:08 UTC 2013
[14:12:12] <icinga-wm> RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Wed Jul 31 14:12:08 UTC 2013
[14:12:13] <icinga-wm> RECOVERY - Puppet freshness on search33 is OK: puppet ran at Wed Jul 31 14:12:08 UTC 2013
[14:12:41] <icinga-wm> RECOVERY - Puppet freshness on mw1122 is OK: puppet ran at Wed Jul 31 14:12:33 UTC 2013
[14:12:41] <icinga-wm> RECOVERY - Puppet freshness on mw1024 is OK: puppet ran at Wed Jul 31 14:12:38 UTC 2013
[14:12:51] <icinga-wm> RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on calcium is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on sq63 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on pc1 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:52] <icinga-wm> RECOVERY - Puppet freshness on db48 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:53] <icinga-wm> RECOVERY - Puppet freshness on db1040 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:53] <icinga-wm> RECOVERY - Puppet freshness on db1002 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:54] <icinga-wm> RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:54] <icinga-wm> RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:55] <icinga-wm> RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:12:55] <icinga-wm> RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Wed Jul 31 14:12:44 UTC 2013
[14:13:01] <icinga-wm> RECOVERY - Puppet freshness on cp1037 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:01] <icinga-wm> RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:01] <icinga-wm> RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:01] <icinga-wm> RECOVERY - Puppet freshness on mw104 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:01] <icinga-wm> RECOVERY - Puppet freshness on srv260 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:02] <icinga-wm> RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Wed Jul 31 14:12:54 UTC 2013
[14:13:02] <icinga-wm> RECOVERY - Puppet freshness on mw112 is OK: puppet ran at Wed Jul 31 14:12:59 UTC 2013
[14:13:03] <icinga-wm> RECOVERY - Puppet freshness on search25 is OK: puppet ran at Wed Jul 31 14:12:59 UTC 2013
[14:13:03] <icinga-wm> RECOVERY - Puppet freshness on mw68 is OK: puppet ran at Wed Jul 31 14:12:59 UTC 2013
[14:13:11] <icinga-wm> RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Wed Jul 31 14:13:04 UTC 2013
[14:13:11] <icinga-wm> RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Wed Jul 31 14:13:04 UTC 2013
[14:13:11] <icinga-wm> RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Wed Jul 31 14:13:04 UTC 2013
[14:13:11] <icinga-wm> RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Wed Jul 31 14:13:04 UTC 2013
[14:13:11] <icinga-wm> RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Wed Jul 31 14:13:09 UTC 2013
[14:13:12] <icinga-wm> RECOVERY - Puppet freshness on mw1092 is OK: puppet ran at Wed Jul 31 14:13:09 UTC 2013
[14:13:12] <icinga-wm> RECOVERY - Puppet freshness on mw1091 is OK: puppet ran at Wed Jul 31 14:13:09 UTC 2013
[14:13:13] <icinga-wm> RECOVERY - Puppet freshness on mw1118 is OK: puppet ran at Wed Jul 31 14:13:09 UTC 2013
[14:13:21] <icinga-wm> RECOVERY - Puppet freshness on fluorine is OK: puppet ran at Wed Jul 31 14:13:14 UTC 2013
[14:13:21] <icinga-wm> RECOVERY - Puppet freshness on search16 is OK: puppet ran at Wed Jul 31 14:13:14 UTC 2013
[14:13:21] <icinga-wm> RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Wed Jul 31 14:13:14 UTC 2013
[14:13:21] <icinga-wm> RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Wed Jul 31 14:13:19 UTC 2013
[14:13:21] <icinga-wm> RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Wed Jul 31 14:13:19 UTC 2013
[14:13:31] <icinga-wm> RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Wed Jul 31 14:13:24 UTC 2013
[14:13:31] <icinga-wm> RECOVERY - Puppet freshness on mw37 is OK: puppet ran at Wed Jul 31 14:13:29 UTC 2013
[14:13:41] <icinga-wm> RECOVERY - Puppet freshness on mw1025 is OK: puppet ran at Wed Jul 31 14:13:34 UTC 2013
[14:13:41] <icinga-wm> RECOVERY - Puppet freshness on db54 is OK: puppet ran at Wed Jul 31 14:13:34 UTC 2013
[14:13:41] <icinga-wm> RECOVERY - Puppet freshness on mw133 is OK: puppet ran at Wed Jul 31 14:13:34 UTC 2013
[14:13:41] <icinga-wm> RECOVERY - Puppet freshness on db1034 is OK: puppet ran at Wed Jul 31 14:13:34 UTC 2013
[14:13:41] <icinga-wm> RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Wed Jul 31 14:13:39 UTC 2013
[14:13:51] <icinga-wm> RECOVERY - Puppet freshness on db66 is OK: puppet ran at Wed Jul 31 14:13:44 UTC 2013
[14:13:51] <icinga-wm> RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Wed Jul 31 14:13:44 UTC 2013
[14:13:51] <icinga-wm> RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Wed Jul 31 14:13:44 UTC 2013
[14:13:51] <icinga-wm> RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Wed Jul 31 14:13:49 UTC 2013
[14:13:51] <icinga-wm> RECOVERY - Puppet freshness on db1042 is OK: puppet ran at Wed Jul 31 14:13:49 UTC 2013
[14:14:01] <icinga-wm> RECOVERY - Puppet freshness on mw79 is OK: puppet ran at Wed Jul 31 14:13:54 UTC 2013
[14:14:01] <icinga-wm> RECOVERY - Puppet freshness on mw28 is OK: puppet ran at Wed Jul 31 14:13:54 UTC 2013
[14:14:01] <icinga-wm> RECOVERY - Puppet freshness on mw131 is OK: puppet ran at Wed Jul 31 14:13:54 UTC 2013
[14:14:01] <icinga-wm> RECOVERY - Puppet freshness on mw23 is OK: puppet ran at Wed Jul 31 14:13:54 UTC 2013
[14:14:01] <icinga-wm> RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:02] <icinga-wm> RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:02] <icinga-wm> RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:03] <icinga-wm> RECOVERY - Puppet freshness on search29 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:03] <icinga-wm> RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:04] <icinga-wm> RECOVERY - Puppet freshness on mw1150 is OK: puppet ran at Wed Jul 31 14:13:59 UTC 2013
[14:14:11] <icinga-wm> RECOVERY - Puppet freshness on mw1054 is OK: puppet ran at Wed Jul 31 14:14:04 UTC 2013
[14:14:11] <icinga-wm> RECOVERY - Puppet freshness on mw1129 is OK: puppet ran at Wed Jul 31 14:14:04 UTC 2013
[14:14:11] <icinga-wm> RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Wed Jul 31 14:14:10 UTC 2013
[14:14:11] <icinga-wm> RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Wed Jul 31 14:14:10 UTC 2013
[14:14:11] <icinga-wm> RECOVERY - Puppet freshness on mw1143 is OK: puppet ran at Wed Jul 31 14:14:10 UTC 2013
[14:14:21] <icinga-wm> RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Wed Jul 31 14:14:15 UTC 2013
[14:14:21] <icinga-wm> RECOVERY - Puppet freshness on search30 is OK: puppet ran at Wed Jul 31 14:14:15 UTC 2013
[14:14:21] <icinga-wm> RECOVERY - Puppet freshness on mw1107 is OK: puppet ran at Wed Jul 31 14:14:15 UTC 2013
[14:14:31] <icinga-wm> RECOVERY - Puppet freshness on mw1211 is OK: puppet ran at Wed Jul 31 14:14:30 UTC 2013
[14:14:31] <icinga-wm> RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Wed Jul 31 14:14:30 UTC 2013
[14:14:31] <icinga-wm> RECOVERY - Puppet freshness on mw1213 is OK: puppet ran at Wed Jul 31 14:14:30 UTC 2013
[14:14:41] <icinga-wm> PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours
[14:14:41] <icinga-wm> RECOVERY - Puppet freshness on sq52 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:41] <icinga-wm> RECOVERY - Puppet freshness on mw1066 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:41] <icinga-wm> RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:41] <icinga-wm> RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:42] <icinga-wm> RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:42] <icinga-wm> RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:43] <icinga-wm> RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Wed Jul 31 14:14:35 UTC 2013
[14:14:43] <icinga-wm> RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Wed Jul 31 14:14:40 UTC 2013
[14:14:44] <icinga-wm> RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Wed Jul 31 14:14:40 UTC 2013
[14:14:44] <icinga-wm> RECOVERY - Puppet freshness on solr1 is OK: puppet ran at Wed Jul 31 14:14:40 UTC 2013
[14:14:45] <icinga-wm> RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Wed Jul 31 14:14:40 UTC 2013
[14:14:45] <icinga-wm> RECOVERY - Puppet freshness on antimony is OK: puppet ran at Wed Jul 31 14:14:40 UTC 2013
[14:14:50] <akosiaris> so... seems like icinga was not processing external commands. A restart fixed it, looking for the reason now
[14:14:51] <icinga-wm> RECOVERY - Puppet freshness on db1014 is OK: puppet ran at Wed Jul 31 14:14:45 UTC 2013
[14:14:51] <icinga-wm> RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Wed Jul 31 14:14:45 UTC 2013
[14:14:51] <icinga-wm> RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Wed Jul 31 14:14:45 UTC 2013
[14:14:51] <icinga-wm> RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Wed Jul 31 14:14:45 UTC 2013
[14:15:01] <icinga-wm> RECOVERY - Puppet freshness on ssl1 is OK: puppet ran at Wed Jul 31 14:14:50 UTC 2013
[14:15:01] <icinga-wm> RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Wed Jul 31 14:14:50 UTC 2013
[14:15:01] <icinga-wm> RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Wed Jul 31 14:14:50 UTC 2013
[14:15:01] <icinga-wm> RECOVERY - Puppet freshness on mw42 is OK: puppet ran at Wed Jul 31 14:14:50 UTC 2013
[14:15:01] <icinga-wm> RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Wed Jul 31 14:14:51 UTC 2013
[14:15:02] <icinga-wm> RECOVERY - Puppet freshness on mw75 is OK: puppet ran at Wed Jul 31 14:14:56 UTC 2013
[14:15:02] <icinga-wm> RECOVERY - Puppet freshness on mw1206 is OK: puppet ran at Wed Jul 31 14:14:56 UTC 2013
[14:15:11] <icinga-wm> RECOVERY - Puppet freshness on mw1208 is OK: puppet ran at Wed Jul 31 14:15:01 UTC 2013
[14:15:11] <icinga-wm> RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Wed Jul 31 14:15:01 UTC 2013
[14:15:11] <icinga-wm> RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Wed Jul 31 14:15:01 UTC 2013
[14:15:11] <icinga-wm> RECOVERY - Puppet freshness on mw1104 is OK: puppet ran at Wed Jul 31 14:15:01 UTC 2013
[14:15:11] <icinga-wm> RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Wed Jul 31 14:15:06 UTC 2013
[14:15:12] <icinga-wm> RECOVERY - Puppet freshness on mw1021 is OK: puppet ran at Wed Jul 31 14:15:06 UTC 2013
[14:15:41] <icinga-wm> RECOVERY - Puppet freshness on db9 is OK: puppet ran at Wed Jul 31 14:15:31 UTC 2013
[14:15:51] <icinga-wm> RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Wed Jul 31 14:15:46 UTC 2013
[14:15:52] <icinga-wm> RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Wed Jul 31 14:15:46 UTC 2013
[14:15:52] <icinga-wm> RECOVERY - Puppet freshness on es1004 is OK: puppet ran at Wed Jul 31 14:15:46 UTC 2013
[14:15:52] <icinga-wm> RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Wed Jul 31 14:15:46 UTC 2013
[14:16:01] <icinga-wm> RECOVERY - Puppet freshness on sodium is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:01] <icinga-wm> RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:01] <icinga-wm> RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:01] <icinga-wm> RECOVERY - Puppet freshness on cp1066 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:01] <icinga-wm> RECOVERY - Puppet freshness on capella is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:02] <icinga-wm> RECOVERY - Puppet freshness on cp1060 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:02] <icinga-wm> RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:03] <icinga-wm> RECOVERY - Puppet freshness on mw61 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:03] <icinga-wm> RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Wed Jul 31 14:15:51 UTC 2013
[14:16:04] <icinga-wm> RECOVERY - Puppet freshness on cp1018 is OK: puppet ran at Wed Jul 31 14:15:56 UTC 2013
[14:16:11] <icinga-wm> RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Wed Jul 31 14:16:01 UTC 2013
[14:16:11] <icinga-wm> RECOVERY - Puppet freshness on mw1049 is OK: puppet ran at Wed Jul 31 14:16:01 UTC 2013
[14:16:11] <icinga-wm> RECOVERY - Puppet freshness on mw1194 is OK: puppet ran at Wed Jul 31 14:16:01 UTC 2013
[14:16:11] <icinga-wm> RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Wed Jul 31 14:16:06 UTC 2013
[14:16:11] <icinga-wm> RECOVERY - Puppet freshness on ssl1005 is OK: puppet ran at Wed Jul 31 14:16:06 UTC 2013
[14:16:12] <icinga-wm> RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Wed Jul 31 14:16:06 UTC 2013
[14:16:21] <icinga-wm> RECOVERY - Puppet freshness on srv299 is OK: puppet ran at Wed Jul 31 14:16:11 UTC 2013
[14:16:21] <icinga-wm> RECOVERY - Puppet freshness on mw93 is OK: puppet ran at Wed Jul 31 14:16:11 UTC 2013
[14:16:21] <icinga-wm> RECOVERY - Puppet freshness on tmh1001 is OK: puppet ran at Wed Jul 31 14:16:11 UTC 2013
[14:16:21] <icinga-wm> RECOVERY - Puppet freshness on mw74 is OK: puppet ran at Wed Jul 31 14:16:16 UTC 2013
[14:16:21] <icinga-wm> RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Wed Jul 31 14:16:16 UTC 2013
[14:16:22] <icinga-wm> RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Wed Jul 31 14:16:16 UTC 2013
[14:16:31] <icinga-wm> RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Wed Jul 31 14:16:21 UTC 2013
[14:16:31] <icinga-wm> RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Wed Jul 31 14:16:21 UTC 2013
[14:16:31] <icinga-wm> RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Wed Jul 31 14:16:26 UTC 2013
[14:16:31] <icinga-wm> RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Wed Jul 31 14:16:26 UTC 2013
[14:16:41] <icinga-wm> RECOVERY - Puppet freshness on sq62 is OK: puppet ran at Wed Jul 31 14:16:31 UTC 2013
[14:16:41] <icinga-wm> RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Wed Jul 31 14:16:31 UTC 2013
[14:16:41] <icinga-wm> RECOVERY - Puppet freshness on cp1017 is OK: puppet ran at Wed Jul 31 14:16:31 UTC 2013
[14:16:41] <icinga-wm> RECOVERY - Puppet freshness on sanger is OK: puppet ran at Wed Jul 31 14:16:36 UTC 2013
[14:16:51] <icinga-wm> RECOVERY - Puppet freshness on cp1016 is OK: puppet ran at Wed Jul 31 14:16:41 UTC 2013
[14:16:51] <icinga-wm> RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Wed Jul 31 14:16:41 UTC 2013
[14:16:51] <icinga-wm> RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Wed Jul 31 14:16:46 UTC 2013
[14:16:51] <icinga-wm> RECOVERY - Puppet freshness on db67 is OK: puppet ran at Wed Jul 31 14:16:46 UTC 2013
[14:16:51] <icinga-wm> RECOVERY - Puppet freshness on db69 is OK: puppet ran at Wed Jul 31 14:16:46 UTC 2013
[14:16:52] <icinga-wm> RECOVERY - Puppet freshness on es9 is OK: puppet ran at Wed Jul 31 14:16:46 UTC 2013
[14:16:52] <icinga-wm> RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Wed Jul 31 14:16:46 UTC 2013
[14:17:01] <icinga-wm> RECOVERY - Puppet freshness on db32 is OK: puppet ran at Wed Jul 31 14:16:51 UTC 2013
[14:17:01] <icinga-wm> RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Wed Jul 31 14:16:51 UTC 2013
[14:17:01] <icinga-wm> RECOVERY - Puppet freshness on mw13 is OK: puppet ran at Wed Jul 31 14:16:56 UTC 2013
[14:17:01] <icinga-wm> RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Wed Jul 31 14:16:56 UTC 2013
[14:17:11] <icinga-wm> RECOVERY - Puppet freshness on mw45 is OK: puppet ran at Wed Jul 31 14:17:01 UTC 2013
[14:17:11] <icinga-wm> RECOVERY - Puppet freshness on cp1062 is OK: puppet ran at Wed Jul 31 14:17:01 UTC 2013
[14:17:11] <icinga-wm> RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Wed Jul 31 14:17:01 UTC 2013
[14:17:11] <icinga-wm> RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Wed Jul 31 14:17:01 UTC 2013
[14:17:11] <icinga-wm> RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Wed Jul 31 14:17:01 UTC 2013
[14:17:12] <icinga-wm> RECOVERY - Puppet freshness on mw1105 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:12] <icinga-wm> RECOVERY - Puppet freshness on mw1179 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:13] <icinga-wm> RECOVERY - Puppet freshness on mw1181 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:13] <icinga-wm> RECOVERY - Puppet freshness on hume is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:14] <icinga-wm> RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:14] <icinga-wm> RECOVERY - Puppet freshness on mw1030 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:15] <icinga-wm> RECOVERY - Puppet freshness on db1005 is OK: puppet ran at Wed Jul 31 14:17:06 UTC 2013
[14:17:21] <icinga-wm> RECOVERY - Puppet freshness on search31 is OK: puppet ran at Wed Jul 31 14:17:11 UTC 2013
[14:17:21] <icinga-wm> RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Wed Jul 31 14:17:11 UTC 2013
[14:17:21] <icinga-wm> RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Wed Jul 31 14:17:11 UTC 2013
[14:17:21] <icinga-wm> RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Wed Jul 31 14:17:11 UTC 2013
[14:17:21] <icinga-wm> RECOVERY - Puppet freshness on mw1165 is OK: puppet ran at Wed Jul 31 14:17:16 UTC 2013
[14:17:31] <icinga-wm> RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Wed Jul 31 14:17:21 UTC 2013
[14:17:41] <icinga-wm> RECOVERY - Puppet freshness on nickel is OK: puppet ran at Wed Jul 31 14:17:31 UTC 2013
[14:17:41] <icinga-wm> RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Wed Jul 31 14:17:31 UTC 2013
[14:17:41] <icinga-wm> RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Wed Jul 31 14:17:31 UTC 2013
[14:17:41] <icinga-wm> RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Wed Jul 31 14:17:36 UTC 2013
[14:17:51] <icinga-wm> RECOVERY - Puppet freshness on db77 is OK: puppet ran at Wed Jul 31 14:17:41 UTC 2013
[14:17:51] <icinga-wm> RECOVERY - Puppet freshness on db65 is OK: puppet ran at Wed Jul 31 14:17:46 UTC 2013
[14:17:52] <icinga-wm> RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Wed Jul 31 14:17:46 UTC 2013
[14:17:52] <icinga-wm> RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Wed Jul 31 14:17:46 UTC 2013
[14:18:01] <icinga-wm> RECOVERY - Puppet freshness on db1058 is OK: puppet ran at Wed Jul 31 14:17:51 UTC 2013
[14:18:01] <icinga-wm> RECOVERY - Puppet freshness on ms-be1007 is OK: puppet ran at Wed Jul 31 14:17:51 UTC 2013
[14:18:01] <icinga-wm> RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Wed Jul 31 14:17:56 UTC 2013
[14:18:01] <icinga-wm> RECOVERY - Puppet freshness on cp1038 is OK: puppet ran at Wed Jul 31 14:17:56 UTC 2013
[14:18:01] <icinga-wm> RECOVERY - Puppet freshness on mw99 is OK: puppet ran at Wed Jul 31 14:17:56 UTC 2013
[14:18:02] <icinga-wm> RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Wed Jul 31 14:17:56 UTC 2013
[14:18:11] <icinga-wm> RECOVERY - Puppet freshness on srv272 is OK: puppet ran at Wed Jul 31 14:18:01 UTC 2013
[14:18:11] <icinga-wm> RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Wed Jul 31 14:18:01 UTC 2013
[14:18:11] <icinga-wm> RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Wed Jul 31 14:18:01 UTC 2013
[14:18:11] <icinga-wm> RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Wed Jul 31 14:18:01 UTC 2013
[14:18:11] <icinga-wm> RECOVERY - Puppet freshness on erbium is OK: puppet ran at Wed Jul 31 14:18:01 UTC 2013
[14:18:12] <icinga-wm> RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Wed Jul 31 14:18:06 UTC 2013
[14:18:12] <icinga-wm> RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Wed Jul 31 14:18:06 UTC 2013
[14:18:13] <icinga-wm> RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Wed Jul 31 14:18:06 UTC 2013
[14:18:13] <icinga-wm> RECOVERY - Puppet freshness on mw1074 is OK: puppet ran at Wed Jul 31 14:18:06 UTC 2013
[14:18:14] <icinga-wm> RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Wed Jul 31 14:18:06 UTC 2013
[14:18:21] <icinga-wm> RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:21] <icinga-wm> RECOVERY - Puppet freshness on mw1116 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:21] <icinga-wm> RECOVERY - Puppet freshness on mw1094 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:21] <icinga-wm> RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:21] <icinga-wm> RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:22] <icinga-wm> RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:22] <icinga-wm> RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Wed Jul 31 14:18:11 UTC 2013
[14:18:31] <icinga-wm> RECOVERY - Puppet freshness on brewster is OK: puppet ran at Wed Jul 31 14:18:26 UTC 2013
[14:18:31] <icinga-wm> RECOVERY - Puppet freshness on analytics1002 is OK: puppet ran at Wed Jul 31 14:18:26 UTC 2013
[14:18:31] <icinga-wm> RECOVERY - Puppet freshness on sq50 is OK: puppet ran at Wed Jul 31 14:18:27 UTC 2013
[14:18:31] <icinga-wm> RECOVERY - Puppet freshness on db57 is OK: puppet ran at Wed Jul 31 14:18:27 UTC 2013
[14:18:41] <icinga-wm> RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Wed Jul 31 14:18:32 UTC 2013
[14:18:41] <icinga-wm> RECOVERY - Puppet freshness on mw1084 is OK: puppet ran at Wed Jul 31 14:18:32 UTC 2013
[14:18:41] <icinga-wm> RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Wed Jul 31 14:18:32 UTC 2013
[14:18:41] <icinga-wm> RECOVERY - Puppet freshness on db51 is OK: puppet ran at Wed Jul 31 14:18:37 UTC 2013
[14:18:41] <icinga-wm> RECOVERY - Puppet freshness on es5 is OK: puppet ran at Wed Jul 31 14:18:37 UTC 2013
[14:18:42] <icinga-wm> RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Wed Jul 31 14:18:37 UTC 2013
[14:18:51] <icinga-wm> RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Wed Jul 31 14:18:42 UTC 2013
[14:18:51] <icinga-wm> RECOVERY - Puppet freshness on wtp1013 is OK: puppet ran at Wed Jul 31 14:18:42 UTC 2013
[14:18:51] <icinga-wm> RECOVERY - Puppet freshness on mc1008 is OK: puppet ran at Wed Jul 31 14:18:42 UTC 2013
[14:18:51] <icinga-wm> RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Wed Jul 31 14:18:42 UTC 2013
[14:18:51] <icinga-wm> RECOVERY - Puppet freshness on mw26 is OK: puppet ran at Wed Jul 31 14:18:47 UTC 2013
[14:18:52] <grrrit-wm> (�03PS2�) �10�Mark Bergsma��: Do not mark HTTPS requests with X-CS header [operations/puppet] - �10https://gerrit.wikimedia.org/r/76873� (owner: �10�Yurik��)
[14:18:52] <icinga-wm> RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Wed Jul 31 14:18:47 UTC 2013
[14:18:52] <icinga-wm> RECOVERY - Puppet freshness on cp1065 is OK: puppet ran at Wed Jul 31 14:18:47 UTC 2013
[14:18:53] <icinga-wm> RECOVERY - Puppet freshness on mw64 is OK: puppet ran at Wed Jul 31 14:18:47 UTC 2013
[14:19:01] <icinga-wm> RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Wed Jul 31 14:18:52 UTC 2013
[14:19:01] <icinga-wm> RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Wed Jul 31 14:18:57 UTC 2013
[14:19:01] <icinga-wm> RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Wed Jul 31 14:18:57 UTC 2013
[14:19:31] <icinga-wm> RECOVERY - Puppet freshness on db46 is OK: puppet ran at Wed Jul 31 14:19:27 UTC 2013
[14:19:31] <icinga-wm> RECOVERY - Puppet freshness on tarin is OK: puppet ran at Wed Jul 31 14:19:27 UTC 2013
[14:19:31] <icinga-wm> RECOVERY - Puppet freshness on mw134 is OK: puppet ran at Wed Jul 31 14:19:27 UTC 2013
[14:19:41] <icinga-wm> RECOVERY - Puppet freshness on labsdb1001 is OK: puppet ran at Wed Jul 31 14:19:32 UTC 2013
[14:19:41] <icinga-wm> RECOVERY - Puppet freshness on db55 is OK: puppet ran at Wed Jul 31 14:19:32 UTC 2013
[14:19:41] <icinga-wm> RECOVERY - Puppet freshness on ms-be1009 is OK: puppet ran at Wed Jul 31 14:19:37 UTC 2013
[14:19:51] <icinga-wm> RECOVERY - Puppet freshness on mw109 is OK: puppet ran at Wed Jul 31 14:19:42 UTC 2013
[14:19:51] <icinga-wm> RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Wed Jul 31 14:19:42 UTC 2013
[14:19:51] <icinga-wm> RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Wed Jul 31 14:19:42 UTC 2013
[14:19:51] <icinga-wm> RECOVERY - Puppet freshness on mw70 is OK: puppet ran at Wed Jul 31 14:19:42 UTC 2013
[14:19:51] <icinga-wm> RECOVERY - Puppet freshness on mw1218 is OK: puppet ran at Wed Jul 31 14:19:47 UTC 2013
[14:20:01] <icinga-wm> RECOVERY - Puppet freshness on snapshot4 is OK: puppet ran at Wed Jul 31 14:19:52 UTC 2013
[14:20:01] <icinga-wm> RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Wed Jul 31 14:19:52 UTC 2013
[14:20:01] <icinga-wm> RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Wed Jul 31 14:19:52 UTC 2013
[14:20:01] <icinga-wm> RECOVERY - Puppet freshness on mw1062 is OK: puppet ran at Wed Jul 31 14:19:52 UTC 2013
[14:20:11] <icinga-wm> RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Wed Jul 31 14:20:02 UTC 2013
[14:20:11] <icinga-wm> RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Wed Jul 31 14:20:02 UTC 2013
[14:20:11] <icinga-wm> RECOVERY - Puppet freshness on srv296 is OK: puppet ran at Wed Jul 31 14:20:02 UTC 2013
[14:20:11] <icinga-wm> RECOVERY - Puppet freshness on wtp1021 is OK: puppet ran at Wed Jul 31 14:20:02 UTC 2013
[14:20:31] <icinga-wm> RECOVERY - Puppet freshness on mw1132 is OK: puppet ran at Wed Jul 31 14:20:27 UTC 2013
[14:20:31] <icinga-wm> RECOVERY - Puppet freshness on mw1001 is OK: puppet ran at Wed Jul 31 14:20:27 UTC 2013
[14:20:35] <mark> apparently yurik's style is "let's do all these ugly inefficient dirty hacks! and then if we figure out we didn't need to, let's undo it!"
[14:20:41] <icinga-wm> RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:41] <icinga-wm> RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:41] <icinga-wm> RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:41] <icinga-wm> RECOVERY - Puppet freshness on sq55 is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:41] <icinga-wm> RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:42] <icinga-wm> RECOVERY - Puppet freshness on aluminium is OK: puppet ran at Wed Jul 31 14:20:32 UTC 2013
[14:20:42] <icinga-wm> RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Wed Jul 31 14:20:37 UTC 2013
[14:20:43] <icinga-wm> RECOVERY - Puppet freshness on amssq40 is OK: puppet ran at Wed Jul 31 14:20:37 UTC 2013
[14:20:51] <icinga-wm> RECOVERY - Puppet freshness on ssl1006 is OK: puppet ran at Wed Jul 31 14:20:42 UTC 2013
[14:20:51] <icinga-wm> RECOVERY - Puppet freshness on srv285 is OK: puppet ran at Wed Jul 31 14:20:42 UTC 2013
[14:20:51] <icinga-wm> RECOVERY - Puppet freshness on mw111 is OK: puppet ran at Wed Jul 31 14:20:42 UTC 2013
[14:20:51] <icinga-wm> RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Wed Jul 31 14:20:42 UTC 2013
[14:20:51] <icinga-wm> RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Wed Jul 31 14:20:42 UTC 2013
[14:21:01] <icinga-wm> RECOVERY - Puppet freshness on mw1219 is OK: puppet ran at Wed Jul 31 14:20:52 UTC 2013
[14:21:01] <icinga-wm> RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Wed Jul 31 14:20:52 UTC 2013
[14:21:01] <icinga-wm> RECOVERY - Puppet freshness on mw1178 is OK: puppet ran at Wed Jul 31 14:20:52 UTC 2013
[14:21:01] <icinga-wm> RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Wed Jul 31 14:20:52 UTC 2013
[14:21:01] <icinga-wm> RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Wed Jul 31 14:20:52 UTC 2013
[14:21:02] <icinga-wm> RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Wed Jul 31 14:20:57 UTC 2013
[14:21:02] <icinga-wm> RECOVERY - Puppet freshness on professor is OK: puppet ran at Wed Jul 31 14:20:57 UTC 2013
[14:21:11] <icinga-wm> RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Wed Jul 31 14:21:02 UTC 2013
[14:21:11] <icinga-wm> RECOVERY - Puppet freshness on db31 is OK: puppet ran at Wed Jul 31 14:21:02 UTC 2013
[14:21:11] <icinga-wm> RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Wed Jul 31 14:21:07 UTC 2013
[14:21:41] <icinga-wm> RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Wed Jul 31 14:21:32 UTC 2013
[14:21:41] <icinga-wm> RECOVERY - Puppet freshness on mw1072 is OK: puppet ran at Wed Jul 31 14:21:32 UTC 2013
[14:21:51] <icinga-wm> RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Wed Jul 31 14:21:47 UTC 2013
[14:22:01] <icinga-wm> RECOVERY - Puppet freshness on search27 is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:01] <icinga-wm> RECOVERY - Puppet freshness on mw1093 is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:01] <icinga-wm> RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:01] <icinga-wm> RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:01] <icinga-wm> RECOVERY - Puppet freshness on wtp1010 is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:02] <icinga-wm> RECOVERY - Puppet freshness on sq78 is OK: puppet ran at Wed Jul 31 14:21:52 UTC 2013
[14:22:02] <icinga-wm> RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:03] <icinga-wm> RECOVERY - Puppet freshness on db60 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:03] <icinga-wm> RECOVERY - Puppet freshness on mw1080 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:04] <icinga-wm> RECOVERY - Puppet freshness on cp1015 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:04] <icinga-wm> RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:05] <icinga-wm> RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:05] <icinga-wm> RECOVERY - Puppet freshness on ms-be1004 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:06] <icinga-wm> RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:06] <icinga-wm> RECOVERY - Puppet freshness on db29 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:07] <icinga-wm> RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:07] <icinga-wm> RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:08] <icinga-wm> RECOVERY - Puppet freshness on db73 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:08] <icinga-wm> RECOVERY - Puppet freshness on mw1139 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:09] <icinga-wm> RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:09] <icinga-wm> RECOVERY - Puppet freshness on srv282 is OK: puppet ran at Wed Jul 31 14:21:57 UTC 2013
[14:22:11] <icinga-wm> RECOVERY - Puppet freshness on mw115 is OK: puppet ran at Wed Jul 31 14:22:03 UTC 2013
[14:22:11] <icinga-wm> RECOVERY - Puppet freshness on srv259 is OK: puppet ran at Wed Jul 31 14:22:03 UTC 2013
[14:22:11] <icinga-wm> RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Wed Jul 31 14:22:03 UTC 2013
[14:22:11] <icinga-wm> RECOVERY - Puppet freshness on srv280 is OK: puppet ran at Wed Jul 31 14:22:03 UTC 2013
[14:22:12] <icinga-wm> RECOVERY - Puppet freshness on tmh1 is OK: puppet ran at Wed Jul 31 14:22:08 UTC 2013
[14:22:12] <icinga-wm> RECOVERY - Puppet freshness on mw1215 is OK: puppet ran at Wed Jul 31 14:22:08 UTC 2013
[14:22:13] <icinga-wm> RECOVERY - Puppet freshness on mw1220 is OK: puppet ran at Wed Jul 31 14:22:08 UTC 2013
[14:22:13] <icinga-wm> RECOVERY - Puppet freshness on db59 is OK: puppet ran at Wed Jul 31 14:22:08 UTC 2013
[14:22:14] <icinga-wm> RECOVERY - Puppet freshness on mw52 is OK: puppet ran at Wed Jul 31 14:22:08 UTC 2013
[14:22:21] <icinga-wm> RECOVERY - Puppet freshness on mw1045 is OK: puppet ran at Wed Jul 31 14:22:13 UTC 2013
[14:22:21] <icinga-wm> RECOVERY - Puppet freshness on mw1086 is OK: puppet ran at Wed Jul 31 14:22:13 UTC 2013
[14:22:21] <icinga-wm> RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Wed Jul 31 14:22:13 UTC 2013
[14:22:21] <icinga-wm> RECOVERY - Puppet freshness on db36 is OK: puppet ran at Wed Jul 31 14:22:13 UTC 2013
[14:22:21] <icinga-wm> RECOVERY - Puppet freshness on mw1203 is OK: puppet ran at Wed Jul 31 14:22:13 UTC 2013
[14:22:22] <icinga-wm> RECOVERY - Puppet freshness on terbium is OK: puppet ran at Wed Jul 31 14:22:18 UTC 2013
[14:22:22] <icinga-wm> RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Wed Jul 31 14:22:18 UTC 2013
[14:22:23] <icinga-wm> RECOVERY - Puppet freshness on srv240 is OK: puppet ran at Wed Jul 31 14:22:18 UTC 2013
[14:22:32] <paravoid> mark: erm, didn't you just remove tag_carrier for all zero traffic?
[14:22:41] <icinga-wm> RECOVERY - Puppet freshness on mw1145 is OK: puppet ran at Wed Jul 31 14:22:38 UTC 2013
[14:22:46] <mark> how?
[14:22:51] <icinga-wm> RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Wed Jul 31 14:22:48 UTC 2013
[14:23:01] <icinga-wm> RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Wed Jul 31 14:22:58 UTC 2013
[14:23:01] <icinga-wm> RECOVERY - Puppet freshness on db56 is OK: puppet ran at Wed Jul 31 14:22:58 UTC 2013
[14:23:01] <icinga-wm> RECOVERY - Puppet freshness on mw65 is OK: puppet ran at Wed Jul 31 14:22:58 UTC 2013
[14:23:10] <paravoid> you moved tag_carrier under if (req.http.X-Forwarded-Proto)
[14:23:11] <icinga-wm> RECOVERY - Puppet freshness on linne is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:11] <icinga-wm> RECOVERY - Puppet freshness on chromium is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:11] <icinga-wm> RECOVERY - Puppet freshness on wtp1020 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:11] <icinga-wm> RECOVERY - Puppet freshness on mw1200 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:11] <icinga-wm> RECOVERY - Puppet freshness on db1057 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:12] <icinga-wm> RECOVERY - Puppet freshness on db44 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:12] <icinga-wm> RECOVERY - Puppet freshness on wtp1006 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:13] <icinga-wm> RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:13] <icinga-wm> RECOVERY - Puppet freshness on mc1006 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:14] <icinga-wm> RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:14] <icinga-wm> RECOVERY - Puppet freshness on mw1141 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:15] <icinga-wm> RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:15] <icinga-wm> RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:16] <icinga-wm> RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:16] <icinga-wm> RECOVERY - Puppet freshness on es1008 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:17] <icinga-wm> RECOVERY - Puppet freshness on cp1039 is OK: puppet ran at Wed Jul 31 14:23:03 UTC 2013
[14:23:17] <icinga-wm> RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:18] <icinga-wm> RECOVERY - Puppet freshness on cp1054 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:18] <icinga-wm> RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:19] <icinga-wm> RECOVERY - Puppet freshness on cp1068 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:19] <icinga-wm> RECOVERY - Puppet freshness on srv298 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:20] <icinga-wm> RECOVERY - Puppet freshness on sq37 is OK: puppet ran at Wed Jul 31 14:23:08 UTC 2013
[14:23:21] <icinga-wm> RECOVERY - Puppet freshness on srv268 is OK: puppet ran at Wed Jul 31 14:23:13 UTC 2013
[14:23:21] <icinga-wm> RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Wed Jul 31 14:23:13 UTC 2013
[14:23:21] <icinga-wm> RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Wed Jul 31 14:23:13 UTC 2013
[14:23:22] <icinga-wm> RECOVERY - Puppet freshness on mw1090 is OK: puppet ran at Wed Jul 31 14:23:13 UTC 2013
[14:23:22] <icinga-wm> RECOVERY - Puppet freshness on mw91 is OK: puppet ran at Wed Jul 31 14:23:13 UTC 2013
[14:23:22] <mark> yes
[14:23:23] <icinga-wm> RECOVERY - Puppet freshness on mw113 is OK: puppet ran at Wed Jul 31 14:23:18 UTC 2013
[14:23:23] <icinga-wm> RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Wed Jul 31 14:23:18 UTC 2013
[14:23:24] <icinga-wm> RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Wed Jul 31 14:23:18 UTC 2013
[14:23:31] <icinga-wm> RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Wed Jul 31 14:23:23 UTC 2013
[14:23:31] <icinga-wm> RECOVERY - Puppet freshness on mw15 is OK: puppet ran at Wed Jul 31 14:23:23 UTC 2013
[14:23:31] <icinga-wm> RECOVERY - Puppet freshness on mw1158 is OK: puppet ran at Wed Jul 31 14:23:28 UTC 2013
[14:23:31] <icinga-wm> RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Wed Jul 31 14:23:28 UTC 2013
[14:23:41] <icinga-wm> RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Wed Jul 31 14:23:33 UTC 2013
[14:23:41] <icinga-wm> RECOVERY - Puppet freshness on pdf1 is OK: puppet ran at Wed Jul 31 14:23:33 UTC 2013
[14:23:41] <icinga-wm> RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Wed Jul 31 14:23:33 UTC 2013
[14:23:41] <icinga-wm> RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Wed Jul 31 14:23:33 UTC 2013
[14:23:41] <icinga-wm> RECOVERY - Puppet freshness on wtp1008 is OK: puppet ran at Wed Jul 31 14:23:33 UTC 2013
[14:23:42] <icinga-wm> RECOVERY - Puppet freshness on cp1053 is OK: puppet ran at Wed Jul 31 14:23:38 UTC 2013
[14:23:42] <icinga-wm> RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Wed Jul 31 14:23:38 UTC 2013
[14:23:43] <icinga-wm> RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Wed Jul 31 14:23:38 UTC 2013
[14:23:47] <mark> oh, we should still tag carriers?
[14:23:48] <mark> alright
[14:23:51] <icinga-wm> RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Wed Jul 31 14:23:43 UTC 2013
[14:23:51] <icinga-wm> RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Wed Jul 31 14:23:48 UTC 2013
[14:23:52] <icinga-wm> RECOVERY - Puppet freshness on emery is OK: puppet ran at Wed Jul 31 14:23:48 UTC 2013
[14:24:01] <icinga-wm> RECOVERY - Puppet freshness on virt1005 is OK: puppet ran at Wed Jul 31 14:23:53 UTC 2013
[14:24:01] <icinga-wm> RECOVERY - Puppet freshness on harmon is OK: puppet ran at Wed Jul 31 14:23:53 UTC 2013
[14:24:01] <icinga-wm> RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Wed Jul 31 14:23:53 UTC 2013
[14:24:01] <icinga-wm> RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Wed Jul 31 14:23:53 UTC 2013
[14:24:01] <icinga-wm> RECOVERY - Puppet freshness on ms-be1010 is OK: puppet ran at Wed Jul 31 14:23:58 UTC 2013
[14:24:01] <paravoid> er?
[14:24:02] <icinga-wm> RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Wed Jul 31 14:23:58 UTC 2013
[14:24:02] <icinga-wm> RECOVERY - Puppet freshness on search35 is OK: puppet ran at Wed Jul 31 14:23:58 UTC 2013
[14:24:03] <icinga-wm> RECOVERY - Puppet freshness on mc1002 is OK: puppet ran at Wed Jul 31 14:23:58 UTC 2013
[14:24:10] <paravoid> so you only tag carriers for https traffic?
[14:24:11] <icinga-wm> RECOVERY - Puppet freshness on ms1004 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:11] <icinga-wm> RECOVERY - Puppet freshness on es2 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:11] <icinga-wm> RECOVERY - Puppet freshness on srv293 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:11] <icinga-wm> RECOVERY - Puppet freshness on wtp1016 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:11] <icinga-wm> RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:12] <icinga-wm> RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Wed Jul 31 14:24:03 UTC 2013
[14:24:12] <icinga-wm> RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:13] <icinga-wm> RECOVERY - Puppet freshness on mw7 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:13] <icinga-wm> RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:14] <icinga-wm> RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:14] <icinga-wm> RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:15] <icinga-wm> RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:15] <icinga-wm> RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:16] <icinga-wm> RECOVERY - Puppet freshness on search23 is OK: puppet ran at Wed Jul 31 14:24:08 UTC 2013
[14:24:17] <paravoid> also, tag_carrier is a misnomer really
[14:24:21] <icinga-wm> RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Wed Jul 31 14:24:13 UTC 2013
[14:24:21] <icinga-wm> RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Wed Jul 31 14:24:18 UTC 2013
[14:24:21] <icinga-wm> RECOVERY - Puppet freshness on mw1060 is OK: puppet ran at Wed Jul 31 14:24:18 UTC 2013
[14:24:21] <icinga-wm> RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Wed Jul 31 14:24:18 UTC 2013
[14:24:21] <icinga-wm> RECOVERY - Puppet freshness on mw1108 is OK: puppet ran at Wed Jul 31 14:24:18 UTC 2013
[14:24:31] <mark> oh you mean I am missing a !
[14:24:31] <icinga-wm> RECOVERY - Puppet freshness on mw25 is OK: puppet ran at Wed Jul 31 14:24:24 UTC 2013
[14:24:31] <icinga-wm> RECOVERY - Puppet freshness on mw1009 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:31] <icinga-wm> RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:31] <icinga-wm> RECOVERY - Puppet freshness on sq66 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:31] <icinga-wm> RECOVERY - Puppet freshness on mw1120 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:32] <icinga-wm> RECOVERY - Puppet freshness on sq82 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:32] <icinga-wm> RECOVERY - Puppet freshness on mw1148 is OK: puppet ran at Wed Jul 31 14:24:29 UTC 2013
[14:24:37] <paravoid> mark: yes :)
[14:24:41] <icinga-wm> RECOVERY - Puppet freshness on cp1008 is OK: puppet ran at Wed Jul 31 14:24:39 UTC 2013
[14:24:41] <paravoid> but not just that
[14:24:41] <icinga-wm> RECOVERY - Puppet freshness on sq75 is OK: puppet ran at Wed Jul 31 14:24:39 UTC 2013
[14:24:41] <icinga-wm> RECOVERY - Puppet freshness on cp1009 is OK: puppet ran at Wed Jul 31 14:24:39 UTC 2013
[14:24:42] <icinga-wm> RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Wed Jul 31 14:24:39 UTC 2013
[14:24:42] <icinga-wm> RECOVERY - Puppet freshness on db1046 is OK: puppet ran at Wed Jul 31 14:24:39 UTC 2013
[14:24:46] <paravoid> tag_carrier is a misnomer
[14:24:46] <mark> but anyway, if it's https, there's no point in tagging carriers is there
[14:24:51] <paravoid> it also sets X-Subdomain: X
[14:24:51] <icinga-wm> RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Wed Jul 31 14:24:44 UTC 2013
[14:24:51] <icinga-wm> RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Wed Jul 31 14:24:44 UTC 2013
[14:24:51] <icinga-wm> RECOVERY - Puppet freshness on db64 is OK: puppet ran at Wed Jul 31 14:24:49 UTC 2013
[14:24:52] <icinga-wm> RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Wed Jul 31 14:24:49 UTC 2013
[14:24:52] <paravoid> er, M
[14:24:59] <paravoid> which isn't set anywhere else
[14:25:01] <icinga-wm> RECOVERY - Puppet freshness on db1015 is OK: puppet ran at Wed Jul 31 14:24:54 UTC 2013
[14:25:01] <icinga-wm> RECOVERY - Puppet freshness on ms-fe1004 is OK: puppet ran at Wed Jul 31 14:24:54 UTC 2013
[14:25:01] <icinga-wm> RECOVERY - Puppet freshness on db1049 is OK: puppet ran at Wed Jul 31 14:24:54 UTC 2013
[14:25:01] <icinga-wm> RECOVERY - Puppet freshness on srv274 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:01] <icinga-wm> RECOVERY - Puppet freshness on cp1064 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:02] <icinga-wm> RECOVERY - Puppet freshness on srv257 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:02] <icinga-wm> RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:03] <icinga-wm> RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:03] <icinga-wm> RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Wed Jul 31 14:24:59 UTC 2013
[14:25:09] <paravoid> so this basically breaks all https non-zero mobile
[14:25:11] <icinga-wm> RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Wed Jul 31 14:25:04 UTC 2013
[14:25:11] <icinga-wm> RECOVERY - Puppet freshness on mw48 is OK: puppet ran at Wed Jul 31 14:25:04 UTC 2013
[14:25:11] <icinga-wm> RECOVERY - Puppet freshness on mw1170 is OK: puppet ran at Wed Jul 31 14:25:09 UTC 2013
[14:25:11] <icinga-wm> RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Wed Jul 31 14:25:09 UTC 2013
[14:25:11] <icinga-wm> RECOVERY - Puppet freshness on mw1085 is OK: puppet ran at Wed Jul 31 14:25:09 UTC 2013
[14:25:21] <icinga-wm> RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:21] <icinga-wm> RECOVERY - Puppet freshness on mw1070 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:21] <icinga-wm> RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:21] <icinga-wm> RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:21] <icinga-wm> RECOVERY - Puppet freshness on mw1157 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:22] <icinga-wm> RECOVERY - Puppet freshness on mw1019 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:22] <icinga-wm> RECOVERY - Puppet freshness on mw1102 is OK: puppet ran at Wed Jul 31 14:25:14 UTC 2013
[14:25:22] <paravoid> } else {
[14:25:23] <paravoid> set req.http.X-Subdomain = "M";
[14:25:31] <icinga-wm> RECOVERY - Puppet freshness on sq56 is OK: puppet ran at Wed Jul 31 14:25:29 UTC 2013
[14:25:31] <icinga-wm> RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Wed Jul 31 14:25:29 UTC 2013
[14:25:31] <icinga-wm> RECOVERY - Puppet freshness on amssq41 is OK: puppet ran at Wed Jul 31 14:25:29 UTC 2013
[14:25:32] <paravoid> we need to move this outside and before of tag_carrier
[14:25:41] <icinga-wm> RECOVERY - Puppet freshness on ersch is OK: puppet ran at Wed Jul 31 14:25:34 UTC 2013
[14:25:41] <icinga-wm> RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Wed Jul 31 14:25:34 UTC 2013
[14:25:51] <paravoid> and let tag_carrier just override it on a match
[14:25:51] <icinga-wm> RECOVERY - Puppet freshness on db68 is OK: puppet ran at Wed Jul 31 14:25:44 UTC 2013
[14:25:52] <icinga-wm> RECOVERY - Puppet freshness on ms-be1005 is OK: puppet ran at Wed Jul 31 14:25:49 UTC 2013
[14:25:52] <icinga-wm> RECOVERY - Puppet freshness on db38 is OK: puppet ran at Wed Jul 31 14:25:49 UTC 2013
[14:25:58] <paravoid> don't you just love mobile VCLs
[14:26:01] <icinga-wm> RECOVERY - Puppet freshness on mw120 is OK: puppet ran at Wed Jul 31 14:25:54 UTC 2013
[14:26:01] <icinga-wm> RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Wed Jul 31 14:25:54 UTC 2013
[14:26:01] <icinga-wm> RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Wed Jul 31 14:25:54 UTC 2013
[14:26:01] <icinga-wm> RECOVERY - Puppet freshness on analytics1024 is OK: puppet ran at Wed Jul 31 14:25:54 UTC 2013
[14:26:01] <paravoid> shall I do that?
[14:26:01] <icinga-wm> RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:02] <icinga-wm> RECOVERY - Puppet freshness on wtp1014 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:02] <icinga-wm> RECOVERY - Puppet freshness on srv247 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:03] <icinga-wm> RECOVERY - Puppet freshness on srv239 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:03] <icinga-wm> RECOVERY - Puppet freshness on cp1057 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:04] <icinga-wm> RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:04] <icinga-wm> RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:05] <icinga-wm> RECOVERY - Puppet freshness on search20 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:05] <icinga-wm> RECOVERY - Puppet freshness on mw67 is OK: puppet ran at Wed Jul 31 14:25:59 UTC 2013
[14:26:11] <icinga-wm> RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:11] <icinga-wm> RECOVERY - Puppet freshness on mw1172 is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:11] <icinga-wm> RECOVERY - Puppet freshness on gallium is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:11] <icinga-wm> RECOVERY - Puppet freshness on mw1184 is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:11] <icinga-wm> RECOVERY - Puppet freshness on mw1110 is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:12] <icinga-wm> RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Wed Jul 31 14:26:04 UTC 2013
[14:26:12] <icinga-wm> RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Wed Jul 31 14:26:09 UTC 2013
[14:26:41] <icinga-wm> RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Wed Jul 31 14:26:34 UTC 2013
[14:26:41] <icinga-wm> RECOVERY - Puppet freshness on mw1191 is OK: puppet ran at Wed Jul 31 14:26:39 UTC 2013
[14:26:51] <icinga-wm> RECOVERY - Puppet freshness on tmh1002 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:51] <icinga-wm> RECOVERY - Puppet freshness on hooper is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:52] <icinga-wm> RECOVERY - Puppet freshness on sq57 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:52] <icinga-wm> RECOVERY - Puppet freshness on db1003 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:52] <icinga-wm> RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:52] <icinga-wm> RECOVERY - Puppet freshness on cp1019 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:52] <icinga-wm> RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:53] <icinga-wm> RECOVERY - Puppet freshness on db33 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:53] <icinga-wm> RECOVERY - Puppet freshness on wtp1005 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:54] <icinga-wm> RECOVERY - Puppet freshness on sq61 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:54] <icinga-wm> RECOVERY - Puppet freshness on ms-fe4 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:55] <icinga-wm> RECOVERY - Puppet freshness on searchidx2 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:55] <icinga-wm> RECOVERY - Puppet freshness on nfs2 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:56] <icinga-wm> RECOVERY - Puppet freshness on wtp1012 is OK: puppet ran at Wed Jul 31 14:26:44 UTC 2013
[14:26:56] <icinga-wm> RECOVERY - Puppet freshness on mw84 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:57] <icinga-wm> RECOVERY - Puppet freshness on srv254 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:57] <icinga-wm> RECOVERY - Puppet freshness on mw17 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:58] <icinga-wm> RECOVERY - Puppet freshness on mw69 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:58] <icinga-wm> RECOVERY - Puppet freshness on mw71 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:59] <icinga-wm> RECOVERY - Puppet freshness on es7 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:26:59] <icinga-wm> RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Wed Jul 31 14:26:49 UTC 2013
[14:27:01] <icinga-wm> RECOVERY - Puppet freshness on mw1195 is OK: puppet ran at Wed Jul 31 14:26:54 UTC 2013
[14:27:01] <icinga-wm> RECOVERY - Puppet freshness on mw1167 is OK: puppet ran at Wed Jul 31 14:26:59 UTC 2013
[14:27:01] <icinga-wm> RECOVERY - Puppet freshness on db37 is OK: puppet ran at Wed Jul 31 14:26:59 UTC 2013
[14:27:01] <icinga-wm> RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Wed Jul 31 14:26:59 UTC 2013
[14:27:02] <icinga-wm> RECOVERY - Puppet freshness on search26 is OK: puppet ran at Wed Jul 31 14:26:59 UTC 2013
[14:27:02] <icinga-wm> RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Wed Jul 31 14:26:59 UTC 2013
[14:27:11] <icinga-wm> RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Wed Jul 31 14:27:04 UTC 2013
[14:27:11] <icinga-wm> RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Wed Jul 31 14:27:09 UTC 2013
[14:27:11] <icinga-wm> RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Wed Jul 31 14:27:09 UTC 2013
[14:27:11] <icinga-wm> RECOVERY - Puppet freshness on mw1076 is OK: puppet ran at Wed Jul 31 14:27:09 UTC 2013
[14:27:21] <icinga-wm> RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Wed Jul 31 14:27:19 UTC 2013
[14:27:21] <icinga-wm> RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Wed Jul 31 14:27:19 UTC 2013
[14:27:31] <paravoid> fcking puppet freshness
[14:27:31] <icinga-wm> RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Wed Jul 31 14:27:25 UTC 2013
[14:27:31] <icinga-wm> RECOVERY - Puppet freshness on silver is OK: puppet ran at Wed Jul 31 14:27:25 UTC 2013
[14:27:31] <icinga-wm> RECOVERY - Puppet freshness on snapshot1001 is OK: puppet ran at Wed Jul 31 14:27:25 UTC 2013
[14:27:31] <icinga-wm> RECOVERY - Puppet freshness on mw63 is OK: puppet ran at Wed Jul 31 14:27:25 UTC 2013
[14:27:31] <icinga-wm> RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Wed Jul 31 14:27:30 UTC 2013
[14:27:32] <icinga-wm> RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Wed Jul 31 14:27:30 UTC 2013
[14:27:32] <icinga-wm> RECOVERY - Puppet freshness on nfs1 is OK: puppet ran at Wed Jul 31 14:27:30 UTC 2013
[14:27:33] <icinga-wm> RECOVERY - Puppet freshness on db50 is OK: puppet ran at Wed Jul 31 14:27:30 UTC 2013
[14:27:41] <icinga-wm> RECOVERY - Puppet freshness on mw80 is OK: puppet ran at Wed Jul 31 14:27:35 UTC 2013
[14:27:41] <icinga-wm> RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Wed Jul 31 14:27:35 UTC 2013
[14:27:51] <icinga-wm> RECOVERY - Puppet freshness on mw1055 is OK: puppet ran at Wed Jul 31 14:27:45 UTC 2013
[14:27:52] <icinga-wm> RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Wed Jul 31 14:27:50 UTC 2013
[14:28:01] <icinga-wm> RECOVERY - Puppet freshness on es3 is OK: puppet ran at Wed Jul 31 14:27:55 UTC 2013
[14:28:01] <icinga-wm> RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Wed Jul 31 14:27:55 UTC 2013
[14:28:01] <icinga-wm> RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Wed Jul 31 14:27:55 UTC 2013
[14:28:01] <icinga-wm> RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Wed Jul 31 14:28:00 UTC 2013
[14:28:01] <icinga-wm> RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Wed Jul 31 14:28:00 UTC 2013
[14:28:02] <icinga-wm> RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Wed Jul 31 14:28:00 UTC 2013
[14:28:02] <icinga-wm> RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Wed Jul 31 14:28:00 UTC 2013
[14:28:03] <icinga-wm> RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Wed Jul 31 14:28:00 UTC 2013
[14:28:11] <icinga-wm> RECOVERY - Puppet freshness on mw1216 is OK: puppet ran at Wed Jul 31 14:28:05 UTC 2013
[14:28:11] <icinga-wm> RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Wed Jul 31 14:28:05 UTC 2013
[14:28:11] <icinga-wm> RECOVERY - Puppet freshness on sq69 is OK: puppet ran at Wed Jul 31 14:28:05 UTC 2013
[14:28:11] <icinga-wm> RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Wed Jul 31 14:28:05 UTC 2013
[14:28:11] <icinga-wm> RECOVERY - Puppet freshness on mw1098 is OK: puppet ran at Wed Jul 31 14:28:05 UTC 2013
[14:28:12] <icinga-wm> RECOVERY - Puppet freshness on mc1014 is OK: puppet ran at Wed Jul 31 14:28:10 UTC 2013
[14:28:12] <icinga-wm> RECOVERY - Puppet freshness on mw1014 is OK: puppet ran at Wed Jul 31 14:28:10 UTC 2013
[14:28:13] <icinga-wm> RECOVERY - Puppet freshness on mw1133 is OK: puppet ran at Wed Jul 31 14:28:10 UTC 2013
[14:28:21] <icinga-wm> RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Wed Jul 31 14:28:15 UTC 2013
[14:28:21] <icinga-wm> RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Wed Jul 31 14:28:15 UTC 2013
[14:28:21] <icinga-wm> RECOVERY - Puppet freshness on search24 is OK: puppet ran at Wed Jul 31 14:28:15 UTC 2013
[14:28:21] <icinga-wm> RECOVERY - Puppet freshness on mw1168 is OK: puppet ran at Wed Jul 31 14:28:20 UTC 2013
[14:28:21] <icinga-wm> RECOVERY - Puppet freshness on mw1192 is OK: puppet ran at Wed Jul 31 14:28:20 UTC 2013
[14:28:22] <grrrit-wm> (�03PS3�) �10�Mark Bergsma��: Do not mark HTTPS requests with X-CS header [operations/puppet] - �10https://gerrit.wikimedia.org/r/76873� (owner: �10�Yurik��)
[14:28:24] <grrrit-wm> (�03PS18�) �10�Ottomata��: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - �10https://gerrit.wikimedia.org/r/76018�
[14:28:31] <icinga-wm> RECOVERY - Puppet freshness on mw1202 is OK: puppet ran at Wed Jul 31 14:28:25 UTC 2013
[14:28:31] <icinga-wm> RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Wed Jul 31 14:28:25 UTC 2013
[14:28:31] <icinga-wm> RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Wed Jul 31 14:28:25 UTC 2013
[14:28:31] <icinga-wm> RECOVERY - Puppet freshness on mw1079 is OK: puppet ran at Wed Jul 31 14:28:30 UTC 2013
[14:28:41] <icinga-wm> RECOVERY - Puppet freshness on db49 is OK: puppet ran at Wed Jul 31 14:28:35 UTC 2013
[14:28:41] <icinga-wm> RECOVERY - Puppet freshness on formey is OK: puppet ran at Wed Jul 31 14:28:40 UTC 2013
[14:28:41] <icinga-wm> RECOVERY - Puppet freshness on db1041 is OK: puppet ran at Wed Jul 31 14:28:40 UTC 2013
[14:28:51] <icinga-wm> RECOVERY - Puppet freshness on db74 is OK: puppet ran at Wed Jul 31 14:28:45 UTC 2013
[14:28:51] <icinga-wm> RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Wed Jul 31 14:28:45 UTC 2013
[14:28:51] <icinga-wm> RECOVERY - Puppet freshness on iron is OK: puppet ran at Wed Jul 31 14:28:45 UTC 2013
[14:28:51] <icinga-wm> RECOVERY - Puppet freshness on db72 is OK: puppet ran at Wed Jul 31 14:28:45 UTC 2013
[14:28:51] <icinga-wm> RECOVERY - Puppet freshness on rdb1001 is OK: puppet ran at Wed Jul 31 14:28:45 UTC 2013
[14:28:52] <icinga-wm> RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Wed Jul 31 14:28:50 UTC 2013
[14:28:52] <icinga-wm> RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Wed Jul 31 14:28:50 UTC 2013
[14:28:53] <icinga-wm> RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Wed Jul 31 14:28:50 UTC 2013
[14:28:53] <icinga-wm> RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Wed Jul 31 14:28:50 UTC 2013
[14:28:54] <icinga-wm> RECOVERY - Puppet freshness on oxygen is OK: puppet ran at Wed Jul 31 14:28:50 UTC 2013
[14:29:01] <icinga-wm> RECOVERY - Puppet freshness on db1016 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:01] <icinga-wm> RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:01] <icinga-wm> RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:01] <icinga-wm> RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:01] <icinga-wm> RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:02] <icinga-wm> RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Wed Jul 31 14:28:55 UTC 2013
[14:29:02] <icinga-wm> RECOVERY - Puppet freshness on mw97 is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:03] <icinga-wm> RECOVERY - Puppet freshness on mw4 is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:03] <icinga-wm> RECOVERY - Puppet freshness on mw72 is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:04] <icinga-wm> RECOVERY - Puppet freshness on search32 is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:04] <icinga-wm> RECOVERY - Puppet freshness on hooft is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:05] <icinga-wm> RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Wed Jul 31 14:29:00 UTC 2013
[14:29:11] <icinga-wm> RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Wed Jul 31 14:29:05 UTC 2013
[14:29:13] <grrrit-wm> (�03PS1�) �10�Hashar��: beta: send purges to both text and mobile caches [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76918�
[14:29:28] <grrrit-wm> (�03CR�) �10�Hashar��: [C: �03��2���] beta: send purges to both text and mobile caches [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76918� (owner: �10�Hashar��)
[14:29:31] <icinga-wm> RECOVERY - Puppet freshness on mw1183 is OK: puppet ran at Wed Jul 31 14:29:25 UTC 2013
[14:29:38] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: beta: send purges to both text and mobile caches [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76918� (owner: �10�Hashar��)
[14:29:41] <icinga-wm> RECOVERY - Puppet freshness on mw1034 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:41] <icinga-wm> RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:41] <icinga-wm> RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:41] <icinga-wm> RECOVERY - Puppet freshness on db1059 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:41] <icinga-wm> RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:42] <icinga-wm> RECOVERY - Puppet freshness on es8 is OK: puppet ran at Wed Jul 31 14:29:40 UTC 2013
[14:29:51] <icinga-wm> RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Wed Jul 31 14:29:45 UTC 2013
[14:29:52] <icinga-wm> RECOVERY - Puppet freshness on db1053 is OK: puppet ran at Wed Jul 31 14:29:45 UTC 2013
[14:30:01] <icinga-wm> RECOVERY - Puppet freshness on cp1055 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:01] <icinga-wm> RECOVERY - Puppet freshness on wtp1023 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:01] <icinga-wm> RECOVERY - Puppet freshness on amssq46 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:01] <icinga-wm> RECOVERY - Puppet freshness on srv241 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:01] <icinga-wm> RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:02] <icinga-wm> RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Wed Jul 31 14:29:51 UTC 2013
[14:30:02] <icinga-wm> RECOVERY - Puppet freshness on mw27 is OK: puppet ran at Wed Jul 31 14:29:56 UTC 2013
[14:30:03] <icinga-wm> RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Wed Jul 31 14:29:56 UTC 2013
[14:30:03] <icinga-wm> RECOVERY - Puppet freshness on mw44 is OK: puppet ran at Wed Jul 31 14:29:56 UTC 2013
[14:30:11] <icinga-wm> RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Wed Jul 31 14:30:01 UTC 2013
[14:30:11] <icinga-wm> RECOVERY - Puppet freshness on snapshot1004 is OK: puppet ran at Wed Jul 31 14:30:01 UTC 2013
[14:30:11] <icinga-wm> RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Wed Jul 31 14:30:01 UTC 2013
[14:30:11] <icinga-wm> RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Wed Jul 31 14:30:01 UTC 2013
[14:30:11] <icinga-wm> RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Wed Jul 31 14:30:06 UTC 2013
[14:30:12] <icinga-wm> RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Wed Jul 31 14:30:06 UTC 2013
[14:30:12] <icinga-wm> RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Wed Jul 31 14:30:06 UTC 2013
[14:30:13] <icinga-wm> RECOVERY - Puppet freshness on mw1096 is OK: puppet ran at Wed Jul 31 14:30:06 UTC 2013
[14:30:13] <icinga-wm> RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Wed Jul 31 14:30:06 UTC 2013
[14:30:15] <paravoid> mark: I actually had another similar idea
[14:30:21] <icinga-wm> RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Wed Jul 31 14:30:11 UTC 2013
[14:30:31] <icinga-wm> RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Wed Jul 31 14:30:21 UTC 2013
[14:30:31] <icinga-wm> RECOVERY - Puppet freshness on mw6 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:31] <icinga-wm> RECOVERY - Puppet freshness on mw1029 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:31] <icinga-wm> RECOVERY - Puppet freshness on mw1077 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:31] <icinga-wm> RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:32] <icinga-wm> RECOVERY - Puppet freshness on ms1001 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:32] <icinga-wm> RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Wed Jul 31 14:30:26 UTC 2013
[14:30:41] <paravoid> mark: vcl_recv { set req.X-Subdomain = "M"; tag_carrier; } and remove the else from tag_carrier
[14:30:41] <icinga-wm> RECOVERY - Puppet freshness on db47 is OK: puppet ran at Wed Jul 31 14:30:31 UTC 2013
[14:30:41] <icinga-wm> RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Wed Jul 31 14:30:31 UTC 2013
[14:30:41] <icinga-wm> RECOVERY - Puppet freshness on db1038 is OK: puppet ran at Wed Jul 31 14:30:36 UTC 2013
[14:30:41] <icinga-wm> RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Wed Jul 31 14:30:36 UTC 2013
[14:30:51] <icinga-wm> RECOVERY - Puppet freshness on mw1023 is OK: puppet ran at Wed Jul 31 14:30:41 UTC 2013
[14:30:51] <icinga-wm> RECOVERY - Puppet freshness on rdb1004 is OK: puppet ran at Wed Jul 31 14:30:41 UTC 2013
[14:30:51] <icinga-wm> RECOVERY - Puppet freshness on db62 is OK: puppet ran at Wed Jul 31 14:30:46 UTC 2013
[14:30:52] <icinga-wm> RECOVERY - Puppet freshness on amssq38 is OK: puppet ran at Wed Jul 31 14:30:46 UTC 2013
[14:30:52] <icinga-wm> RECOVERY - Puppet freshness on wtp1019 is OK: puppet ran at Wed Jul 31 14:30:46 UTC 2013
[14:30:58] <paravoid> (where tag_carrier can be your X-F-P guard)
[14:31:01] <icinga-wm> RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:01] <icinga-wm> RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:01] <icinga-wm> RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:01] <icinga-wm> RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:01] <icinga-wm> RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:02] <icinga-wm> RECOVERY - Puppet freshness on cp1059 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:02] <icinga-wm> RECOVERY - Puppet freshness on wtp1017 is OK: puppet ran at Wed Jul 31 14:30:51 UTC 2013
[14:31:03] <icinga-wm> RECOVERY - Puppet freshness on locke is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:03] <icinga-wm> RECOVERY - Puppet freshness on db35 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:04] <icinga-wm> RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:04] <icinga-wm> RECOVERY - Puppet freshness on mw89 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:05] <icinga-wm> RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:05] <icinga-wm> RECOVERY - Puppet freshness on cp1011 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:06] <icinga-wm> RECOVERY - Puppet freshness on ms-be6 is OK: puppet ran at Wed Jul 31 14:30:56 UTC 2013
[14:31:11] <icinga-wm> RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Wed Jul 31 14:31:01 UTC 2013
[14:31:11] <icinga-wm> RECOVERY - Puppet freshness on mw1209 is OK: puppet ran at Wed Jul 31 14:31:01 UTC 2013
[14:31:11] <icinga-wm> RECOVERY - Puppet freshness on mw1016 is OK: puppet ran at Wed Jul 31 14:31:06 UTC 2013
[14:31:11] <icinga-wm> RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Wed Jul 31 14:31:06 UTC 2013
[14:31:11] <icinga-wm> RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Wed Jul 31 14:31:06 UTC 2013
[14:31:12] <icinga-wm> RECOVERY - Puppet freshness on mw1152 is OK: puppet ran at Wed Jul 31 14:31:06 UTC 2013
[14:31:12] <icinga-wm> RECOVERY - Puppet freshness on mw21 is OK: puppet ran at Wed Jul 31 14:31:06 UTC 2013
[14:31:51] <icinga-wm> RECOVERY - Puppet freshness on mw1005 is OK: puppet ran at Wed Jul 31 14:31:46 UTC 2013
[14:31:51] <icinga-wm> RECOVERY - Puppet freshness on cp1052 is OK: puppet ran at Wed Jul 31 14:31:46 UTC 2013
[14:31:58] * paravoid just ignored .*Puppet freshness.*�
[14:32:01] <icinga-wm> RECOVERY - Puppet freshness on es1001 is OK: puppet ran at Wed Jul 31 14:31:51 UTC 2013
[14:32:01] <icinga-wm> RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Wed Jul 31 14:31:51 UTC 2013
[14:32:01] <icinga-wm> RECOVERY - Puppet freshness on wtp1009 is OK: puppet ran at Wed Jul 31 14:31:51 UTC 2013
[14:32:01] <icinga-wm> RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Wed Jul 31 14:31:51 UTC 2013
[14:32:01] <icinga-wm> RECOVERY - Puppet freshness on wtp1022 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:02] <icinga-wm> RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:02] <icinga-wm> RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:03] <icinga-wm> RECOVERY - Puppet freshness on carbon is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:03] <icinga-wm> RECOVERY - Puppet freshness on sq60 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:04] <icinga-wm> RECOVERY - Puppet freshness on zhen is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:04] <icinga-wm> RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:05] <icinga-wm> RECOVERY - Puppet freshness on db71 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:05] <icinga-wm> RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:06] <icinga-wm> RECOVERY - Puppet freshness on amssq42 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:06] <icinga-wm> RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:07] <icinga-wm> RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:07] <icinga-wm> RECOVERY - Puppet freshness on cp1040 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:08] <icinga-wm> RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:08] <icinga-wm> RECOVERY - Puppet freshness on cp1067 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:09] <icinga-wm> RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Wed Jul 31 14:31:52 UTC 2013
[14:32:09] <icinga-wm> RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Wed Jul 31 14:31:57 UTC 2013
[14:32:10] <icinga-wm> RECOVERY - Puppet freshness on cp1014 is OK: puppet ran at Wed Jul 31 14:31:57 UTC 2013
[14:32:10] <icinga-wm> RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Wed Jul 31 14:31:57 UTC 2013
[14:32:11] <icinga-wm> RECOVERY - Puppet freshness on srv236 is OK: puppet ran at Wed Jul 31 14:31:57 UTC 2013
[14:32:11] <icinga-wm> RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:12] <icinga-wm> RECOVERY - Puppet freshness on mw19 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:12] <icinga-wm> RECOVERY - Puppet freshness on mw24 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:13] <icinga-wm> RECOVERY - Puppet freshness on search34 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:13] <icinga-wm> RECOVERY - Puppet freshness on search15 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:14] <icinga-wm> RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:14] <icinga-wm> RECOVERY - Puppet freshness on mw1193 is OK: puppet ran at Wed Jul 31 14:32:02 UTC 2013
[14:32:15] <icinga-wm> RECOVERY - Puppet freshness on mw77 is OK: puppet ran at Wed Jul 31 14:32:07 UTC 2013
[14:32:15] <icinga-wm> RECOVERY - Puppet freshness on mw1012 is OK: puppet ran at Wed Jul 31 14:32:07 UTC 2013
[14:32:21] <icinga-wm> RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Wed Jul 31 14:32:12 UTC 2013
[14:32:21] <icinga-wm> RECOVERY - Puppet freshness on mw73 is OK: puppet ran at Wed Jul 31 14:32:12 UTC 2013
[14:32:21] <icinga-wm> RECOVERY - Puppet freshness on cp1070 is OK: puppet ran at Wed Jul 31 14:32:12 UTC 2013
[14:32:21] <icinga-wm> RECOVERY - Puppet freshness on ssl2 is OK: puppet ran at Wed Jul 31 14:32:17 UTC 2013
[14:32:21] <icinga-wm> RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Wed Jul 31 14:32:17 UTC 2013
[14:32:22] <icinga-wm> RECOVERY - Puppet freshness on yvon is OK: puppet ran at Wed Jul 31 14:32:17 UTC 2013
[14:32:31] <icinga-wm> RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Wed Jul 31 14:32:22 UTC 2013
[14:32:31] <icinga-wm> RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Wed Jul 31 14:32:22 UTC 2013
[14:32:31] <icinga-wm> RECOVERY - Puppet freshness on cp3008 is OK: puppet ran at Wed Jul 31 14:32:22 UTC 2013
[14:32:31] <icinga-wm> RECOVERY - Puppet freshness on tridge is OK: puppet ran at Wed Jul 31 14:32:22 UTC 2013
[14:32:31] <icinga-wm> RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Wed Jul 31 14:32:22 UTC 2013
[14:32:41] <icinga-wm> RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Wed Jul 31 14:32:32 UTC 2013
[14:32:41] <icinga-wm> RECOVERY - Puppet freshness on cp3004 is OK: puppet ran at Wed Jul 31 14:32:37 UTC 2013
[14:32:41] <icinga-wm> RECOVERY - Puppet freshness on mw11 is OK: puppet ran at Wed Jul 31 14:32:37 UTC 2013
[14:32:41] <icinga-wm> RECOVERY - Puppet freshness on cp3005 is OK: puppet ran at Wed Jul 31 14:32:37 UTC 2013
[14:32:51] <icinga-wm> RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Wed Jul 31 14:32:42 UTC 2013
[14:32:51] <icinga-wm> RECOVERY - Puppet freshness on xenon is OK: puppet ran at Wed Jul 31 14:32:42 UTC 2013
[14:32:51] <icinga-wm> RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Wed Jul 31 14:32:42 UTC 2013
[14:33:01] <icinga-wm> RECOVERY - Puppet freshness on eeden is OK: puppet ran at Wed Jul 31 14:32:52 UTC 2013
[14:33:01] <icinga-wm> RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Wed Jul 31 14:32:53 UTC 2013
[14:33:01] <icinga-wm> RECOVERY - Puppet freshness on analytics1025 is OK: puppet ran at Wed Jul 31 14:32:53 UTC 2013
[14:33:01] <icinga-wm> RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Wed Jul 31 14:32:53 UTC 2013
[14:33:01] <icinga-wm> RECOVERY - Puppet freshness on mw103 is OK: puppet ran at Wed Jul 31 14:32:53 UTC 2013
[14:33:02] <icinga-wm> RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Wed Jul 31 14:32:53 UTC 2013
[14:33:02] <icinga-wm> RECOVERY - Puppet freshness on search28 is OK: puppet ran at Wed Jul 31 14:32:58 UTC 2013
[14:33:03] <icinga-wm> RECOVERY - Puppet freshness on mw54 is OK: puppet ran at Wed Jul 31 14:32:58 UTC 2013
[14:33:03] <icinga-wm> RECOVERY - Puppet freshness on db1035 is OK: puppet ran at Wed Jul 31 14:32:58 UTC 2013
[14:33:11] <icinga-wm> RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Wed Jul 31 14:33:03 UTC 2013
[14:33:11] <icinga-wm> RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Wed Jul 31 14:33:08 UTC 2013
[14:33:11] <icinga-wm> RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Wed Jul 31 14:33:08 UTC 2013
[14:33:11] <icinga-wm> RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Wed Jul 31 14:33:08 UTC 2013
[14:33:21] <icinga-wm> RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Wed Jul 31 14:33:13 UTC 2013
[14:33:21] <icinga-wm> RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Wed Jul 31 14:33:13 UTC 2013
[14:33:21] <icinga-wm> RECOVERY - Puppet freshness on mw29 is OK: puppet ran at Wed Jul 31 14:33:13 UTC 2013
[14:33:21] <icinga-wm> RECOVERY - Puppet freshness on cp1047 is OK: puppet ran at Wed Jul 31 14:33:13 UTC 2013
[14:33:21] <icinga-wm> RECOVERY - Puppet freshness on srv295 is OK: puppet ran at Wed Jul 31 14:33:18 UTC 2013
[14:33:22] <icinga-wm> RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Wed Jul 31 14:33:18 UTC 2013
[14:33:31] <icinga-wm> RECOVERY - Puppet freshness on mw1187 is OK: puppet ran at Wed Jul 31 14:33:23 UTC 2013
[14:33:31] <icinga-wm> RECOVERY - Puppet freshness on mw1037 is OK: puppet ran at Wed Jul 31 14:33:23 UTC 2013
[14:33:31] <icinga-wm> RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Wed Jul 31 14:33:28 UTC 2013
[14:33:31] <icinga-wm> RECOVERY - Puppet freshness on mw1113 is OK: puppet ran at Wed Jul 31 14:33:28 UTC 2013
[14:33:41] <icinga-wm> RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Wed Jul 31 14:33:33 UTC 2013
[14:33:41] <icinga-wm> RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Wed Jul 31 14:33:39 UTC 2013
[14:33:41] <icinga-wm> RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Wed Jul 31 14:33:39 UTC 2013
[14:33:41] <icinga-wm> RECOVERY - Puppet freshness on db52 is OK: puppet ran at Wed Jul 31 14:33:39 UTC 2013
[14:33:51] <icinga-wm> RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Wed Jul 31 14:33:44 UTC 2013
[14:33:52] <icinga-wm> RECOVERY - Puppet freshness on mc1003 is OK: puppet ran at Wed Jul 31 14:33:49 UTC 2013
[14:33:52] <icinga-wm> RECOVERY - Puppet freshness on stafford is OK: puppet ran at Wed Jul 31 14:33:49 UTC 2013
[14:33:52] <icinga-wm> RECOVERY - Puppet freshness on neon is OK: puppet ran at Wed Jul 31 14:33:49 UTC 2013
[14:34:01] <icinga-wm> RECOVERY - Puppet freshness on srv287 is OK: puppet ran at Wed Jul 31 14:33:54 UTC 2013
[14:34:01] <icinga-wm> RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Wed Jul 31 14:33:54 UTC 2013
[14:34:01] <icinga-wm> RECOVERY - Puppet freshness on mw51 is OK: puppet ran at Wed Jul 31 14:33:54 UTC 2013
[14:34:01] <icinga-wm> RECOVERY - Puppet freshness on mw16 is OK: puppet ran at Wed Jul 31 14:33:54 UTC 2013
[14:34:01] <icinga-wm> RECOVERY - Puppet freshness on mw1176 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:02] <icinga-wm> RECOVERY - Puppet freshness on tmh2 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:02] <icinga-wm> RECOVERY - Puppet freshness on search13 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:03] <icinga-wm> RECOVERY - Puppet freshness on mw1164 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:03] <icinga-wm> RECOVERY - Puppet freshness on mw1217 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:04] <icinga-wm> RECOVERY - Puppet freshness on mw1018 is OK: puppet ran at Wed Jul 31 14:33:59 UTC 2013
[14:34:11] <icinga-wm> RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Wed Jul 31 14:34:04 UTC 2013
[14:34:11] <icinga-wm> RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Wed Jul 31 14:34:04 UTC 2013
[14:34:11] <icinga-wm> RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Wed Jul 31 14:34:04 UTC 2013
[14:34:11] <icinga-wm> RECOVERY - Puppet freshness on mw1117 is OK: puppet ran at Wed Jul 31 14:34:04 UTC 2013
[14:34:11] <icinga-wm> RECOVERY - Puppet freshness on mw1100 is OK: puppet ran at Wed Jul 31 14:34:09 UTC 2013
[14:34:12] <icinga-wm> RECOVERY - Puppet freshness on mw1099 is OK: puppet ran at Wed Jul 31 14:34:09 UTC 2013
[14:34:51] <icinga-wm> RECOVERY - Puppet freshness on sq71 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:51] <icinga-wm> RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:52] <icinga-wm> RECOVERY - Puppet freshness on labsdb1002 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:52] <icinga-wm> RECOVERY - Puppet freshness on labsdb1003 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:52] <icinga-wm> RECOVERY - Puppet freshness on es1 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:52] <icinga-wm> RECOVERY - Puppet freshness on cp1007 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:52] <icinga-wm> RECOVERY - Puppet freshness on fenari is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:53] <icinga-wm> RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:53] <icinga-wm> RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Wed Jul 31 14:34:44 UTC 2013
[14:34:54] <icinga-wm> RECOVERY - Puppet freshness on mw122 is OK: puppet ran at Wed Jul 31 14:34:49 UTC 2013
[14:35:01] <icinga-wm> RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:01] <icinga-wm> RECOVERY - Puppet freshness on db1054 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:01] <icinga-wm> RECOVERY - Puppet freshness on srv238 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:01] <icinga-wm> RECOVERY - Puppet freshness on mw82 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:01] <icinga-wm> RECOVERY - Puppet freshness on db40 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:02] <icinga-wm> RECOVERY - Puppet freshness on mw66 is OK: puppet ran at Wed Jul 31 14:34:54 UTC 2013
[14:35:02] <icinga-wm> RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:03] <icinga-wm> RECOVERY - Puppet freshness on mw34 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:03] <icinga-wm> RECOVERY - Puppet freshness on ms-be8 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:04] <icinga-wm> RECOVERY - Puppet freshness on mw20 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:04] <icinga-wm> RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:05] <icinga-wm> RECOVERY - Puppet freshness on search17 is OK: puppet ran at Wed Jul 31 14:34:59 UTC 2013
[14:35:11] <icinga-wm> RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:11] <icinga-wm> RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:11] <icinga-wm> RECOVERY - Puppet freshness on cp3007 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:11] <icinga-wm> RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:11] <icinga-wm> RECOVERY - Puppet freshness on mw1101 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:12] <icinga-wm> RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:12] <icinga-wm> RECOVERY - Puppet freshness on mw1123 is OK: puppet ran at Wed Jul 31 14:35:04 UTC 2013
[14:35:13] <icinga-wm> RECOVERY - Puppet freshness on cp1056 is OK: puppet ran at Wed Jul 31 14:35:09 UTC 2013
[14:35:24] <icinga-wm> RECOVERY - Puppet freshness on mw1042 is OK: puppet ran at Wed Jul 31 14:35:14 UTC 2013
[14:35:24] <icinga-wm> RECOVERY - Puppet freshness on mw1017 is OK: puppet ran at Wed Jul 31 14:35:19 UTC 2013
[14:35:34] <icinga-wm> RECOVERY - Puppet freshness on mw1065 is OK: puppet ran at Wed Jul 31 14:35:24 UTC 2013
[14:35:34] <icinga-wm> RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Wed Jul 31 14:35:24 UTC 2013
[14:35:35] <icinga-wm> RECOVERY - Puppet freshness on mw1095 is OK: puppet ran at Wed Jul 31 14:35:29 UTC 2013
[14:35:44] <icinga-wm> RECOVERY - Puppet freshness on cp1012 is OK: puppet ran at Wed Jul 31 14:35:34 UTC 2013
[14:35:44] <icinga-wm> RECOVERY - Puppet freshness on db1023 is OK: puppet ran at Wed Jul 31 14:35:39 UTC 2013
[14:35:54] <icinga-wm> RECOVERY - Puppet freshness on williams is OK: puppet ran at Wed Jul 31 14:35:49 UTC 2013
[14:35:54] <icinga-wm> RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Wed Jul 31 14:35:49 UTC 2013
[14:36:04] <icinga-wm> RECOVERY - Puppet freshness on srv286 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:04] <icinga-wm> RECOVERY - Puppet freshness on es1003 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:04] <icinga-wm> RECOVERY - Puppet freshness on srv258 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:04] <icinga-wm> RECOVERY - Puppet freshness on mw14 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:04] <icinga-wm> RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:05] <icinga-wm> RECOVERY - Puppet freshness on es1005 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:05] <icinga-wm> RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:06] <icinga-wm> RECOVERY - Puppet freshness on db1019 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:06] <icinga-wm> RECOVERY - Puppet freshness on cp1020 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:07] <icinga-wm> RECOVERY - Puppet freshness on mw62 is OK: puppet ran at Wed Jul 31 14:35:54 UTC 2013
[14:36:07] <icinga-wm> RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Wed Jul 31 14:35:59 UTC 2013
[14:36:08] <icinga-wm> RECOVERY - Puppet freshness on search22 is OK: puppet ran at Wed Jul 31 14:35:59 UTC 2013
[14:36:08] <icinga-wm> RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Wed Jul 31 14:35:59 UTC 2013
[14:36:14] <icinga-wm> RECOVERY - Puppet freshness on ms-be1 is OK: puppet ran at Wed Jul 31 14:36:04 UTC 2013
[14:36:14] <icinga-wm> RECOVERY - Puppet freshness on mw1214 is OK: puppet ran at Wed Jul 31 14:36:04 UTC 2013
[14:36:14] <icinga-wm> RECOVERY - Puppet freshness on mw1127 is OK: puppet ran at Wed Jul 31 14:36:04 UTC 2013
[14:36:14] <icinga-wm> RECOVERY - Puppet freshness on mw125 is OK: puppet ran at Wed Jul 31 14:36:04 UTC 2013
[14:36:24] <icinga-wm> RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Wed Jul 31 14:36:14 UTC 2013
[14:36:24] <icinga-wm> RECOVERY - Puppet freshness on dobson is OK: puppet ran at Wed Jul 31 14:36:19 UTC 2013
[14:36:34] <icinga-wm> RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Wed Jul 31 14:36:29 UTC 2013
[14:36:44] <icinga-wm> RECOVERY - Puppet freshness on mw9 is OK: puppet ran at Wed Jul 31 14:36:34 UTC 2013
[14:36:44] <icinga-wm> RECOVERY - Puppet freshness on mw47 is OK: puppet ran at Wed Jul 31 14:36:34 UTC 2013
[14:36:44] <icinga-wm> RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Wed Jul 31 14:36:34 UTC 2013
[14:36:44] <icinga-wm> RECOVERY - Puppet freshness on srv284 is OK: puppet ran at Wed Jul 31 14:36:39 UTC 2013
[14:36:44] <icinga-wm> RECOVERY - Puppet freshness on gurvin is OK: puppet ran at Wed Jul 31 14:36:39 UTC 2013
[14:36:45] <icinga-wm> RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Wed Jul 31 14:36:39 UTC 2013
[14:36:45] <icinga-wm> RECOVERY - Puppet freshness on mw132 is OK: puppet ran at Wed Jul 31 14:36:39 UTC 2013
[14:36:46] <icinga-wm> RECOVERY - Puppet freshness on db53 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:46] <icinga-wm> RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:47] <icinga-wm> RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:47] <icinga-wm> RECOVERY - Puppet freshness on nescio is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:48] <icinga-wm> RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:48] <icinga-wm> RECOVERY - Puppet freshness on sq45 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:49] <icinga-wm> RECOVERY - Puppet freshness on es10 is OK: puppet ran at Wed Jul 31 14:36:40 UTC 2013
[14:36:54] <icinga-wm> RECOVERY - Puppet freshness on db1052 is OK: puppet ran at Wed Jul 31 14:36:45 UTC 2013
[14:36:54] <icinga-wm> RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Wed Jul 31 14:36:45 UTC 2013
[14:36:54] <icinga-wm> RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:54] <icinga-wm> RECOVERY - Puppet freshness on cp1003 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:54] <icinga-wm> RECOVERY - Puppet freshness on cp1051 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:55] <icinga-wm> RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:55] <icinga-wm> RECOVERY - Puppet freshness on cp1069 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:56] <icinga-wm> RECOVERY - Puppet freshness on search21 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:56] <icinga-wm> RECOVERY - Puppet freshness on mw90 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:57] <icinga-wm> RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:36:57] <icinga-wm> RECOVERY - Puppet freshness on es1002 is OK: puppet ran at Wed Jul 31 14:36:50 UTC 2013
[14:37:04] <icinga-wm> RECOVERY - Puppet freshness on mw1196 is OK: puppet ran at Wed Jul 31 14:36:55 UTC 2013
[14:37:04] <icinga-wm> RECOVERY - Puppet freshness on search14 is OK: puppet ran at Wed Jul 31 14:36:55 UTC 2013
[14:37:04] <icinga-wm> RECOVERY - Puppet freshness on mw1162 is OK: puppet ran at Wed Jul 31 14:36:55 UTC 2013
[14:37:04] <icinga-wm> RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Wed Jul 31 14:37:00 UTC 2013
[14:37:04] <icinga-wm> RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Wed Jul 31 14:37:00 UTC 2013
[14:37:05] <icinga-wm> RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Wed Jul 31 14:37:00 UTC 2013
[14:37:05] <icinga-wm> RECOVERY - Puppet freshness on ms5 is OK: puppet ran at Wed Jul 31 14:37:00 UTC 2013
[14:37:06] <icinga-wm> RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Wed Jul 31 14:37:00 UTC 2013
[14:37:14] <icinga-wm> RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Wed Jul 31 14:37:05 UTC 2013
[14:37:24] <icinga-wm> RECOVERY - Puppet freshness on mw5 is OK: puppet ran at Wed Jul 31 14:37:15 UTC 2013
[14:37:24] <icinga-wm> RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Wed Jul 31 14:37:20 UTC 2013
[14:37:34] <icinga-wm> RECOVERY - Puppet freshness on zirconium is OK: puppet ran at Wed Jul 31 14:37:25 UTC 2013
[14:37:34] <icinga-wm> RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Wed Jul 31 14:37:25 UTC 2013
[14:37:34] <icinga-wm> RECOVERY - Puppet freshness on mw1044 is OK: puppet ran at Wed Jul 31 14:37:30 UTC 2013
[14:37:34] <icinga-wm> RECOVERY - Puppet freshness on db1048 is OK: puppet ran at Wed Jul 31 14:37:30 UTC 2013
[14:37:44] <icinga-wm> RECOVERY - Puppet freshness on cp1006 is OK: puppet ran at Wed Jul 31 14:37:35 UTC 2013
[14:37:54] <icinga-wm> RECOVERY - Puppet freshness on mw118 is OK: puppet ran at Wed Jul 31 14:37:45 UTC 2013
[14:37:54] <icinga-wm> RECOVERY - Puppet freshness on tin is OK: puppet ran at Wed Jul 31 14:37:50 UTC 2013
[14:37:54] <icinga-wm> RECOVERY - Puppet freshness on cp1046 is OK: puppet ran at Wed Jul 31 14:37:50 UTC 2013
[14:37:54] <icinga-wm> RECOVERY - Puppet freshness on cp1050 is OK: puppet ran at Wed Jul 31 14:37:50 UTC 2013
[14:38:04] <icinga-wm> RECOVERY - Puppet freshness on mw119 is OK: puppet ran at Wed Jul 31 14:37:55 UTC 2013
[14:38:04] <icinga-wm> RECOVERY - Puppet freshness on srv301 is OK: puppet ran at Wed Jul 31 14:37:55 UTC 2013
[14:38:04] <icinga-wm> RECOVERY - Puppet freshness on ms-be4 is OK: puppet ran at Wed Jul 31 14:38:00 UTC 2013
[14:38:04] <icinga-wm> RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Wed Jul 31 14:38:00 UTC 2013
[14:38:04] <icinga-wm> RECOVERY - Puppet freshness on mw1130 is OK: puppet ran at Wed Jul 31 14:38:00 UTC 2013
[14:38:05] <icinga-wm> RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Wed Jul 31 14:38:00 UTC 2013
[14:38:14] <icinga-wm> RECOVERY - Puppet freshness on mw1111 is OK: puppet ran at Wed Jul 31 14:38:05 UTC 2013
[14:38:14] <icinga-wm> RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Wed Jul 31 14:38:05 UTC 2013
[14:38:14] <icinga-wm> RECOVERY - Puppet freshness on mc1005 is OK: puppet ran at Wed Jul 31 14:38:05 UTC 2013
[14:38:14] <icinga-wm> RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Wed Jul 31 14:38:05 UTC 2013
[14:38:44] <icinga-wm> RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Wed Jul 31 14:38:40 UTC 2013
[14:38:44] <icinga-wm> RECOVERY - Puppet freshness on mw38 is OK: puppet ran at Wed Jul 31 14:38:40 UTC 2013
[14:38:44] <icinga-wm> RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Wed Jul 31 14:38:40 UTC 2013
[14:38:44] <icinga-wm> RECOVERY - Puppet freshness on search36 is OK: puppet ran at Wed Jul 31 14:38:40 UTC 2013
[14:38:54] <icinga-wm> RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:54] <icinga-wm> RECOVERY - Puppet freshness on mw1161 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:54] <icinga-wm> RECOVERY - Puppet freshness on sq51 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:54] <icinga-wm> RECOVERY - Puppet freshness on rubidium is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:54] <icinga-wm> RECOVERY - Puppet freshness on es4 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:55] <icinga-wm> RECOVERY - Puppet freshness on mw1124 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:55] <icinga-wm> RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:56] <icinga-wm> RECOVERY - Puppet freshness on db45 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:56] <icinga-wm> RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:57] <icinga-wm> RECOVERY - Puppet freshness on db43 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:57] <icinga-wm> RECOVERY - Puppet freshness on amssq44 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:58] <icinga-wm> RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:58] <icinga-wm> RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Wed Jul 31 14:38:45 UTC 2013
[14:38:59] <icinga-wm> RECOVERY - Puppet freshness on mc1009 is OK: puppet ran at Wed Jul 31 14:38:50 UTC 2013
[14:38:59] <icinga-wm> RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Wed Jul 31 14:38:50 UTC 2013
[14:39:00] <icinga-wm> RECOVERY - Puppet freshness on ms-be2 is OK: puppet ran at Wed Jul 31 14:38:50 UTC 2013
[14:39:00] <icinga-wm> RECOVERY - Puppet freshness on wtp1011 is OK: puppet ran at Wed Jul 31 14:38:50 UTC 2013
[14:39:04] <icinga-wm> RECOVERY - Puppet freshness on wtp1018 is OK: puppet ran at Wed Jul 31 14:38:55 UTC 2013
[14:39:04] <icinga-wm> RECOVERY - Puppet freshness on srv253 is OK: puppet ran at Wed Jul 31 14:38:55 UTC 2013
[14:39:04] <icinga-wm> RECOVERY - Puppet freshness on vanadium is OK: puppet ran at Wed Jul 31 14:38:55 UTC 2013
[14:39:04] <icinga-wm> RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Wed Jul 31 14:38:55 UTC 2013
[14:39:04] <icinga-wm> RECOVERY - Puppet freshness on analytics1023 is OK: puppet ran at Wed Jul 31 14:38:55 UTC 2013
[14:39:05] <icinga-wm> RECOVERY - Puppet freshness on srv245 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:05] <icinga-wm> RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:06] <icinga-wm> RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:06] <icinga-wm> RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:07] <icinga-wm> RECOVERY - Puppet freshness on mw30 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:07] <icinga-wm> RECOVERY - Puppet freshness on cp1063 is OK: puppet ran at Wed Jul 31 14:39:00 UTC 2013
[14:39:14] <icinga-wm> RECOVERY - Puppet freshness on db1020 is OK: puppet ran at Wed Jul 31 14:39:10 UTC 2013
[14:39:14] <icinga-wm> RECOVERY - Puppet freshness on ms-fe3001 is OK: puppet ran at Wed Jul 31 14:39:10 UTC 2013
[14:39:14] <icinga-wm> RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Wed Jul 31 14:39:10 UTC 2013
[14:39:44] <icinga-wm> RECOVERY - Puppet freshness on wtp1024 is OK: puppet ran at Wed Jul 31 14:39:40 UTC 2013
[14:40:44] <icinga-wm> RECOVERY - Puppet freshness on lvs5 is OK: puppet ran at Wed Jul 31 14:40:36 UTC 2013
[14:42:14] <icinga-wm> RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Wed Jul 31 14:42:06 UTC 2013
[14:43:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:44:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time
[14:52:11] <grrrit-wm> (�03PS6�) �10�Hashar��: beta: all caches in $wgSquidServersNoPurge [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/71932� (owner: �10�Alex Monk��)
[14:52:24] <icinga-wm> PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours
[14:52:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:53:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[14:54:29] <grrrit-wm> (�03CR�) �10�Hashar��: [C: �03��2���] "rebased, rewrote summary message." [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/71932� (owner: �10�Alex Monk��)
[14:54:37] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: beta: all caches in $wgSquidServersNoPurge [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/71932� (owner: �10�Alex Monk��)
[14:58:04] <icinga-wm> RECOVERY - Puppet freshness on mw18 is OK: puppet ran at Wed Jul 31 14:57:55 UTC 2013
[15:05:14] <grrrit-wm> (�03PS5�) �10�Hashar��: replicate Gerrit repos to Jenkins slave lanthanum [operations/puppet] - �10https://gerrit.wikimedia.org/r/75499�
[15:07:07] <grrrit-wm> (�03CR�) �10�Hashar��: "This change can land on Gerrit. Lanthanum.eqiad.wmnet has been installed with the role::gerrit::replicationdest class." [operations/puppet] - �10https://gerrit.wikimedia.org/r/75499� (owner: �10�Hashar��)
[15:15:48] <grrrit-wm> (�03CR�) �10�Hashar��: [V: �03��2���] "Ran in production successfully." [operations/puppet] - �10https://gerrit.wikimedia.org/r/55059� (owner: �10�Hashar��)
[15:17:57] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: [C: �03��2���] define ishmael_config() parameters lacked dollar signs [operations/puppet] - �10https://gerrit.wikimedia.org/r/76280� (owner: �10�Hashar��)
[15:19:37] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: [C: �03��2���] style for ipv6relay.pp [operations/puppet] - �10https://gerrit.wikimedia.org/r/76285� (owner: �10�Hashar��)
[15:21:25] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:21:37] <andrewbogott> mark, could you have a look at https://gerrit.wikimedia.org/r/#/c/75347/ ? (since you volunteered :) )
[15:21:57] <hashar> andrewbogott: while you are in Gerrit could you land https://gerrit.wikimedia.org/r/#/c/55059 which adapt foreachwiki (a mw tool)
[15:22:15] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.730 second response time
[15:24:07] <andrewbogott> hashar, is there a change to MWRealm.sh elsewhere? This whole getRealmSpecificFileName bit seems kind of magical :)
[15:24:15] <hashar> ah
[15:24:19] <hashar> that sudo as root and gives us a shell
[15:24:25] <hashar> let me find it
[15:24:44] <mark> andrewbogott: I think a copy and paste would just work fine indeed
[15:25:12] <andrewbogott> mark, great. d'you think I should make one big patch that adds that module and cuts things over, or split it into two steps?
[15:25:27] <hashar> andrewbogott: https://git.wikimedia.org/blob/operations%2Fmediawiki-config.git/51f1d582b1444ad2b76cd9e15d6153d2e462ab76/multiversion%2FMWRealm.sh
[15:25:35] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:25:46] <mark> andrewbogott: there shouldn't be any name conflicts, so why not do it in steps?
[15:25:56] <mark> less risky, and I don't see the downsides :)
[15:25:56] <andrewbogott> 'k
[15:26:12] <mark> but
[15:26:23] <andrewbogott> Yeah, as long as the two patches go in at the same time there's no risk of drift.
[15:26:24] <mark> i noticed that the classes/definitions are not always named after the file names
[15:26:25] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[15:26:27] <mark> so the autoloader won't work?
[15:27:05] <icinga-wm> RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Wed Jul 31 15:27:02 UTC 2013
[15:27:07] <andrewbogott> mark, which one?
[15:27:46] <mark> e.g. in aggregate.pp
[15:28:15] <mark> it has ::aggregate_member and ::bonding-tools
[15:28:32] <andrewbogott> oh, I believe that aggregate_member and bonding-tools are only referenced by the aggregate class.
[15:28:41] <andrewbogott> So I regard them as subfunctions, no need to expose to the loader.
[15:28:46] <mark> oh ok
[15:29:29] <mark> yeah then it'll probably work
[15:30:00] <mark> i'm not so worried about drift
[15:30:10] <andrewbogott> hashar, that script has so many levels of substitution that my head is spinning, but I'm happy to merge if you promise that it'll work :)
[15:30:13] <mark> but because these things can potentially take servers offline
[15:30:14] <mark> let's be careful
[15:30:25] <andrewbogott> ashar, maybe add some comments explaning what's going on?
[15:30:28] <andrewbogott> *hashar
[15:30:30] <mark> we can put a big warning comment near the old definitions
[15:30:41] <mark> and then test the new module class by class
[15:30:45] <mark> by migrating some things over
[15:30:57] <mark> i can do the majority of those in the next week or two if you want
[15:31:05] <andrewbogott> yeah, that sounds like a good plan. OK, I'll revise that first patch to include lots of DANGER comments.
[15:31:07] <grrrit-wm> (�03CR�) �10�Hashar��: [C: �03��2���] set some paths to use $wmfHostnames 'bits' and 'test' [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/71774� (owner: �10�Hashar��)
[15:31:07] <andrewbogott> cool.
[15:31:08] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: set some paths to use $wmfHostnames 'bits' and 'test' [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/71774� (owner: �10�Hashar��)
[15:32:19] <hashar> andrewbogott: it will work. Got reviewed by original author and I did test it in both beta and prod :-]
[15:34:15] <icinga-wm> RECOVERY - Puppet freshness on mw1103 is OK: puppet ran at Wed Jul 31 15:34:06 UTC 2013
[15:34:22] <mark> i need to finish that ganglia migration too
[15:36:54] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03�1��] Create an 'interface' module. [operations/puppet] - �10https://gerrit.wikimedia.org/r/75347� (owner: �10�Andrew Bogott��)
[15:36:57] <andrewbogott> hashar: I still want comments, at least one that says "This will return either X or Y depending" above the ALL_DBLIST= line
[15:37:40] <hashar> andrewbogott: will amend :-]
[15:38:08] <bd808> can anyone point me to a git repo that contains a script that ops would use to monitor something via http calls? Looking for a starting place for writing a POC tool that will check squids for cache age
[15:38:33] <bd808> See https://www.mediawiki.org/wiki/Multimedia/Cache_Invalidation_Misses#Recent_Change_Monitoring for some context
[15:39:39] <grrrit-wm> (�03PS2�) �10�Andrew Bogott��: Create an 'interface' module. [operations/puppet] - �10https://gerrit.wikimedia.org/r/75347�
[15:40:16] <icinga-wm> RECOVERY - Puppet freshness on lanthanum is OK: puppet ran at Wed Jul 31 15:40:14 UTC 2013
[15:40:40] <logmsgbot> !log hashar synchronized wmf-config '{{gerrit|71774}} set some paths to use 'bits' and 'test''
[15:40:52] <morebots> Logged the message, Master
[15:43:36] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:46:23] <grrrit-wm> (�03PS9�) �10�Hashar��: adapt `foreachwiki` for labs [operations/puppet] - �10https://gerrit.wikimedia.org/r/55059�
[15:46:25] <grrrit-wm> (�03CR�) �10�Hashar��: "comments for andrewbogott describing the result of the cryptic getRealmSpecificFilename() :D" [operations/puppet] - �10https://gerrit.wikimedia.org/r/55059� (owner: �10�Hashar��)
[15:46:26] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: [C: �03��2���] "Great, thank you for adding explanation!" [operations/puppet] - �10https://gerrit.wikimedia.org/r/55059� (owner: �10�Hashar��)
[15:46:27] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[15:46:52] <andrewbogott> mark, I'm going to merge that module, then I'll leave it to you to cut things over. If you reach a point where you're confident that it's working and there's more busywork left to do feel free to dump that back on me.
[15:47:09] <mark> awesome, thanks
[15:47:43] <grrrit-wm> (�03PS1�) �10�Mark Bergsma��: Remove obsolete comment [operations/puppet] - �10https://gerrit.wikimedia.org/r/76931�
[15:49:19] <grrrit-wm> (�03PS2�) �10�Mark Bergsma��: Remove obsolete comment [operations/puppet] - �10https://gerrit.wikimedia.org/r/76931�
[15:49:20] <grrrit-wm> (�03PS4�) �10�Mark Bergsma��: Do not mark HTTPS requests with X-CS header [operations/puppet] - �10https://gerrit.wikimedia.org/r/76873� (owner: �10�Yurik��)
[15:50:04] <grrrit-wm> (�03CR�) �10�Mark Bergsma��: [C: �03��2���] Remove obsolete comment [operations/puppet] - �10https://gerrit.wikimedia.org/r/76931� (owner: �10�Mark Bergsma��)
[15:52:09] <yuri_k> mark, i don't think we want to treat all https traffic as M subdomain for zero
[15:52:50] <yuri_k> lets try to keep it as it was before your patch - by unsetting X-CS for HTTPS
[15:53:23] <yuri_k> this way users will get red "not supported for your carrier" banner
[15:56:26] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:59:29] <YuviPanda> MaxSem: grrrit-wm is as fast as toollabs :)
[15:59:41] <MaxSem> RIP
[15:59:52] <YuviPanda> MaxSem: it only touches the disk for logging, so I guess that is what causes some of the delays
[16:00:44] <MaxSem> YuviPanda, if logging is async (as it should be in node), NFS problems shouldn't cause a lag
[16:00:57] <YuviPanda> MaxSem: true, but I haven't checked if winston is async or not
[16:01:14] <YuviPanda> MaxSem: also the 'gerrit-to-redis' service is in python, not node. so logging *there* might also cause things to be slow
[16:01:26] <hashar> pfff
[16:01:32] <hashar> like python is slower than node ? :D
[16:01:45] <MaxSem> dude
[16:01:53] <MaxSem> python is 27 years old
[16:02:21] <YuviPanda> who is slower, a 27 year old person or a 6 year old?
[16:02:24] <YuviPanda> TAKE THAT, LOGIC!
[16:02:24] <MaxSem> which means that node is 27/4 times more awesome
[16:02:26] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.891 second response time
[16:03:37] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: [C: �03��2���] Create an 'interface' module. [operations/puppet] - �10https://gerrit.wikimedia.org/r/75347� (owner: �10�Andrew Bogott��)
[16:05:37] <Reedy> YuviPanda: Usain Bolt is 26...
[16:06:12] <YuviPanda> my point exactly :P
[16:06:26] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:07:26] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.303 second response time
[16:19:34] <grrrit-wm> (�03CR�) �10�Yurik��: [C: �04�-1��] "Mark, I don't think your change is good - with my approach, we can easily add an ability to conditionally allow a few carriers that suppor" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76873� (owner: �10�Yurik��)
[16:21:27] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:22:17] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time
[16:27:17] <icinga-wm> PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours
[16:29:59] <grrrit-wm> (�03CR�) �10�Demon��: [C: �03�1��] replicate Gerrit repos to Jenkins slave lanthanum [operations/puppet] - �10https://gerrit.wikimedia.org/r/75499� (owner: �10�Hashar��)
[16:44:29] <grrrit-wm> (�03CR�) �10�Demon��: [C: �03�1��] replicate Gerrit repos to Jenkins slave gallium [operations/puppet] - �10https://gerrit.wikimedia.org/r/75500� (owner: �10�Hashar��)
[16:47:05] <^d> manybubbles1: Sooo, on those improved reindex times. I profiled it once, and with such a high cache hit rate the pcache hits + memcached requests end up taking the bulk of the time (as expected). Everything else is minimal by comparison.
[16:47:32] <^d> So, I think that means our approach is sound :)
[16:47:36] <manybubbles1> ^d: sounds great.
[16:47:55] <manybubbles1> ^d: one option would be to make a mothership that fires off multiple processes so we can pretend we have threads
[16:48:13] <manybubbles1> but other than that I don't think it'd be worth improving for now
[16:48:45] <^d> Yeah, and frankly I don't know if it's needed right now.
[16:48:51] <^d> We can always go back later and bolt that on
[16:50:18] <manybubbles> ^d: yeah
[16:50:34] <manybubbles> and what with the ability to reindex solely with elasticsearch it might not come up that frequently
[16:50:45] <manybubbles> so I've been digging through the bugs I filed.
[16:53:56] <manybubbles> I might just bang away at beta for a while longer and see if I can find anything else weird. Or turn it on for all languages and wait for folks to complain
[16:56:25] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:57:15] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time
[17:09:12] <grrrit-wm> (�03CR�) �10�Ori.livneh��: "I haven't found Puppet spec tests to be especially useful -- they're primarily handy for ensuring that a module that targets multiple oper" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76059� (owner: �10�Ori.livneh��)
[17:09:42] <grrrit-wm> (�03CR�) �10�Faidon��: [C: �03�1��] "I on the other hand I'm actually quite in favor. This seems like a reasonable change to me." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76059� (owner: �10�Ori.livneh��)
[17:17:15] <notpeter> manybubbles: hey, just wanted to check in. are you waiting on anything from me right now?
[17:17:23] <notpeter> I'm pretty scattered, so wanted to ask proactively :)
[17:17:30] <ori-l> bd808: hi, welcome! we use ganglia quite a lot for things like that. each host on the cluster runs a daemon, gmond, that spawns whatever metric-collecting modules are defined in /etc/ganglia/gmond.conf (and conf.d/*), polls these modules for metric data, and forwards them to a central collection point
[17:17:45] <manybubbles> notpeter: I was waiting on a machine to test with the big dataset
[17:17:58] <notpeter> oh man
[17:18:04] <ori-l> bd808: you can write metric modules in C and Python. the Python ones are really easy to right. you just have to implement three functions or so. the script can be multithreaded.
[17:18:13] <notpeter> I'm really sorry for not communicating
[17:18:13] <ori-l> *to write
[17:18:15] <notpeter> you have 3
[17:18:30] <manybubbles> great! how do I connect to them!
[17:18:36] <notpeter> mw131-134
[17:18:43] <notpeter> mw131 is an app server
[17:18:50] <notpeter> mw132-134 have nothing
[17:18:53] <notpeter> you have sudo on all 3
[17:19:00] <notpeter> ssh first to a bastion host
[17:19:02] <notpeter> like bast1001
[17:19:05] <notpeter> and then ssh to them from there
[17:19:13] <notpeter> er, sudo on all 4
[17:19:28] <notpeter> I just realized that I have to straighten out mw131
[17:19:42] <notpeter> but, for ES stuff, you can start with mw132-mw134
[17:20:13] <ori-l> bd808: if you search for *.py files in the Puppet repo, many (perhaps most) of the results will be metric modules, so you can study those. there's also a whole collection of them here: https://github.com/ganglia/gmond_python_modules
[17:25:44] <bd808> ori-l: thanks! I'll start looking in puppet and see how far I can get.
[17:27:40] <ori-l> bd808: probably easiest to just implement the metric collection without regard for ganglia at first; it's easy to add after the fact. if you get to the point where you can print to stdout some average staleness figure every N seconds then I or someone else could show you how to turn it into a collection module pretty easily
[17:28:40] <bd808> ori-l: cool beans. I've done quite a few nagios monitor scripts in the past but never ganglia
[17:30:08] <grrrit-wm> (�03PS1�) �10�Pyoungmeister��: also adding chad to ES hardware test boxes [operations/puppet] - �10https://gerrit.wikimedia.org/r/76944�
[17:32:06] <grrrit-wm> (�03CR�) �10�Pyoungmeister��: [C: �03��2���] also adding chad to ES hardware test boxes [operations/puppet] - �10https://gerrit.wikimedia.org/r/76944� (owner: �10�Pyoungmeister��)
[17:32:29] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:33:07] <manybubbles> ^d: after merging my fix for nonsense suggestions I had to rebuild the index. Using the --reindexAndRemoveOk flag on the config update script that I mentioned in email it reindexed everything about 20 seconds.
[17:33:19] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time
[17:33:24] <^d> Yay :)
[17:33:31] <notpeter> ^d: ok, you're in there. will have access after puppet run
[17:33:38] <^d> yw
[17:33:41] <^d> *ty!
[17:33:45] <notpeter> hehhe
[17:33:46] <notpeter> yw
[17:39:40] <manybubbles> ^d: have you had a chance to look at something that allows users to switch between search systems? I'm told someone is working on a beta tool..... and you just disconnected. to email
[17:43:01] <grrrit-wm> (�03PS1�) �10�Demon��: Provide reviewer counts per patch [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945�
[17:48:25] <grrrit-wm> (�03CR�) �10�Ryan Lane��: [C: �04�-1��] "(1 comment)" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945� (owner: �10�Demon��)
[17:52:29] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:53:19] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.154 second response time
[18:00:15] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: "Hm... maybe I'll start an email thread about spec tests." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76059� (owner: �10�Ori.livneh��)
[18:01:29] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:02:19] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[18:13:22] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:13:45] <grrrit-wm> (�03PS1�) �10�Yuvipanda��: Move UploadWizard tracking categories out of AutoAdd [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76946�
[18:13:52] <YuviPanda> hey Reedy!
[18:14:15] <YuviPanda> mediawiki-config changes also go out only with wmfXX branches, right? not before that?
[18:14:32] <RoanKattouw> mw-config changes should be deployed immediately once +2ed
[18:14:35] <YuviPanda> ah
[18:14:46] <RoanKattouw> Anyone who merges an mw-config change without deploying it is a Bad Citizen
[18:14:52] <YuviPanda> RoanKattouw: so I'll have to wait until the wmfXX branches get deployed to get that +2'd?
[18:14:59] <YuviPanda> I'll probably file a bug then
[18:15:06] <YuviPanda> let me -2 that then
[18:15:19] <RoanKattouw> It can be done at basically any time, including today's lightning deploy window at 4pm PDT
[18:15:40] <YuviPanda> RoanKattouw: it depends on a UploadWizard deploy, and it isn't 'pressing' enough to do as an LD
[18:15:47] <YuviPanda> plus... I don't have deployer access :)
[18:16:12] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[18:17:00] <grrrit-wm> (�03CR�) �10�Yuvipanda��: [C: �04�-1��] "Should get merged / deployed when https://gerrit.wikimedia.org/r/#/c/76707/ goes out, which should be wmf13" [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76946� (owner: �10�Yuvipanda��)
[18:20:45] <logmsgbot> !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf12
[18:20:56] <morebots> Logged the message, Master
[18:22:22] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:23:12] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[18:25:42] <icinga-wm> PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:26:16] <grrrit-wm> (�03PS1�) �10�Reedy��: Wikipedias to 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76947�
[18:26:24] <grrrit-wm> (�03CR�) �10�Ori.livneh��: "Unit tests test units in isolation. They are inherently unsuitable for testing the interactions of the module with the underlying environm" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76059� (owner: �10�Ori.livneh��)
[18:26:38] <grrrit-wm> (�03CR�) �10�Reedy��: [C: �03��2���] Wikipedias to 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76947� (owner: �10�Reedy��)
[18:26:42] <icinga-wm> RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)
[18:26:50] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Wikipedias to 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76947� (owner: �10�Reedy��)
[18:27:03] <Reedy> greg-g: Are we deploying https://gerrit.wikimedia.org/r/#/c/76856 now then?
[18:27:43] <grrrit-wm> (�03PS1�) �10�Hashar��: mediawiki_singlenode: exec[] -> Exec[] [operations/puppet] - �10https://gerrit.wikimedia.org/r/76948�
[18:28:31] <greg-g> Reedy: I'd love one other person to +1 it.... but mutate isn't online
[18:28:40] <greg-g> I thought I saw him here
[18:29:27] <greg-g> "here" being "the office" since that isn't clear :)
[18:30:32] <greg-g> Reedy: though I guess, you're the other person who knows enough to review that patch, right?
[18:30:42] <icinga-wm> PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[18:30:43] <greg-g> :)
[18:31:15] <greg-g> Reedy: short answer is "yes"
[18:31:18] <Reedy> It's fairly clear what is being done, and why..
[18:31:22] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:31:42] <greg-g> yeah, and with Ryan's +1 it gives me hope we won't overload our servers... so :)
[18:31:52] <greg-g> go for it
[18:31:53] <cmjohnson1> !log amaranth currently unreachable according to nosy....going to try rebooting
[18:32:04] <Ryan_Lane> it should ideally do the opposite over overloading them
[18:32:04] <morebots> Logged the message, Master
[18:32:12] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time
[18:32:35] <Reedy> Said wikis should push you to https anyway
[18:32:35] <Ryan_Lane> the ones that are being set https are already forced https
[18:32:35] <grrrit-wm> (�03PS1�) �10�Yuvipanda��: Add python bindings for zeromq to toollabs [operations/puppet] - �10https://gerrit.wikimedia.org/r/76949�
[18:32:39] <nosy> hello
[18:32:42] <icinga-wm> RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)
[18:32:43] <YuviPanda> Coren: ^^ trivial merge?
[18:32:50] <Elsie> Hi nosy.
[18:32:51] <Ryan_Lane> exactly
[18:32:59] <greg-g> Ryan_Lane: yeah, ideally :)
[18:33:02] <Ryan_Lane> so those are really just being set for proper consistently
[18:33:07] * greg-g nods�
[18:33:07] <Ryan_Lane> *consistency
[18:33:09] <greg-g> cool
[18:33:13] <Coren> YuviPanda: Which?
[18:33:13] <nosy> hello Elsie
[18:33:17] <grrrit-wm> (�03PS2�) �10�Reedy��: Send rel=canonical pointing from HTTPS to HTTP [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76856� (owner: �10�Tim Starling��)
[18:33:19] <Elsie> nosy: Can you join #wikimedia-toolserver ?
[18:33:20] <YuviPanda> Coren: https://gerrit.wikimedia.org/r/76949
[18:33:25] <nosy> ok
[18:33:42] <grrrit-wm> (�03CR�) �10�Reedy��: [C: �03��2���] Send rel=canonical pointing from HTTPS to HTTP [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76856� (owner: �10�Tim Starling��)
[18:33:45] <grrrit-wm> (�03CR�) �10�coren��: [C: �03��2���] "Trivial package addition." [operations/puppet] - �10https://gerrit.wikimedia.org/r/76949� (owner: �10�Yuvipanda��)
[18:33:50] <YuviPanda> ty Coren
[18:33:54] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Send rel=canonical pointing from HTTPS to HTTP [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76856� (owner: �10�Tim Starling��)
[18:34:03] <Ryan_Lane> I'd imagine we won't see much of a difference for a while
[18:34:14] * greg-g nods�
[18:34:18] <Ryan_Lane> depending on how quickly google reindexes us
[18:34:23] <Elsie> Reedy: Grrr, I wanted to fix that commit message.
[18:34:24] <Elsie> Oh well.
[18:34:31] <Elsie> It wasn't using "Bug: XXX" syntax.
[18:34:42] <Ryan_Lane> actually, it would also depend on when they exit the cache, too
[18:34:42] <greg-g> oh, at the end, yeah :/
[18:34:50] <Ryan_Lane> so likely not a major change for 30 days or so
[18:34:55] <Ryan_Lane> or whenever the pages are edited
[18:34:59] <greg-g> right
[18:35:04] <nosy> cmjohnson1: am I in the right chat now?
[18:35:09] <logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
[18:35:15] <cmjohnson1> nosy: so the elom/ilom is not responding to my shutdown request. going to need sbernardin to go to DC
[18:35:21] <morebots> Logged the message, Master
[18:35:23] <cmjohnson1> nosy yes
[18:35:34] <nosy> cmjohnson1: just had a similar problem today
[18:35:44] <nosy> you can try to reset the service processor
[18:36:12] <nosy> cmjohnson1: could be like reset /SP
[18:36:22] <grrrit-wm> (�03PS1�) �10�Reedy��: Remove a couple of overrides same as default [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76951�
[18:36:53] <cmjohnson1> okay...will give it a go
[18:36:56] <grrrit-wm> (�03CR�) �10�Reedy��: [C: �03��2���] Remove a couple of overrides same as default [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76951� (owner: �10�Reedy��)
[18:36:58] <nosy> i did this with another host today and afterwards i was able to restart /SYS
[18:37:01] <greg-g> Reedy: hah, I was going to do that securepoll one today :)
[18:37:16] <nosy> cmjohnson1: thx
[18:37:18] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Remove a couple of overrides same as default [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76951� (owner: �10�Reedy��)
[18:37:19] <greg-g> (well, do as in submit the change)
[18:37:58] <logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
[18:37:59] <nosy> cmjohnson1: nagios says amaranth comes up
[18:38:29] <nosy> cmjohnson1: it did i have ssh now - thanks
[18:38:35] <cmjohnson1> really? cuz the reset hung on me...cool
[18:38:54] <nosy> probably it resetted its ssh demon too
[18:38:55] <aude> hmmmm, wikipedias being updated today?
[18:39:06] <nosy> then youll see it only on the serial console
[18:39:06] <aude> greg-g: are we making a new branch today?
[18:39:08] <cmjohnson1> that's what i am thinking
[18:39:21] <greg-g> aude: nope
[18:39:34] <greg-g> aude: sitting pretty with wmf12 everywhere until post-wikimania
[18:39:56] <aude> ok
[18:40:11] <aude> then maybe folks can look at https://gerrit.wikimedia.org/r/#/c/76481/ at some point :)
[18:40:12] <greg-g> aude: we'll branch wmf13 and push to testwikis on August 15th
[18:40:19] <aude> but understand any hesitance
[18:40:21] <grrrit-wm> (�03PS2�) �10�Demon��: Provide reviewer counts per patch [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945�
[18:40:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:40:26] <aude> ok
[18:40:43] <aude> i will be on holiday, but sure i'm not needed and everything will be fine
[18:41:11] <greg-g> :)
[18:41:12] <grrrit-wm> (�03CR�) �10�Andrew Bogott��: [C: �03��2���] "I still want tests, but I can write 'em myself :)" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76059� (owner: �10�Ori.livneh��)
[18:41:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time
[18:41:24] <Reedy> I wonder how many ad-hoc deploys Wikimania is going to generate
[18:41:41] <aude> heh
[18:41:48] <greg-g> :(
[18:42:02] <YuviPanda> greg-g: oh, no deployments this week?
[18:42:17] <ori-l> YuviPanda: no, we're saving everything for wikimania
[18:42:28] <YuviPanda> hmm, deployment parties?
[18:42:48] <greg-g> ori-l: shush you
[18:42:54] <grrrit-wm> (�03CR�) �10�MZMcBride��: "(1 comment)" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945� (owner: �10�Demon��)
[18:43:02] <greg-g> YuviPanda: no deploys next week, during wikimania, resuming the week of Aug 12th
[18:43:10] <Reedy> Whilst at Wikimania no one will be using wikipedia et al, so we can break it and no one will notice
[18:43:12] <YuviPanda> greg-g: heh, okay!
[18:43:13] <greg-g> I sent a message to engineering@ and ops@ ;)
[18:43:33] <greg-g> will also send a message to wikitech-l with the normal deploy highlights email
[18:43:39] <Elsie> I'm not sure what "no deployments" means.
[18:43:40] <greg-g> Reedy: oh right!
[18:43:43] <YuviPanda> greg-g: I sortof skimmed it and sortof assumed that menat no deployments the week of wikimania, but nevermind :)
[18:43:46] <Elsie> Usually things get fixed at Wikimania.
[18:44:03] <Elsie> Because you can harass people in-person.
[18:44:06] <YuviPanda> greg-g: I also saw the mobile-tech mail, and confused general deployment with mobile deployment. nevermind :)
[18:44:12] <greg-g> ah
[18:44:29] <greg-g> Elsie: well, you can get patches merged, but they won't go out unless there is an extreme need to do so
[18:45:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:45:24] <Elsie> greg-g: That isn't really in keeping with past hackathons/meetups/Wikimanias. :-)
[18:46:01] <greg-g> yeah, I know :/
[18:46:03] <Elsie> I'm also not sure I understand the concern.
[18:46:20] <Elsie> Usually there's a higher concentration of people available and willing to help.
[18:47:05] <nosy> cmjohnson1: thanks for your help
[18:47:07] <nosy> bye
[18:47:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time
[18:47:17] <ori-l> ..which inevitably means an alarming paucity of people available and willing to help if the conference center network link goes down.
[18:47:24] <greg-g> Elsie: well, it's also sporadic travel dates/etc. we made a spreadsheet of availability and we decided the moratorium was warranted
[18:47:41] <greg-g> ori-l: yeah, "high concentration all on one uplink..."
[18:48:11] <ori-l> andrewbogott_afk: thanks
[18:52:04] <icinga-wm> PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:52:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:53:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[18:53:54] <icinga-wm> RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0)
[18:55:29] <Reedy> WIKIPEDIA IS DOWN MASH F5
[18:56:08] * greg-g breathes�
[18:56:24] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:58:14] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.207 second response time
[19:16:17] <grrrit-wm> (�03PS2�) �10�Hashar��: exec[] -> Exec[] [operations/puppet] - �10https://gerrit.wikimedia.org/r/76948�
[19:42:05] <grrrit-wm> (�03PS1�) �10�Catrope��: Remove config variables for VE gender survey [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76958�
[19:51:23] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:52:13] <icinga-wm> RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time
[19:55:53] <icinga-wm> PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[19:55:53] <icinga-wm> PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[19:55:53] <icinga-wm> PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[19:55:53] <icinga-wm> PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours
[19:55:53] <icinga-wm> PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[19:55:54] <icinga-wm> PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours
[19:58:35] <grrrit-wm> (�03PS1�) �10�Manybubbles��: Turn CirrusSearch on for all wikis in beta. [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77003�
[19:58:52] <grrrit-wm> (�03CR�) �10�MZMcBride��: "Good riddance." [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76958� (owner: �10�Catrope��)
[20:01:22] <grrrit-wm> (�03CR�) �10�Mattflaschen��: [C: �03��2���] "Corresponding VE change has been merged." [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76958� (owner: �10�Catrope��)
[20:01:54] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Remove config variables for VE gender survey [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76958� (owner: �10�Catrope��)
[20:03:56] <grrrit-wm> (�03CR�) �10�Catrope��: "Please don't merge changes in mediawiki-config without deploying them immediately. mediawiki-config should reflect the actual state of the" [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/76958� (owner: �10�Catrope��)
[20:04:06] <grrrit-wm> (�03PS1�) �10�Catrope��: Revert "Remove config variables for VE gender survey" [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77033�
[20:04:19] <grrrit-wm> (�03CR�) �10�Catrope��: [C: �03��2��� V: �03��2���] Revert "Remove config variables for VE gender survey" [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77033� (owner: �10�Catrope��)
[20:07:19] <icinga-wm> PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error
[20:11:13] <grrrit-wm> (�03PS1�) �10�Hashar��: adjust jobrunner/videoscaler role for beta [operations/puppet] - �10https://gerrit.wikimedia.org/r/77034�
[20:18:25] <grrrit-wm> (�03CR�) �10�Hashar��: "That is yet another crazy change :-]" [operations/puppet] - �10https://gerrit.wikimedia.org/r/77034� (owner: �10�Hashar��)
[20:19:58] <Reedy> hashar: Are you trying to hint you're due a holiday? ;)
[20:20:24] <hashar> Reedy: let me google search your english :-D
[20:20:31] <hashar> what do you mean by "due a holiday" ?
[20:22:14] <Reedy> err
[20:22:16] <grrrit-wm> (�03CR�) �10�Demon��: [C: �03��2���] Turn CirrusSearch on for all wikis in beta. [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77003� (owner: �10�Manybubbles��)
[20:22:25] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Turn CirrusSearch on for all wikis in beta. [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77003� (owner: �10�Manybubbles��)
[20:23:37] <hashar> Reedy: but yeah the manifests for the videoscaler/jobrunner are unlikely to land before friday :-D
[20:24:20] <hashar> btw, I had a quick look at the jobrunner shell script. That piece has grown to an unmanageable piece of soft we probably should phase out
[20:28:55] <andrewbogott> Ryan_Lane, what are your opinions about base.pp? Do you think that the pieces in there belong legitimately grouped together, or should they be spun out into a bunch of roles?
[20:29:44] <Ryan_Lane> ummm
[20:29:45] <Ryan_Lane> heh
[20:29:46] <Ryan_Lane> hmm
[20:30:10] <Ryan_Lane> I personally think split out
[20:30:37] <Ryan_Lane> not sure if others would agree
[20:31:18] <andrewbogott> There are some nodes that don't include 'standard' or 'base' but then pull in a bunch of base:: things piecemeal.
[20:31:29] <andrewbogott> I'm not sure that's an argument one way or the other...
[20:32:24] * hashar sends andrew to split site.pp as well�
[20:33:17] <hashar> Ryan_Lane: I was wondering earlier if we could get the puppet branch renamed from production to master :D
[20:33:21] <andrewbogott> I don't /want/ to split base.pp, I'd rather just cram it into a module...
[20:33:42] <andrewbogott> But it seems like quite the grab-bag right now.
[20:34:19] <hashar> could even be several modules
[20:34:23] <Ryan_Lane> yeah, it's a giant grab bag right now
[20:34:58] <hashar> some are related to low level configuration (hardware, network), some other are tweaking the distribution (motd, common packages)
[20:35:15] <hashar> and there is the ugly syslog stuff :-D
[20:37:32] <andrewbogott> I want to just ignore it for now, but I need a module to stick custom facts into…
[20:45:03] <grrrit-wm> (�03CR�) �10�QChris��: [C: �03�1��] "(2 comments)" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945� (owner: �10�Demon��)
[20:57:09] <icinga-wm> RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time
[20:57:09] <icinga-wm> RECOVERY - Parsoid on wtp1017 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.007 second response time
[20:58:14] <greg-g> chrismcmahon: so, beta cluster isn't loading for me, getting a 503
[20:58:37] <chrismcmahon> greg-g: bah. looking
[20:58:59] <icinga-wm> PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours
[21:00:10] <hashar> greg-g: before I disappear, I replied on some bug related to monitoring beta
[21:00:37] <hashar> greg-g: basically, that needs an Icinga instances that get populated by the already existing puppet manifests :)
[21:01:59] <greg-g> hashar: /me looks for them
[21:03:05] <jeremyb> greg-g: loads for me...
[21:03:09] <jeremyb> chrismcmahon: did you fix something?
[21:03:29] <greg-g> jeremyb: loads now, yeah
[21:03:58] <chrismcmahon> ? srsly?
[21:04:39] <chrismcmahon> missing CSS for me again
[21:04:46] <chrismcmahon> also tons of fatals in the log
[21:07:08] <greg-g> chrismcmahon: css there for me
[21:07:28] <chrismcmahon> greg-g: OK, looks better now
[21:07:38] <greg-g> what are the fatals referencing?
[21:07:55] <chrismcmahon> greg-g: manybubbles this change might have given beta a little freakout https://gerrit.wikimedia.org/r/#/c/77003/
[21:08:01] <hashar> greg-g: finally found my reply https://bugzilla.wikimedia.org/show_bug.cgi?id=51497#c1
[21:08:36] <hashar> can't talk anymore, too tired need sleep sorry
[21:08:43] <greg-g> hashar: go sleep!
[21:09:10] <manybubbles> chrismcmahon: I'm sorry! Did i break something important?
[21:10:03] <hashar> I looked a bit at the log, did not find anything obvious
[21:10:18] <hashar> there might be cache issues as well though an error 503 should not ends up being cached
[21:11:15] <chrismcmahon> manybubbles: don't know yet (and I'm not even sure that change is the culprit-- I was seeing tons of GETs to e.g. http://www.tr.wikipedia.beta.wmflabs.org/ returning fatal ..."/MWMultiVersion.php(351): trigger_error('Invalid host name...
[21:11:28] <hashar> chrismcmahon: that host does not exist anymore
[21:11:38] <hashar> must be some bot attempting to access it via its robots.txt
[21:11:53] <hashar> Invalid host name (www.tr.wikipedia.beta.wmflabs.org), can't determine language.
[21:11:56] <chrismcmahon> dunno
[21:12:21] <hashar> the www is not recognized by our lame URL routing system
[21:12:28] <hashar> http://tr.wikipedia.beta.wmflabs.org/ gives the proper "No wiki found"
[21:12:58] <hashar> and http://hashar.sleeps.beta.wmflabs.org gives a "Domain not configured"
[21:12:59] <manybubbles> Is this up to date? http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:SiteMatrix
[21:13:20] <hashar> that covers three� types of errors we send, all of them having different layout unfortunately
[21:13:48] <hashar> manybubbles: yeah it is supposed to load the list from the all-labs.dblist
[21:13:54] <hashar> (in operations/puppet-config )
[21:14:07] <hashar> we might want to edit SiteMatrix to let it skip languages with no projects
[21:14:26] <manybubbles> hashar: I was looking for that file! I knew there must be one but I couldn't find it. it was right there!
[21:14:42] <hashar> in operations/mediawiki-config
[21:14:56] <hashar> !beta
[21:14:56] <wm-bot> !log deployment-prep $*
[21:15:07] <morebots> Logged the message, Master
[21:15:13] <hashar> oh
[21:15:37] <hashar> manybubbles: work in progress at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Add_a_wiki
[21:16:09] <greg-g> huh, never knew we had eg: http://en.wikisource.beta.wmflabs.org/wiki/Main_Page
[21:16:32] <greg-g> why did wm-bot say that?
[21:16:46] <jeremyb> greg-g: hashar said !beta
[21:16:53] <jeremyb> !beta hi greg-g
[21:16:53] <wm-bot> !log deployment-prep hi greg-g
[21:16:53] <hashar> !del beta
[21:16:54] <wm-bot> if you want to delete a key, this is wrong way
[21:17:01] <jeremyb> hah
[21:17:03] <jeremyb> !del
[21:17:03] <wm-bot> if you want to delete a key, this is wrong way
[21:17:05] <morebots> Logged the message, Master
[21:17:09] <greg-g> :)
[21:17:14] <hashar> SUPER HELPFUL MESSAGE AWARD TO wm-bot
[21:17:19] <hashar> 'this is wrong way'
[21:17:49] <hashar> some program gives you something similar when using --help : "no help there, please use -help"
[21:17:59] <hashar> @del beta
[21:18:05] <hashar> @unalias beta
[21:18:08] <jeremyb> hah
[21:18:15] <jeremyb> no, it's !beta del
[21:18:17] <greg-g> !forgetthedangthing beta
[21:18:20] <hashar> !beta
[21:18:20] <wm-bot> !log deployment-prep $*
[21:18:23] <greg-g> hah
[21:18:37] <jeremyb> morebots: is slow
[21:18:37] <morebots> I am a logbot running on wikitech-static.
[21:18:37] <morebots> Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log.
[21:18:37] <morebots> To log a message, type !log <msg>.
[21:18:46] <greg-g> except...
[21:19:03] * hashar looks at https://meta.wikimedia.org/wiki/Wm-bot�
[21:19:12] <jeremyb> hashar: !beta del!
[21:19:37] <hashar> !beta del!
[21:19:37] <wm-bot> !log deployment-prep del!
[21:19:42] <hashar> ;-]
[21:19:42] <jeremyb> grrr
[21:19:45] <jeremyb> !beta del
[21:19:45] <wm-bot> Successfully removed beta
[21:19:48] <morebots> Logged the message, Master
[21:19:56] <hashar> OH MY GOD
[21:20:04] <hashar> !beta am i gone
[21:20:09] <jeremyb> sleep?
[21:20:18] <hashar> !beta is https://wikitech.wikimedia.org/wiki/Beta
[21:20:18] <wm-bot> Key was added
[21:20:23] <hashar> !beta | manybubbles
[21:20:23] <wm-bot> manybubbles: https://wikitech.wikimedia.org/wiki/Beta
[21:20:25] <hashar> !!
[21:21:13] * hashar suddenly realize he is not dreaming about interacting with a weird irc bot but is actually awake�
[21:21:21] <hashar> *waves guys*
[21:21:25] <greg-g> g'night!
[21:23:24] <manybubbles> gnight
[21:43:31] <grrrit-wm> (�03CR�) �10�Demon��: "(1 comment)" [operations/puppet] - �10https://gerrit.wikimedia.org/r/76945� (owner: �10�Demon��)
[21:53:59] <icinga-wm> PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours
[21:58:18] <andrewbogott> Jeff_Green, can you tell me why aluminium excludes a couple of classes from the base class? It looks like you've pruned out… base::remote-syslog, base::screenconfig and role::salt::minions
[21:58:22] <andrewbogott> Is that importand and on purpose?
[21:58:25] <andrewbogott> *important
[22:00:05] <Jeff_Green> andrewbogott: aluminium really has no business being puppetized on prod puppet anymore
[22:00:41] <Jeff_Green> I don't think I was the one who excluded those, and al predates at least the salt stuff--my guess is people left it out when they deployed new things
[22:01:21] <andrewbogott> It's your patch, sha commit c158c726609cf4ab414a81b26e4466c459b4c193
[22:01:59] <andrewbogott> But, anyway… what would it mean for a box to not be puppetized? I didn't know we did that, ever.
[22:03:03] <Jeff_Green> all the fundraising hosts are moving within the frack walled garden
[22:03:22] <Jeff_Green> they're on their own puppet instance, which is much leaner
[22:03:46] <andrewbogott> OK -- do you mind submitting a patch that removes aluminium from site.pp and cleans up the unused roles? that'll make my life a lot simpler
[22:03:56] <andrewbogott> Or I guess I can do it and leave the patch for you to review… as you like.
[22:04:03] <Jeff_Green> not yet
[22:04:23] <Jeff_Green> I'll do that once I cut over puppet, but we need to test before I can do that
[22:04:37] <grrrit-wm> (�03PS1�) �10�Catrope��: Remove config variables for VE gender survey [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77058�
[22:04:40] <andrewbogott> d'you know when that will happen?
[22:05:09] <Jeff_Green> maybe toward the end of august?
[22:05:20] <Jeff_Green> I don't think I'll be able to get to it before I go on vacation on the 12th
[22:05:20] <andrewbogott> hm, ok.
[22:05:32] * andrewbogott wonders how to handle all those base exceptions in the meantime�
[22:09:19] <Jeff_Green> andrewbogott: why does it have to change at all?
[22:09:53] <andrewbogott> That is quite a long story :)
[22:10:04] <Jeff_Green> seems as though it's just a bit of nonstandardness that's contained in role/fundraising.pp
[22:10:10] <grrrit-wm> (�03PS1�) �10�Reedy��: Point php at 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77061�
[22:10:35] <grrrit-wm> (�03CR�) �10�Reedy��: [C: �03��2���] Point php at 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77061� (owner: �10�Reedy��)
[22:11:33] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Point php at 1.22wmf12 [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77061� (owner: �10�Reedy��)
[22:11:37] <andrewbogott> Jeff_Green, the short-term question I'm trying to answer is, 'which of these classes are local and only used in base.' The answer is, almost all of them come the end of August, but almost none of them today.
[22:11:44] <andrewbogott> It's OK, I'm not sure that's an immediate stumbling block...
[22:12:01] <Jeff_Green> looking at the specifics here
[22:13:20] <logmsgbot> !log csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'eventlogging patch'
[22:13:30] <morebots> Logged the message, Master
[22:13:53] <Jeff_Green> remote-syslog: fundraising has its own central log collection scheme
[22:14:36] <andrewbogott> ok, so that's probably why it's different
[22:17:23] <Jeff_Green> screenconfig I'm less sure about, that one probably doesn't matter
[22:20:38] <Jeff_Green> salt::minons--looks like that was added just before I split out the base stuff, so it wasn't there to consider
[22:20:47] * andrewbogott nods�
[22:22:38] <Jeff_Green> I might be wrong. my brain isn't handling the timezone difference on the commmits. but it was close anyway :-P
[22:23:12] <andrewbogott> I'm sure you're right -- anyway for the moment I'm interested in whether or not it can be 100% reconciled, which it sounds like it can't.
[22:23:57] <Jeff_Green> I gotta head out. dinner
[22:24:02] <andrewbogott> 'k
[22:28:12] <andrewbogott> Jeff_Green, will the 'payments' boxes be moving to the frack puppet server as well?
[22:28:43] <jeremyb> it's hard not to think of hydrofracking
[22:29:27] <andrewbogott> I assume that frack is three more than crack
[22:30:14] <jeremyb> maybe depends on your locale. that's true for LC_ALL=C at least
[22:31:52] <andrewbogott> Hm, I was going to make a cyrillic joke but looks like they don't even have an 'f'
[22:44:44] <andrewbogott> Coren, can you tell me why streber uses a subset of base classes rather than just including base?
[22:44:47] <andrewbogott> Or, better yet, 'standard'?
[22:48:11] <andrewbogott> Or, actually, Coren, looks like all you did was retab those lines, not write them.
[22:49:00] <Coren> Not my fault! *phew*
[22:51:56] <bd808> ori-l: do you have a few minutes to chat about monitoring?
[22:53:01] <greg-g> yay, monitoring
[22:53:05] <greg-g> :P
[22:53:31] <bd808> greg-g: I'm accepting help and tips from all parties
[22:54:28] <bd808> I'm looking at some of the existing ganglia plugins. They mostly make sense
[22:54:50] <greg-g> "mostly"
[22:55:14] <bd808> the task I've been given will involve getting some info from the wiki db and then making some http requests
[22:55:44] <bd808> I haven't found any that do something like that: db access
[22:55:56] <bd808> wondering if I'm barking up the wrong tree
[22:57:07] <bd808> basically we want to lookup some smallish number of files that should have been purged from squids and then poke the squid to see how fresh the data there is
[22:57:39] <bd808> and output some data about number of stales found
[22:58:03] <bd808> so we can watch for trends of more stales than "expected" over time
[22:58:08] <greg-g> huh, neat, and yeah, can't help ya there, bd808 :)
[22:58:46] <bd808> sheesh. I guess if it was easy it would have already been done
[22:58:48] <bd808> :)
[23:05:59] <icinga-wm> PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours
[23:06:01] <Jeff_Green> andrewbogott: i don't have the energy right now to get involved in the email thread but to reduce confusion: payments* can be removed entirely from puppet if they're still there
[23:06:14] <andrewbogott> Jeff_Green, awesome, good to know.
[23:06:23] <Jeff_Green> they've been on frack puppet for a very long time
[23:06:41] <Jeff_Green> and! don't care about the screen one
[23:07:25] <Jeff_Green> and! i think the remote logging one is sorta flawed by design, imho log destination should be more configurable than by-colo
[23:07:31] <andrewbogott> Jeff_Green, can the lines like "$sip['payments'][$::site],"� also be scrubbed from the load balancers?
[23:07:42] <Jeff_Green> ooh, definitely not
[23:08:01] <Jeff_Green> the LBs *do* use production puppet.
[23:08:09] <andrewbogott> ok
[23:08:28] <Jeff_Green> I can go through this a bit tomorrow. right now I'm suffering some kind of bug-plague
[23:08:37] <Jeff_Green> i'll try to clean out any cruft I can
[23:08:48] <andrewbogott> Yeah, no rush, I'm about done for the day anyway
[23:08:53] * Jeff_Green nods off for the day. have a good one�
[23:09:05] <grrrit-wm> (�03PS1�) �10�Andrew Bogott��: Remove payment nodes from site.pp [operations/puppet] - �10https://gerrit.wikimedia.org/r/77069�
[23:16:53] <logmsgbot> !log bsitu synchronized php-1.22wmf12/extensions/Echo 'Update Echo to master'
[23:17:03] <morebots> Logged the message, Master
[23:24:25] <logmsgbot> !log rmoen Started syncing Wikimedia installation... : update visualeditor to master
[23:24:36] <morebots> Logged the message, Master
[23:31:04] <logmsgbot> !log rmoen Finished syncing Wikimedia installation... : update visualeditor to master
[23:31:13] <morebots> Logged the message, Master
[23:34:42] <grrrit-wm> (�03CR�) �10�Robmoen��: [C: �03��2��� V: �03��2���] Remove config variables for VE gender survey [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77058� (owner: �10�Catrope��)
[23:35:04] <grrrit-wm> (�03Merged�) �10�jenkins-bot��: Remove config variables for VE gender survey [operations/mediawiki-config] - �10https://gerrit.wikimedia.org/r/77058� (owner: �10�Catrope��)
[23:37:41] <logmsgbot> !log rmoen synchronized wmf-config/CommonSettings.php 'Remove config variables for VE gender survey'
[23:37:52] <morebots> Logged the message, Master
[23:38:51] <logmsgbot> !log rmoen synchronized wmf-config/InitialiseSettings.php 'Remove config variables for VE gender survey'
[23:39:02] <morebots> Logged the message, Master
