@@ -0,0 +1,266 @@ | |||
Title: Backups... and restores | |||
Date: 2018-05-07 14:17 | |||
Author: Wxcafé | |||
Category: Note | |||
Slug: backups-and-restore | |||
So, as you might have noticed if you're following me on twitter/mastodon, or if | |||
you check your rss reader logs, or if you just happened to check this website in | |||
the last week, my server has been down for about four days last week following | |||
a hardware failure. Here's what happened. | |||
So, on Monday morning (30th of April), I started seeing hardware errors in dmesg | |||
and broadcast on consoles. I figured that a kernel message about a hardware | |||
failure that was broadcast on all consoles was probably important enough to at | |||
least investigate, and I found out that it was related to the motherboard dying. | |||
I immediately opened a ticket with my hosting provider (Online.net) to ask them | |||
to replace the motherboard. It took them 5 hours to react, and in the meantime | |||
the server had gone down. I pressed them on, the support agent tried to reboot | |||
the machine in rescue mode which obviously didn't work since the mobo was toast, | |||
and then decided that the machine was lost and *gave me a new one*. Which meant | |||
that I didn't have access to my data anymore. | |||
I tried to have them plug the disk of the old machine in the new one, but they | |||
"couldn't do that on this hardware" (I've since checked, and that hardware uses | |||
2.5" SATA drives, which means they'd only have had to unplug the disk from the | |||
old machine and put it in the new one. At the most, four screws might be | |||
involved. But anyway.), so they told me that they were sorry but I'd have to | |||
restore from my backups. | |||
Which, thankfully, I had! Complete backups from that same day, 4:15am. Obviously | |||
the situation would have been much worse otherwise, and I thanked the day I had | |||
decided to setup a sensible backup strategy. So I set to work on restoring | |||
these. | |||
My backups are managed via duplicity; I have a setup where the first puppet run | |||
on a server installs some basic backup definitions, and some more targeted | |||
configuration once they're configured, depending on what they're used for. This | |||
setup is described at the end of the post, if you're interested. | |||
Anyway, these are broken up into what duplicity calls "targets", which are | |||
ensembles of folders that are backed-up with the same rules (frequency, time | |||
before expiration, etc...). The main ones in my setup are `homedir`, which | |||
includes... my home directory, yes; `conf_files`, which includes `/etc`, `/var`, | |||
`/opt` and `/usr/local`; `srv_data`, which includes *most* of `/srv`, and | |||
finally `mysql` and `pgsql`, which have a pre-run hook to dump the respective | |||
databases and then backs them up. | |||
So, on the evening of the 30th, I started restoring these. After fiddling for | |||
a bit to figure out how duplicity restores work, I started restoring the | |||
`homedir` target. And that's when I found out that restoring data from an sftp | |||
server running behind an ADSL connection takes *ages*, a fact that's only made | |||
worse by the insistence of duplicity to copy to the remote the signature files | |||
and indexes for *all* the full backups, and not just the latest ones | |||
applicable. In this case, it took about three days. | |||
I managed to restore email first, as that was the most urgent, to avoid having | |||
bounces (most MTAs retry for 3-5 days before giving up on delivery), and then | |||
slowly walked my way back to restoring all of /var (including the cache, which | |||
I had forgotten to exclude from my backups...), and /srv/pub, which holds | |||
https://pub.wxcafe.net and https://wxcafe.net/pub, and which included (among | |||
other things) a few HD movies, some taking over 4GB. | |||
Needless to say, this restore took a long time. I've learned a few lessons from | |||
that whole thing, though: | |||
- never assume the hosting provider is gonna do the right thing, | |||
- decide how much downtime you are willing to live with | |||
- check your backups regularly and see how fast they restore | |||
- define prioritized restoration targets (i.e. website and mail server. That | |||
xmpp server can probably wait.) | |||
- don't stress out too much about this. it's gonna be okay, and rebuilding can | |||
always work. you'll find a solution. | |||
Anyways, in the end all I lost was a few months of my RSS subscriptions, which, | |||
while annoying, is definitely something I can live with. It worked out alright | |||
in the end. | |||
Now for that puppet/duplicity config... | |||
I use the very good | |||
[puppet-duplicity](https://github.com/tohuwabohu/puppet-duplicity) module, | |||
which defines most of what you need already. Then, it so happens that there's | |||
a bug in the paramiko version most of my servers have, so I have taken to | |||
replacing the file with that bug with a fixed version, which you can find | |||
[here](https://git.wxcafe.net/snippets/13) | |||
I then define a `backup` class, that can be used where ever it's needed in | |||
host definitions: | |||
```ruby | |||
## Puppet backups with duplicity | |||
# definitions | |||
class backups { | |||
file { '/var/backups/mysql/': | |||
ensure => directory, | |||
} | |||
file { '/var/backups/pgsql/': | |||
ensure => directory, | |||
} | |||
class { 'duplicity': | |||
backup_target_url => "sftp://censored//srv/backups/$hostname", | |||
backup_target_username => 'duplicity', | |||
backup_target_password => 'censored', | |||
} | |||
## dirty hotfix | |||
if $facts['os']['name'] == 'freebsd' { | |||
file { '/usr/local/lib/python2.7/site-packages/duplicity/backends/_ssh_paramiko.py': | |||
ensure => present, | |||
content => file('base/backups/_ssh_paramiko.py'), | |||
require => Package['duply'] | |||
} | |||
} else { | |||
file { '/usr/lib/python2.7/dist-packages/duplicity/backends/_ssh_paramiko.py': | |||
ensure => present, | |||
content => file('base/backups/_ssh_paramiko.py'), | |||
require => Package['duply'] | |||
} | |||
} | |||
if $facts['os']['name'] == 'freebsd' { | |||
package {'py27-pip': | |||
ensure => present, | |||
} | |||
package {'py27-cryptography': | |||
ensure => present, | |||
} | |||
} else { | |||
package {'python-pip': | |||
ensure => present, | |||
} | |||
package {'python-cryptography': | |||
ensure => present | |||
} | |||
} | |||
duplicity::profile { 'conf_file': | |||
full_if_older_than => "2W", | |||
max_full_backups => 3, | |||
cron_hour => '05', | |||
cron_minute => '20', | |||
cron_enabled => true, | |||
gpg_encryption => false | |||
} | |||
duplicity::profile {'homedir': | |||
full_if_older_than => "1M", | |||
max_full_backups => 3, | |||
cron_hour => '04', | |||
cron_minute => '40', | |||
cron_enabled => true, | |||
gpg_encryption => false, | |||
} | |||
duplicity::profile {'srv_data': | |||
full_if_older_than => "1M", | |||
max_full_backups => 3, | |||
cron_hour => '05', | |||
cron_minute => '35', | |||
cron_enabled => true, | |||
gpg_encryption => false | |||
} | |||
duplicity::profile { 'pgsql': | |||
full_if_older_than => "1W", | |||
max_full_backups => 2, | |||
cron_hour => '04', | |||
cron_minute => '20', | |||
cron_enabled => true, | |||
gpg_encryption => false, | |||
exec_before_content => 'sudo pg_dumpall -h 127.0.0.1 -U postgres -f /var/backups/pgsql/db.sql' | |||
} | |||
duplicity::profile { 'mysql': | |||
full_if_older_than => "1W", | |||
max_full_backups => 2, | |||
cron_hour => '04', | |||
cron_minute => '20', | |||
cron_enabled => true, | |||
gpg_encryption => false, | |||
exec_before_content => 'sudo mysqldump -pcensored --all-databases --result-file=/var/backups/mysql/db.sql' | |||
} | |||
} | |||
``` | |||
And then here's a sample from a node definition: | |||
```ruby | |||
node 'yoshi.wxcafe.net' { | |||
$physical_location = "Illiad - DC2, Vitry-sur-Seine" | |||
include base | |||
include backups | |||
duplicity::file {'/var/backups/mysql/': | |||
profile => 'mysql', | |||
ensure => 'present' | |||
} | |||
duplicity::file {'/var/backups/pgsql': | |||
profile => 'pgsql', | |||
ensure => 'present' | |||
} | |||
duplicity::file {'/etc/': | |||
profile => 'conf_file', | |||
ensure => 'present' | |||
} | |||
duplicity::file {'/var/': | |||
profile => 'conf_file', | |||
ensure => present | |||
} | |||
duplicity::file {'/usr/local/': | |||
profile => 'conf_file', | |||
ensure => present | |||
} | |||
duplicity::file {'/opt/': | |||
profile => 'conf_file', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/lists/': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/mail/': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/pub/': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/rpg/': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/wallabag/': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/srv/www': | |||
profile => 'srv_data', | |||
ensure => present | |||
} | |||
duplicity::file {'/home/': | |||
profile => 'homedir', | |||
ensure => present, | |||
} | |||
} | |||
``` |
@@ -0,0 +1,105 @@ | |||
Title: [N]olife | |||
Date: 2018-04-23T11:00+02:00 | |||
Author: Wxcafé | |||
Category: | |||
Slug: nolife | |||
Exceptionally, this post will be in french, because it's about a very | |||
french-specific cultural phenomenon. `Nolife` was a french TV station that started | |||
out in 2007, was centered on "geek, nerd and otaku" culture (but was done by | |||
real enthusiasts), was run with almost no money nor means, and always stayed | |||
true to the people who were running it. It died because of lack of funds after | |||
11 years of broadcasts on the 8th of April, 2018. In this column I'm going to | |||
talk about my personal relationship with it. | |||
Bon, ça y est, `Nolife` est morte. Ça faisait longtemps que ça devait arriver, | |||
depuis les débuts de la chaîne, en fait, mais... au bout d'un moment, après | |||
l'avoir vue traverser toutes ces morts annoncées certaines et en ressortir sans | |||
une égratignure, voire parfois plus grande encore, on pouvait finir par se dire | |||
que ça n'arriverait en fin de compte jamais, que les personnes qui font tourner | |||
la chaîne arriveraient toujours à trouver des solutions pour qu'elle continue. | |||
Il se trouve que non. | |||
J'ai bien conscience que pour beaucoup de gens, `Nolife` se situe entre "juste une | |||
chaîne de télé" et "c'était sympa mais voilà quoi", et c'est tout a fait normal | |||
d'avoir ce point de vue. Mais `Nolife`, ça a été pour moi quelque chose d'un peu | |||
différent. | |||
J'ai découvert nolife quand j'étais en lycée, en seconde en fait. Je connaissais | |||
déjà un peu les cultures traitées par la chaîne, parce que je lisais pas mal de | |||
fantasy et de SF, et que je m'intéressais beaucoup au fonctionnement des | |||
ordinateurs, et aux jeux vidéos; et j'avais déjà un certain contact avec des | |||
communautés liées, notamment le [Netophonix](https://www.netophonix.com/) (une | |||
communauté dédiée aux sagas mp3). J'ai du coup assez vite accroché a `Nolife` et | |||
aux émissions qui étaient présentes sur la chaîne, et ça a été une porte (parmi | |||
d'autre, mon groupe d'ami.e.s de l'époque ayant aussi pas mal aidé) vers la | |||
culture japonaise. C'était vers 2010, donc pas longtemps après la création de | |||
`Nolife` Online (le service de VOD créé par la chaîne a l'époque), et comme | |||
je n'avais pas énormément la possibilité de regarder la télé aux moments ou les | |||
émissions (autres que des clips musicaux, donc, qui prenaient 50% du temps | |||
d'antenne) passaient, j'ai rapidement décidé de m'abonner, et a partir de ce | |||
moment là ma consommation des programmes de `Nolife` ne passait quasiment que par | |||
`Nolife` Online. | |||
Comme dit, j'avais un contact avec le Netophonix, notamment par le channel IRC | |||
sur FreeNode. Logiquement, quand j'ai voulu connaître un peu la communauté qui | |||
traînait autour de `Nolife`, je suis tout de suite allé trouver le channel IRC | |||
associé. Et sur ce channel, j'ai recontré pas mal de gens, qui a leur tour et de | |||
leur propre façon m'ont fait découvrir de nombreuses choses, techniques ou non, | |||
dont l'exemple le plus parlant aujourd'hui est Twitter, qui bien qu'étant un | |||
réseau social... assez imparfait, a tout de même eu le mérite de me faire | |||
rencontrer énormément de gens qui ont été essentiels a la façon dont j'ai évolué | |||
en grandissant. Sans ces gens, je ne serais probablement pas éveillé a de | |||
nombreuses causes qui sont particulièrement importantes pour moi aujourd'hui (et | |||
ont participé a faire de moi la personne que je suis), notamment le féminisme, | |||
les luttes LGBT, et les luttes de classe. Je n'aurais pas découvert des domaines | |||
entiers, de littérature, de cinéma, de jeux vidéos, des domaines techniques, ... | |||
Alors, bien entendu, `Nolife` en soit ne m'a pas apporté tout cela. Sa communauté, | |||
et les gens que j'ai découvert par les amis que je me suis faits dans cette | |||
communauté, m'ont permis de me construire de cette façon, et la chaîne n'a eu | |||
qu'un apport mineur a ce sujet. Cependant, elle a tout de même eu des intérêts | |||
intrinsèques. En produisant énormément d'émissions qui **jamais** n'auraient pu | |||
être diffusées sur d'autres chaînes du paysage audiovisuel français (une | |||
émission parlant du hardware des vieilles consoles en partenariat avec | |||
[Mo5.com](http://mo5.com)? Une émissions expliquant en détail le fonctionnement | |||
de bugs utilisés par des superplayers et des speedrunners dans des jeux? ... Un | |||
top 5 de J-Music? Et même quelque chose qui semble évident aujourd'hui, mais... | |||
des let's play a la télé en 2007? Une émission sur l'e-sport, en 2010? Et il | |||
y a plein d'autres exemples...), cette chaîne a bien montré qu'en se battant, on | |||
pouvait faire des choses que tout le monde pensait impossible. En montrant des | |||
gens passionnés, pas forcément parfaitement a l'aise devant une caméra mais | |||
prêts a se mettre sur le devant de la scène pour présenter des choses qu'iels | |||
aiment, elle a permit a beaucoup de gens d'accepter et d'assumer leurs passions. | |||
Elle a démocratisé de nombreux sujets qui jusqu'ici étaient réservés à des | |||
communautés minuscules (je pense notamment aux speedruns, aux démos, a l'e-sport | |||
même, a la musique japonaise...). En diffusant des émissions japonaises ou | |||
parlant de la culture japonaise, non pas en envoyant des occidentaux visiter le | |||
japon et en rapporter ce qu'ils voulaient bien en montrer au public occidental, | |||
mais en diffusant des programmes japonais entièrement traduits (et sous-titrés, | |||
pas doublés!), type Japan in Motion, ou en employant des personnes japonaises | |||
prêtes a montrer ce qu'elles connaissaient du japon (Tokyo Café), voire en | |||
interviewant des personnes sur place pour qu'elles présentent leur vision de | |||
leur quartier ou de leur ville et de la culture qui l'accompagnait (Toco Toco). | |||
Bref, en donnant accès, a moi et a énormément d'autres personnes, non seulement | |||
a ces cultures et a ces sujets fermés, mais aussi a la passion claire des gens | |||
qui participaient a la chaine envers les sujets qu'iels traitaient, cette chaîne | |||
a fait beaucoup pour la construction de beaucoup de personnes dans mon cas, et | |||
a ouvert une fenêtre dans le paysage audiovisuel français. | |||
Cette fenêtre s'est refermée voici deux semaines. Je n'ai maintenant plus aucun | |||
regret a voir la télévision mourir. | |||
Je voudrais par ce post remercier les personnes qui ont fait `Nolife`, devant ou | |||
derrière (ou a côté de) la caméra. Je peux citer celles et ceux que je connais, | |||
certains avec lesquels j'ai pu prendre un verre au Kawaii café ou au Dernier Bar | |||
une fois ou deux (je n'étais pas encore majeur, je peux vous le dire | |||
maintenant), Mathilde, Seb et Alex évidemment, Medoc et Moguri, Thierry, | |||
Sylvain, Clément, Caroline, Benoît, Anne, Julien Pirou, Pili, Slimane, Radigo, | |||
Mickey, Macha, DamDam, Obliv, Josaudio... | |||
Merci pour ces 11 ans, merci pour votre acharnement a faire cette chaîne. Rien | |||
ne sera jamais tout a fait pareil... Mais heureusement, y a pas que la vraie vie | |||
dans la vie. |
@@ -0,0 +1,57 @@ | |||
================================================================== | |||
https://keybase.io/wxcafe | |||
-------------------------------------------------------------------- | |||
I hereby claim: | |||
* I am an admin of https://wxcafe.net | |||
* I am wxcafe (https://keybase.io/wxcafe) on keybase. | |||
* I have a public key ASATUYKHRl5bGwMh3n0KNIruvG-ivARCUOhsgIOTq3Ayfwo | |||
To do so, I am signing this object: | |||
{ | |||
"body": { | |||
"key": { | |||
"eldest_kid": "01205da51ba3f465c9f4df080b8583b497e87670e60493f353ae0128b888520b77df0a", | |||
"host": "keybase.io", | |||
"kid": "012013518287465e5b1b0321de7d0a348aeebc6fa2bc044250e86c808393ab70327f0a", | |||
"uid": "e3254eba7cdb3de6a427f80ab715cb19", | |||
"username": "wxcafe" | |||
}, | |||
"merkle_root": { | |||
"ctime": 1525542893, | |||
"hash": "5ab026b8dcb33a11f3128ce41929a6a6494ddd1c5e44392f39ce3c558da4120b59c8d23c55232a2e5aa09d1997b87e3411f1f5ce70af8ed17c1b36b43b38c1a9", | |||
"hash_meta": "2d86b87f9588d335c9325dbf95bf2fea52010ed271f23177572977e21593fdc1", | |||
"seqno": 2489929 | |||
}, | |||
"service": { | |||
"entropy": "DxIM4bWz8Ezd/Is8ltCpjK8u", | |||
"hostname": "wxcafe.net", | |||
"protocol": "https:" | |||
}, | |||
"type": "web_service_binding", | |||
"version": 2 | |||
}, | |||
"client": { | |||
"name": "keybase.io go client", | |||
"version": "1.0.47" | |||
}, | |||
"ctime": 1525542918, | |||
"expire_in": 504576000, | |||
"prev": "483b1bcfee9f94fc9bf512ab56a679ae297e336991443f761fc585893ff77ec5", | |||
"seqno": 36, | |||
"tag": "signature" | |||
} | |||
which yields the signature: | |||
hKRib2R5hqhkZXRhY2hlZMOpaGFzaF90eXBlCqNrZXnEIwEgE1GCh0ZeWxsDId59CjSK7rxvorwEQlDobICDk6twMn8Kp3BheWxvYWTESpcCJMQgSDsbz+6flPyb9RKrVqZ5ril+M2mRRD92H8WFiT/3fsXEIDnrmqfzoylRfSLJXT7+sAWGU/pBJhpzdvujtz+iBTTXAgHCo3NpZ8RAaLXPWzACNLZPAdvl53TVjvicOfKTBoR8Ewv0wxQbcS/syGxw5HdWaL1heUbWJYjeu2ktQXXMjRBHUne88e/1AKhzaWdfdHlwZSCkaGFzaIKkdHlwZQildmFsdWXEIFDUsxH3LBt2V95xdAykBXAkonHJmdpkr4hY3VoOmJgJo3RhZ80CAqd2ZXJzaW9uAQ== | |||
And finally, I am proving ownership of this host by posting or | |||
appending to this document. | |||
View my publicly-auditable identity here: https://keybase.io/wxcafe | |||
================================================================== | |||