parent
474eddd53a
commit
d890be6a99
@ -0,0 +1,266 @@ |
||||
Title: Backups... and restores |
||||
Date: 2018-05-07 14:17 |
||||
Author: Wxcafé |
||||
Category: Note |
||||
Slug: backups-and-restore |
||||
|
||||
So, as you might have noticed if you're following me on twitter/mastodon, or if |
||||
you check your rss reader logs, or if you just happened to check this website in |
||||
the last week, my server has been down for about four days last week following |
||||
a hardware failure. Here's what happened. |
||||
|
||||
So, on Monday morning (30th of April), I started seeing hardware errors in dmesg |
||||
and broadcast on consoles. I figured that a kernel message about a hardware |
||||
failure that was broadcast on all consoles was probably important enough to at |
||||
least investigate, and I found out that it was related to the motherboard dying. |
||||
|
||||
I immediately opened a ticket with my hosting provider (Online.net) to ask them |
||||
to replace the motherboard. It took them 5 hours to react, and in the meantime |
||||
the server had gone down. I pressed them on, the support agent tried to reboot |
||||
the machine in rescue mode which obviously didn't work since the mobo was toast, |
||||
and then decided that the machine was lost and *gave me a new one*. Which meant |
||||
that I didn't have access to my data anymore. |
||||
|
||||
I tried to have them plug the disk of the old machine in the new one, but they |
||||
"couldn't do that on this hardware" (I've since checked, and that hardware uses |
||||
2.5" SATA drives, which means they'd only have had to unplug the disk from the |
||||
old machine and put it in the new one. At the most, four screws might be |
||||
involved. But anyway.), so they told me that they were sorry but I'd have to |
||||
restore from my backups. |
||||
|
||||
Which, thankfully, I had! Complete backups from that same day, 4:15am. Obviously |
||||
the situation would have been much worse otherwise, and I thanked the day I had |
||||
decided to setup a sensible backup strategy. So I set to work on restoring |
||||
these. |
||||
|
||||
My backups are managed via duplicity; I have a setup where the first puppet run |
||||
on a server installs some basic backup definitions, and some more targeted |
||||
configuration once they're configured, depending on what they're used for. This |
||||
setup is described at the end of the post, if you're interested. |
||||
|
||||
Anyway, these are broken up into what duplicity calls "targets", which are |
||||
ensembles of folders that are backed-up with the same rules (frequency, time |
||||
before expiration, etc...). The main ones in my setup are `homedir`, which |
||||
includes... my home directory, yes; `conf_files`, which includes `/etc`, `/var`, |
||||
`/opt` and `/usr/local`; `srv_data`, which includes *most* of `/srv`, and |
||||
finally `mysql` and `pgsql`, which have a pre-run hook to dump the respective |
||||
databases and then backs them up. |
||||
|
||||
So, on the evening of the 30th, I started restoring these. After fiddling for |
||||
a bit to figure out how duplicity restores work, I started restoring the |
||||
`homedir` target. And that's when I found out that restoring data from an sftp |
||||
server running behind an ADSL connection takes *ages*, a fact that's only made |
||||
worse by the insistence of duplicity to copy to the remote the signature files |
||||
and indexes for *all* the full backups, and not just the latest ones |
||||
applicable. In this case, it took about three days. |
||||
|
||||
I managed to restore email first, as that was the most urgent, to avoid having |
||||
bounces (most MTAs retry for 3-5 days before giving up on delivery), and then |
||||
slowly walked my way back to restoring all of /var (including the cache, which |
||||
I had forgotten to exclude from my backups...), and /srv/pub, which holds |
||||
https://pub.wxcafe.net and https://wxcafe.net/pub, and which included (among |
||||
other things) a few HD movies, some taking over 4GB. |
||||
|
||||
Needless to say, this restore took a long time. I've learned a few lessons from |
||||
that whole thing, though: |
||||
|
||||
- never assume the hosting provider is gonna do the right thing, |
||||
- decide how much downtime you are willing to live with |
||||
- check your backups regularly and see how fast they restore |
||||
- define prioritized restoration targets (i.e. website and mail server. That |
||||
xmpp server can probably wait.) |
||||
- don't stress out too much about this. it's gonna be okay, and rebuilding can |
||||
always work. you'll find a solution. |
||||
|
||||
Anyways, in the end all I lost was a few months of my RSS subscriptions, which, |
||||
while annoying, is definitely something I can live with. It worked out alright |
||||
in the end. |
||||
|
||||
Now for that puppet/duplicity config... |
||||
|
||||
I use the very good |
||||
[puppet-duplicity](https://github.com/tohuwabohu/puppet-duplicity) module, |
||||
which defines most of what you need already. Then, it so happens that there's |
||||
a bug in the paramiko version most of my servers have, so I have taken to |
||||
replacing the file with that bug with a fixed version, which you can find |
||||
[here](https://git.wxcafe.net/snippets/13) |
||||
I then define a `backup` class, that can be used where ever it's needed in |
||||
host definitions: |
||||
|
||||
```ruby |
||||
## Puppet backups with duplicity |
||||
|
||||
# definitions |
||||
|
||||
class backups { |
||||
file { '/var/backups/mysql/': |
||||
ensure => directory, |
||||
} |
||||
|
||||
file { '/var/backups/pgsql/': |
||||
ensure => directory, |
||||
} |
||||
|
||||
class { 'duplicity': |
||||
backup_target_url => "sftp://censored//srv/backups/$hostname", |
||||
backup_target_username => 'duplicity', |
||||
backup_target_password => 'censored', |
||||
} |
||||
|
||||
## dirty hotfix |
||||
if $facts['os']['name'] == 'freebsd' { |
||||
file { '/usr/local/lib/python2.7/site-packages/duplicity/backends/_ssh_paramiko.py': |
||||
ensure => present, |
||||
content => file('base/backups/_ssh_paramiko.py'), |
||||
require => Package['duply'] |
||||
} |
||||
} else { |
||||
file { '/usr/lib/python2.7/dist-packages/duplicity/backends/_ssh_paramiko.py': |
||||
ensure => present, |
||||
content => file('base/backups/_ssh_paramiko.py'), |
||||
require => Package['duply'] |
||||
} |
||||
} |
||||
|
||||
if $facts['os']['name'] == 'freebsd' { |
||||
package {'py27-pip': |
||||
ensure => present, |
||||
} |
||||
package {'py27-cryptography': |
||||
ensure => present, |
||||
} |
||||
} else { |
||||
package {'python-pip': |
||||
ensure => present, |
||||
} |
||||
package {'python-cryptography': |
||||
ensure => present |
||||
} |
||||
} |
||||
|
||||
duplicity::profile { 'conf_file': |
||||
full_if_older_than => "2W", |
||||
max_full_backups => 3, |
||||
cron_hour => '05', |
||||
cron_minute => '20', |
||||
cron_enabled => true, |
||||
gpg_encryption => false |
||||
} |
||||
|
||||
duplicity::profile {'homedir': |
||||
full_if_older_than => "1M", |
||||
max_full_backups => 3, |
||||
cron_hour => '04', |
||||
cron_minute => '40', |
||||
cron_enabled => true, |
||||
gpg_encryption => false, |
||||
} |
||||
|
||||
duplicity::profile {'srv_data': |
||||
full_if_older_than => "1M", |
||||
max_full_backups => 3, |
||||
cron_hour => '05', |
||||
cron_minute => '35', |
||||
cron_enabled => true, |
||||
gpg_encryption => false |
||||
} |
||||
|
||||
duplicity::profile { 'pgsql': |
||||
full_if_older_than => "1W", |
||||
max_full_backups => 2, |
||||
cron_hour => '04', |
||||
cron_minute => '20', |
||||
cron_enabled => true, |
||||
gpg_encryption => false, |
||||
exec_before_content => 'sudo pg_dumpall -h 127.0.0.1 -U postgres -f /var/backups/pgsql/db.sql' |
||||
} |
||||
|
||||
duplicity::profile { 'mysql': |
||||
full_if_older_than => "1W", |
||||
max_full_backups => 2, |
||||
cron_hour => '04', |
||||
cron_minute => '20', |
||||
cron_enabled => true, |
||||
gpg_encryption => false, |
||||
exec_before_content => 'sudo mysqldump -pcensored --all-databases --result-file=/var/backups/mysql/db.sql' |
||||
} |
||||
|
||||
} |
||||
``` |
||||
|
||||
And then here's a sample from a node definition: |
||||
|
||||
```ruby |
||||
|
||||
node 'yoshi.wxcafe.net' { |
||||
$physical_location = "Illiad - DC2, Vitry-sur-Seine" |
||||
include base |
||||
include backups |
||||
|
||||
duplicity::file {'/var/backups/mysql/': |
||||
profile => 'mysql', |
||||
ensure => 'present' |
||||
} |
||||
|
||||
duplicity::file {'/var/backups/pgsql': |
||||
profile => 'pgsql', |
||||
ensure => 'present' |
||||
} |
||||
|
||||
duplicity::file {'/etc/': |
||||
profile => 'conf_file', |
||||
ensure => 'present' |
||||
} |
||||
|
||||
duplicity::file {'/var/': |
||||
profile => 'conf_file', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/usr/local/': |
||||
profile => 'conf_file', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/opt/': |
||||
profile => 'conf_file', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/lists/': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/mail/': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/pub/': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/rpg/': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/wallabag/': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/srv/www': |
||||
profile => 'srv_data', |
||||
ensure => present |
||||
} |
||||
|
||||
duplicity::file {'/home/': |
||||
profile => 'homedir', |
||||
ensure => present, |
||||
} |
||||
} |
||||
|
||||
``` |
||||
@ -0,0 +1,105 @@ |
||||
Title: [N]olife |
||||
Date: 2018-04-23T11:00+02:00 |
||||
Author: Wxcafé |
||||
Category: |
||||
Slug: nolife |
||||
|
||||
Exceptionally, this post will be in french, because it's about a very |
||||
french-specific cultural phenomenon. `Nolife` was a french TV station that started |
||||
out in 2007, was centered on "geek, nerd and otaku" culture (but was done by |
||||
real enthusiasts), was run with almost no money nor means, and always stayed |
||||
true to the people who were running it. It died because of lack of funds after |
||||
11 years of broadcasts on the 8th of April, 2018. In this column I'm going to |
||||
talk about my personal relationship with it. |
||||
|
||||
Bon, ça y est, `Nolife` est morte. Ça faisait longtemps que ça devait arriver, |
||||
depuis les débuts de la chaîne, en fait, mais... au bout d'un moment, après |
||||
l'avoir vue traverser toutes ces morts annoncées certaines et en ressortir sans |
||||
une égratignure, voire parfois plus grande encore, on pouvait finir par se dire |
||||
que ça n'arriverait en fin de compte jamais, que les personnes qui font tourner |
||||
la chaîne arriveraient toujours à trouver des solutions pour qu'elle continue. |
||||
|
||||
Il se trouve que non. |
||||
|
||||
J'ai bien conscience que pour beaucoup de gens, `Nolife` se situe entre "juste une |
||||
chaîne de télé" et "c'était sympa mais voilà quoi", et c'est tout a fait normal |
||||
d'avoir ce point de vue. Mais `Nolife`, ça a été pour moi quelque chose d'un peu |
||||
différent. |
||||
|
||||
J'ai découvert nolife quand j'étais en lycée, en seconde en fait. Je connaissais |
||||
déjà un peu les cultures traitées par la chaîne, parce que je lisais pas mal de |
||||
fantasy et de SF, et que je m'intéressais beaucoup au fonctionnement des |
||||
ordinateurs, et aux jeux vidéos; et j'avais déjà un certain contact avec des |
||||
communautés liées, notamment le [Netophonix](https://www.netophonix.com/) (une |
||||
communauté dédiée aux sagas mp3). J'ai du coup assez vite accroché a `Nolife` et |
||||
aux émissions qui étaient présentes sur la chaîne, et ça a été une porte (parmi |
||||
d'autre, mon groupe d'ami.e.s de l'époque ayant aussi pas mal aidé) vers la |
||||
culture japonaise. C'était vers 2010, donc pas longtemps après la création de |
||||
`Nolife` Online (le service de VOD créé par la chaîne a l'époque), et comme |
||||
je n'avais pas énormément la possibilité de regarder la télé aux moments ou les |
||||
émissions (autres que des clips musicaux, donc, qui prenaient 50% du temps |
||||
d'antenne) passaient, j'ai rapidement décidé de m'abonner, et a partir de ce |
||||
moment là ma consommation des programmes de `Nolife` ne passait quasiment que par |
||||
`Nolife` Online. |
||||
|
||||
Comme dit, j'avais un contact avec le Netophonix, notamment par le channel IRC |
||||
sur FreeNode. Logiquement, quand j'ai voulu connaître un peu la communauté qui |
||||
traînait autour de `Nolife`, je suis tout de suite allé trouver le channel IRC |
||||
associé. Et sur ce channel, j'ai recontré pas mal de gens, qui a leur tour et de |
||||
leur propre façon m'ont fait découvrir de nombreuses choses, techniques ou non, |
||||
dont l'exemple le plus parlant aujourd'hui est Twitter, qui bien qu'étant un |
||||
réseau social... assez imparfait, a tout de même eu le mérite de me faire |
||||
rencontrer énormément de gens qui ont été essentiels a la façon dont j'ai évolué |
||||
en grandissant. Sans ces gens, je ne serais probablement pas éveillé a de |
||||
nombreuses causes qui sont particulièrement importantes pour moi aujourd'hui (et |
||||
ont participé a faire de moi la personne que je suis), notamment le féminisme, |
||||
les luttes LGBT, et les luttes de classe. Je n'aurais pas découvert des domaines |
||||
entiers, de littérature, de cinéma, de jeux vidéos, des domaines techniques, ... |
||||
|
||||
Alors, bien entendu, `Nolife` en soit ne m'a pas apporté tout cela. Sa communauté, |
||||
et les gens que j'ai découvert par les amis que je me suis faits dans cette |
||||
communauté, m'ont permis de me construire de cette façon, et la chaîne n'a eu |
||||
qu'un apport mineur a ce sujet. Cependant, elle a tout de même eu des intérêts |
||||
intrinsèques. En produisant énormément d'émissions qui **jamais** n'auraient pu |
||||
être diffusées sur d'autres chaînes du paysage audiovisuel français (une |
||||
émission parlant du hardware des vieilles consoles en partenariat avec |
||||
[Mo5.com](http://mo5.com)? Une émissions expliquant en détail le fonctionnement |
||||
de bugs utilisés par des superplayers et des speedrunners dans des jeux? ... Un |
||||
top 5 de J-Music? Et même quelque chose qui semble évident aujourd'hui, mais... |
||||
des let's play a la télé en 2007? Une émission sur l'e-sport, en 2010? Et il |
||||
y a plein d'autres exemples...), cette chaîne a bien montré qu'en se battant, on |
||||
pouvait faire des choses que tout le monde pensait impossible. En montrant des |
||||
gens passionnés, pas forcément parfaitement a l'aise devant une caméra mais |
||||
prêts a se mettre sur le devant de la scène pour présenter des choses qu'iels |
||||
aiment, elle a permit a beaucoup de gens d'accepter et d'assumer leurs passions. |
||||
Elle a démocratisé de nombreux sujets qui jusqu'ici étaient réservés à des |
||||
communautés minuscules (je pense notamment aux speedruns, aux démos, a l'e-sport |
||||
même, a la musique japonaise...). En diffusant des émissions japonaises ou |
||||
parlant de la culture japonaise, non pas en envoyant des occidentaux visiter le |
||||
japon et en rapporter ce qu'ils voulaient bien en montrer au public occidental, |
||||
mais en diffusant des programmes japonais entièrement traduits (et sous-titrés, |
||||
pas doublés!), type Japan in Motion, ou en employant des personnes japonaises |
||||
prêtes a montrer ce qu'elles connaissaient du japon (Tokyo Café), voire en |
||||
interviewant des personnes sur place pour qu'elles présentent leur vision de |
||||
leur quartier ou de leur ville et de la culture qui l'accompagnait (Toco Toco). |
||||
|
||||
Bref, en donnant accès, a moi et a énormément d'autres personnes, non seulement |
||||
a ces cultures et a ces sujets fermés, mais aussi a la passion claire des gens |
||||
qui participaient a la chaine envers les sujets qu'iels traitaient, cette chaîne |
||||
a fait beaucoup pour la construction de beaucoup de personnes dans mon cas, et |
||||
a ouvert une fenêtre dans le paysage audiovisuel français. |
||||
|
||||
Cette fenêtre s'est refermée voici deux semaines. Je n'ai maintenant plus aucun |
||||
regret a voir la télévision mourir. |
||||
|
||||
Je voudrais par ce post remercier les personnes qui ont fait `Nolife`, devant ou |
||||
derrière (ou a côté de) la caméra. Je peux citer celles et ceux que je connais, |
||||
certains avec lesquels j'ai pu prendre un verre au Kawaii café ou au Dernier Bar |
||||
une fois ou deux (je n'étais pas encore majeur, je peux vous le dire |
||||
maintenant), Mathilde, Seb et Alex évidemment, Medoc et Moguri, Thierry, |
||||
Sylvain, Clément, Caroline, Benoît, Anne, Julien Pirou, Pili, Slimane, Radigo, |
||||
Mickey, Macha, DamDam, Obliv, Josaudio... |
||||
|
||||
Merci pour ces 11 ans, merci pour votre acharnement a faire cette chaîne. Rien |
||||
ne sera jamais tout a fait pareil... Mais heureusement, y a pas que la vraie vie |
||||
dans la vie. |
||||
@ -0,0 +1,57 @@ |
||||
================================================================== |
||||
https://keybase.io/wxcafe |
||||
-------------------------------------------------------------------- |
||||
|
||||
I hereby claim: |
||||
|
||||
* I am an admin of https://wxcafe.net |
||||
* I am wxcafe (https://keybase.io/wxcafe) on keybase. |
||||
* I have a public key ASATUYKHRl5bGwMh3n0KNIruvG-ivARCUOhsgIOTq3Ayfwo |
||||
|
||||
To do so, I am signing this object: |
||||
|
||||
{ |
||||
"body": { |
||||
"key": { |
||||
"eldest_kid": "01205da51ba3f465c9f4df080b8583b497e87670e60493f353ae0128b888520b77df0a", |
||||
"host": "keybase.io", |
||||
"kid": "012013518287465e5b1b0321de7d0a348aeebc6fa2bc044250e86c808393ab70327f0a", |
||||
"uid": "e3254eba7cdb3de6a427f80ab715cb19", |
||||
"username": "wxcafe" |
||||
}, |
||||
"merkle_root": { |
||||
"ctime": 1525542893, |
||||
"hash": "5ab026b8dcb33a11f3128ce41929a6a6494ddd1c5e44392f39ce3c558da4120b59c8d23c55232a2e5aa09d1997b87e3411f1f5ce70af8ed17c1b36b43b38c1a9", |
||||
"hash_meta": "2d86b87f9588d335c9325dbf95bf2fea52010ed271f23177572977e21593fdc1", |
||||
"seqno": 2489929 |
||||
}, |
||||
"service": { |
||||
"entropy": "DxIM4bWz8Ezd/Is8ltCpjK8u", |
||||
"hostname": "wxcafe.net", |
||||
"protocol": "https:" |
||||
}, |
||||
"type": "web_service_binding", |
||||
"version": 2 |
||||
}, |
||||
"client": { |
||||
"name": "keybase.io go client", |
||||
"version": "1.0.47" |
||||
}, |
||||
"ctime": 1525542918, |
||||
"expire_in": 504576000, |
||||
"prev": "483b1bcfee9f94fc9bf512ab56a679ae297e336991443f761fc585893ff77ec5", |
||||
"seqno": 36, |
||||
"tag": "signature" |
||||
} |
||||
|
||||
which yields the signature: |
||||
|
||||
hKRib2R5hqhkZXRhY2hlZMOpaGFzaF90eXBlCqNrZXnEIwEgE1GCh0ZeWxsDId59CjSK7rxvorwEQlDobICDk6twMn8Kp3BheWxvYWTESpcCJMQgSDsbz+6flPyb9RKrVqZ5ril+M2mRRD92H8WFiT/3fsXEIDnrmqfzoylRfSLJXT7+sAWGU/pBJhpzdvujtz+iBTTXAgHCo3NpZ8RAaLXPWzACNLZPAdvl53TVjvicOfKTBoR8Ewv0wxQbcS/syGxw5HdWaL1heUbWJYjeu2ktQXXMjRBHUne88e/1AKhzaWdfdHlwZSCkaGFzaIKkdHlwZQildmFsdWXEIFDUsxH3LBt2V95xdAykBXAkonHJmdpkr4hY3VoOmJgJo3RhZ80CAqd2ZXJzaW9uAQ== |
||||
|
||||
And finally, I am proving ownership of this host by posting or |
||||
appending to this document. |
||||
|
||||
View my publicly-auditable identity here: https://keybase.io/wxcafe |
||||
|
||||
================================================================== |
||||
|
||||
Loading…
Reference in new issue