Depooling and repooling servers
From Blitzed
This article covers how to depool and repool an IRC server so that it either ceases to be available or returns to availability in the irc.blitzed.org DNS pool.
Contents |
[edit] Requirements
To do this you will need to:
- be a member of the systems team
- have shell access to lamancha
- have CVS write access
If you are a systems team member who does not meet the rest of the requirements but you are still interested in being able to de/repool servers, please submit a ticket to RT's systems queue. Be aware that the addition to the list of systems team members does not happen automatically, just drop the others a mail if you are not yet added. You do no longer need to be in the wheel group, as long as you have got the additional sudo privileges which result out of this list in sudoers.
All de/repooling servers needs to be done with scripts from the blitzed-dns module in CVS, in your checkout of that module, on lamancha.
[edit] Depooling
Depooling servers is at the discretion of systems team members. It should be done whenever a server has been absent from the network for a reasonable period of time (say, more than 15 minutes), or when a server is unable to stay connected reliably. It is also used in advance of scheduled downtime.
The process goes something like this:
- Change directory to your checkout
- Run ./depool server, example:
$ ./depool lik-m-aid
- Execute ./update.pl if the previous step is successful, to update the zone file.
- A CVS commit will now take place. Please give a sensible log message explaining which server you have depooled and why.
[edit] Repooling
Do not be too eager to repool servers - we have enough that some can stay depooled longer than strictly necessary with no problem. You would typically do this after a server has returned to the network without issue for several hours.
We have a script that checks our list of pooled servers against those that are currently linked and reports any that may need repooling. The ./depool script takes an optional argument for the number of hours to ignore the server for, so that it is possible to depool and ignore a server for a period of time even when it links itself back to the network.
Therefore, it is generally best not to repool servers unless
- the script complains that they should be repooled; or
- you know they should be.
The process is exactly the same as for depooling, except the ./repool script is used.
[edit] Monitoring
We have scripts in place to warn us when servers may need to be de/repooled. Every hour, scripts will look to see which servers are linked, and advise about repooling any servers which are linked but not in DNS.
Sometimes servers are able to remain connected to the network but are still too unreliable for users, or need to be depooled in advance of scheduled downtime. In these cases it can be useful to specify the number of hours that a server is expected to be depooled for:
$ ./depool lik-m-aid 72
will depool lik-m-aid.ca.us.blitzed.org and will not warn about repooling it again for 72 hours, whether it is linked or not.