Sysadmin tasks

Dear systers,

For a long time now, there have been rumours about SysAdmins, creatures who work in the shadows, avoid sunlight, enjoy Korean cuisine and are pure nerds. You aspire to join this magnificent community and you feel ready to dive into this dark, joyful, mysterious world... But you wonder:
What do SysAdmins actually do? What are their responsibilities and everyday tasks?

Here you will find an attempt for slow, ongoing documentation of some essential SysAdmin tasks written by feminists. Some of us have long experience in this kind of work, while others are new to this. Together, we enjoy exploring SysAdmin work, collectively educating ourselves and making efforts to maintain our feminist infrastructures: Systerserver and Anarchaserver.

REBOOT

As a Sysadmin, one of the frequent activities is to reboot a server.

-That looks cool, but hey, I have a dull question! Why do we need to reboot the server? When do we know that this is something we have to do?
-A server reboot could be essential because of regular maintenance activity such as software and operating system (OS) updates for security reasons.
-Do we have to do these updates/upgrades manually or are they automated processes?
-There are Automatic Security Updates. Administrators receive an email when this happens, so we know when to perform the reboot/shutdown. Would you like to be added to this list, so you also receive these emails?
-Yes, let's do that!

Sysadmins emails should be added in /etc/aliases

# vi /etc/aliases
root: admin1@domain.net, admin2@domain.net, admin3@domain.net

WHY: The server sends emails for the security updates that have been installed overnight and inform the sysadmins if a reboot is necessary.

REQUIRES: that we have enabled unattended updates, see how-to: https://wiki.debian.org/UnattendedUpgrades

OPTIONAL: In order to receive the latest Debian security advisories, subscribe to the debian-security-announce mailing list.

Server reboot

PREPARATIONS: Inform the community that a reboot would take place and who wants to join, preferably do it with a fellow sysadmin rather than alone.

COMMANDS:

List and save all services currently running in a file with a timestamp in the name, before and after the reboot:

# systemctl list-units --type=service --state=running,exited,failed > DD-MM-YY-beforeboot

ref: https://linuxhint.com/list-all-running-services-debian/

# shutdown -r +2

the flag -r is for reboot
the flag +m will execute the shutdown after 2min. Handy to make sure users can logout before shutting down

When server is back again, save the list of services and check for differences

# systemctl list-units --type=service --state=running,exited,failed > DD-MM-YY-aftereboot
# diff service-<timestamp>-beboreboot DD-MM-YY-afterboot > diff-services
# less diff-services

If a service has not started, it would be listed in the diff-services file. Start the service and check its status (example):

# systemctl start apache
# systemctl status apache

We can use the tab completion for finding the exact name of a service

ALTERNATIVES:

systemctl reboot

We didn't find a flag to add specific time of the reboot execution, so we opted for the shutdown command to make sure that we have time to finish any pending tasks we run and to exit the server.

DEBUG

journalctl -u peertube -x 
journalctl -u peertube -x | grep "err"

TROUBLESHOOTING

...

UPDATE SSL CERTIFICATES

...

UPDATE SOFTWARE

...

GitLab