Video Tutorials for SysAdmins

Download Video (497 KB)

This video describes how to create a private Gentoo Linux mirror of Portage Tree and source files on your server. Also, the tutorial shows how to update mirror using cron and rsync, how to serve the mirror via HTTP (with apache), via FTP (with vsftpd) and via RSYNC (with rsyncd) and how to set up the clients to use the mirror.

Video tutorial consists from 5 parts:
Part 1. Sync portage tree and Gentoo source files (distfiles, releases, snapshots etc.) with one of the official Gentoo rsync servers (03:40 min)
Part 2. Make mirror accesible via RSYNC (05:30 min)
Part 3. Make Gentoo source files accesible via HTTP (04:38 min)
Part 4. Make Gentoo source files accesible via FTP (03:17 min)
Part 5. Clients setup (02:07 min)

Installation steps.
Part 1. Sync portage tree and Gentoo source files (distfiles, releases, snapshots etc.) with one of the official Gentoo rsync servers

1. First, check if installed rsync has latest version. If itsn't upgrade it:

   emerge --sync
   emerge -av rsync

2. Create a new user with no login privileges that will be used for sync operations, and as the user to serve the files via HTTP, FTP and RSYNC:

   useradd -m -s /sbin/nologin mirror

3. Make dirs for our Gentoo mirror:

   sudo -u mirror mkdir /home/mirror/gentoo-portage
   sudo -u mirror mkdir /home/mirror/gentoo

4. Sync Portage tree from one of of the official Gentoo rsync mirrors:

   sudo -u mirror rsync --verbose --recursive --links --perms \
   --times --devices --delete --timeout=300 \
   rsync://trumpetti.atm.tut.fi/gentoo-portage \
   /home/mirror/gentoo-portage

5. Sync Gentoo files (distfiles, releases, snapshots etc.) from one of of the official Gentoo rsync mirrors. Note: you need to have at least 50-60GB free on your HDD:

   sudo -u mirror rsync --verbose --recursive --links --perms \
   --times --devices --delete --timeout=300 \
   rsync://trumpetti.atm.tut.fi/gentoo /home/mirror/gentoo

6. Create a cron job to sync mirror every night:

   vi /etc/crontab

   # Sync Portage Tree every night at 22.00
   0 22 * * * mirror rsync -vrlpt --delete --timeout=300 rsync://trumpetti.atm.tut.fi/gentoo-portage /home/mirror/gentoo-portage
   

   # Sync Gentoo source files every night at 23:45
   45 23 * * * mirror rsync -vrlpt --delete --timeout=300 rsync://trumpetti.atm.tut.fi/gentoo /home/mirror/gentoo
   

Part 2. Make mirror accesible via RSYNC

1. Create a new configuration file for rsync in daemon mode:

   mv /etc/rsyncd.conf /etc/rsyncd.conf.old
   vi /etc/rsyncd.conf

   # Specify a "message of the day" file to display
   # to clients on each connect
   motd file = /etc/rsyncd.motd
   

   # The "pid file"
   pid file = /var/run/rsyncd.pid
   

   # Specify the file to use to support the
   # "max connections" option
   lock file = /var/run/rsyncd.lock
   

   # Enable the rsyncd to run in chroot
   use chroot = yes
   

   # Specify the maximum number of simultaneous
   # connections to 10
   max connections = 10
   

   # Log messages to log file rather than using syslog
   # The file is opened before chroot() is called
   log file = /var/log/rsyncd.log
   

   # Prevent users to upload files
   read only = yes
   

   # Permissions (run with mirror username)
   uid = mirror
   gid = mirror
   

   # Limit access to specified hosts or LAN's
   hosts allow = 127.0.0.1 172.16.50.4 172.16.50.63
   

   [gentoo-portage]
   path = /home/mirror/gentoo-portage
   comment = Gentoo Portage Tree mirror
   

   [gentoo]
   path = /home/mirror/gentoo
   comment = Gentoo Source mirror
   

2. Create a "message of the day" file for rsyncd:

   vi /etc/rsyncd.motd

   Welcome to Video 4 Admin dot com Gentoo rsync mirror

   Server Address : 172.16.50.4
   Contact Name : root[no_spam]video4admin.com
   Hardware : 2 x Intel(R) Pentium(R) D CPU 2.80GHz, 1024MB RAM

3. Run rsync as daemon and watch logs to verify config file:

   rsync --daemon --config=/etc/rsyncd.conf -v --no-detach
   tail -f /var/log/rsyncd.log
   rsync --list-only 127.0.0.1::
   rsync --list-only 127.0.0.1::gentoo-portage

4. Configure logrotate for rotating the rsyncd logs:

   vi /etc/logrotate.d/rsyncd

   /var/log/rsyncd.log {
   monthly
   # rotate the logs 50 times
   # before removing the old logs
   rotate 50
   # copy and truncate the original log file
   # in place instead of renaming it and
   # creating a new logfile
   copytruncate
   compress
   # Do not rotate the log if it is empty
   notifempty
   missingok
   sharedscripts
   }
   

5. Start rsyncd and add it to runlevel default:

   /etc/init.d/rsyncd start && rc-update add rsyncd default

Part 3. Make Gentoo source files accesible via HTTP

1. Compile apache and apache-tools without ssl support and use the newer threaded Multi-Processing Module called worker for apache:

   echo www-servers/apache -ssl mpm-worker >> /etc/portage/package.use
   echo app-admin/apache-tools -ssl >> /etc/portage/package.use

2. Install apache:

   emerge -av apache

3. Tune apache. Edit options as follows:

   vi /etc/apache2/modules.d/00_default_settings.conf

   Timeout 60
   MaxKeepAliveRequests 256
   UseCanonicalName On
   ServerSignature Off
   <Directory />
   Options -FollowSymLinks
   AllowOverride None
   Order deny,allow
   Deny from all
   </Directory>
   

4. Configure virtual host for Gentoo source files:

   vi /etc/apache2/vhosts.d/gentoo.video4admin.com.conf

   <VirtualHost gentoo.video4admin.com:80>
   ServerAdmin root[no_spam]gentoo.video4admin.com
   DocumentRoot /home/www/gentoo
   ServerName gentoo.video4admin.com
   ErrorLog /var/log/apache2/gentoo.video4admin.com/error_log
   TransferLog /var/log/apache2/gentoo.video4admin.com/access_log
   CustomLog /var/log/apache2/gentoo.video4admin.com/httpd-access_log combined
   <Directory "/home/www/gentoo">
   Options Indexes SymLinksIfOwnerMatch
   AllowOverride None
   Order allow,deny
   Allow from all
   </Directory>
   </VirtualHost>
   

5. Create log dir for virtual host gentoo.video4admin.com:

   mkdir /var/log/apache2/gentoo.video4admin.com
   chown apache:apache /var/log/apache2/gentoo.video4admin.com
   chmod 600 /var/log/apache2/gentoo.video4admin.com

6. Start apache and verify if mirror is accesible via browser:

   /etc/init.d/apache2 start
   links gentoo.video4admin.com

If all it's OK - add apache to runlevel default:

rc-update add apache2 default

Part 4. Make Gentoo source files accesible via FTP

1. Very Secure FTP Daemon (vsftpd) will serve as FTP server.
   For vsftpd I will enable TCP wrappers, capability to control privileges and Pluggable Authentication Modules. Also, for this daemon, I will use logrotate for rotating logs and disable ssl support:

   echo net-ftp/vsftpd caps logrotate pam tcpd -ssl \
   >> /etc/portage/package.use

2. Install FTP server:

   emerge -av vsftpd

3. Configure vsftpd:

   mv /etc/vsftpd/vsftpd.conf /etc/vsftpd/vsftpd.conf.old
   vi /etc/vsftpd/vsftpd.conf

   # Listen (non-xinetd) mode
   listen=YES
   

   # Use tcp_wrappers to control connections
   tcp_wrappers=YES
   

   # Use localtimes instead of GMT for files
   use_localtime=YES
   

   # Hide the true user/group ID of files, display as "ftp"
   hide_ids=YES
   

   # Permit anonymous logins
   anonymous_enable=YES
   

   # Use "mirror" user for anonymous logins
   ftp_username=mirror
   

   # User to run vsftpd as (same as ftp_username)
   nopriv_user=mirror
   

   # Chroot directory for anonymous user
   anon_root=/home/mirror/gentoo
   

   # Prevents vsftpd from asking for an anonymous password
   no_anon_password=YES
   

   # Enable recursive "ls" listing (allow the use of "ls -R")
   ls_recurse_enable=YES
   

   # Destroy sessions after 120 seconds of inactivity
   idle_session_timeout=120
   

   # Stop sending data after 30 seconds of inactivity
   # during a transfer
   data_connection_timeout=30
   

4. Start vsftpd and add it to runlevel default:

   /etc/init.d/vsftpd start && rc-update add vsftpd default

5. Verify if mirrorr is accesible via FTP:

   links ftp://gentoo.video4admin.com

Part 5. Clients setup

1. Edit make.conf:

   vi /etc/make.conf

2. The SYNC variable overrides the default location where Portage looks for the Portage tree updates:

   SYNC="rsync://gentoo.video4admin.com/gentoo-portage"

3. Specify mirror to use for source files retrieval:

   GENTOO_MIRRORS="http://gentoo.video4admin.com ftp://gentoo.video4admin.com"

4. Test1 for mirror: try to sync the Portage Tree:

   emerge --sync

5. Test2 for mirror: try to fetch files:

   emerge -fav gentoo-sources