Creating a Local Network Repository for Linux Systems

(:title Creating a Local Network Repository for Linux Systems:)

Linux systems have a natural ability to use a variety of installation sources.

A repository is a location on the network which houses all the source files needed by the system to install and update software. Typically these repositories are located on the internet, but they can be copied to the local network so you don't have 100's of workstations hitting your internet connection, trying to stay up to date. This also give you the benefit of being able to control exactly what is in the repository.

The advantages of a repository are obvious:

1. You don't need to carry around all those CDs for all your applications.
2. When you update a repository it is available instantly to all the client machines.
3. With the speedy 1GB networks nowadays, network utilization is no longer an issue!

SuSE and Novell have a couple of solutions. Novell has ZENWorks Linux Management, while SuSE has a YaST Utility called "Installation Server". However for a recent project I needed to have both copies of multiple DVDs, and mirror internet repositorys to a local source. (client machines do not have access to the internet) Unfortunately neither of the ready made solutions exactly fit the bill, so I made my own. For this project I had to do a good deal of research to figure out what exactly I was doing :) Hopefully my research and experience with this will help others (and myself if I ever need a refresher)

The goal of this project is to create a local repository with the following properties:

1. I must have the ability to use data from any number of local media sources.
2. I must have the ability to create any number of my own custom repositories.
3. I must have the ability to mirror numerous repositories on the internet.
4. It must work behind a proxy.
5. The repositories must be available via HTTP and FTP:
6. The repositories should be advertised via SLP.

While starting to break the problem down, we find that we really have three different types of repositories:

1. Those which we copy directly off of media (i.e. CD or DVD)
2. Those which we build ourselves (i.e. A collection of RPMs)
3. Those which we copy (or mirror) from internet locations.

The next problem is that during all the mirroring and whatnot, the actual directory structure and file names can become very messy, so we will also what to create a "public" directory with links to the actual locations. This way the actual specifics and mechanics of what is going on is hidden from the users and junior administrators. To to start with I choose to use the following directory structure for our repository server:

(:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

 / (root directory)                                                         
 |--repodata                                                                
 |   |--rawdata   
 |   |   |--SLED10                                                          
 |   |   |   |--data copied from SLED10 ISO file 
 |   |   |--Custom
 |   |       |--RPMs either custom built or collected separately
 |   |                                 
 |   |--mirrors                                                             
 |       |--various ftp and http sites with their own structures             
 |
 |--repository                                                              
     |--a number of links to the data in the repodata directory (regardless 
         of how deep the data resides in the tree)                          

(:divend:)

Now that we have the structure, we need to copy the data. The rawdata directory is just a collection of folders with the contents of the CDs, DVDs and custom folders. This will save us from having to carry a source disk with us everytime we touch a workstation. The mirrors directory is the base directory for the mirrored sites. For example, if we mirror a repository (I will use packman for an example) http://packman.iu-bremen.de/suse/10.1 we will have a directory structure of /repodata/mirrors/packman.iu-bremen.de/suse/10.1 And since we could be mirroring a number of sites, each with their own directory structure, you will see why you need to have a separate directory which only contains links to the actual data at the end of each structure.

To do all of the heavy-lifting I created a BASH script. My script uses a separate configuration file to store all the repository information. A sample configuration file is below. One important fact I soon discovered is to make sure that you do not try to mirror directly from the SuSE site!! Unless you are an approved "mirror", you will get redirected to another site. Normally this is not a problem, but because the file is placed in a specific path based on where it is downloaded from, you will find yourself in a directory-hell really quick! (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

##############################################################################
##############################################################################
#                                                                            #
# /etc/repository/repo.sources                                               #
#                                                                            #
# This config file has the following format:                                 #
#  Three columns using TAB as the delimitation character                     #
#  The first column contains the local name of the repository.  This name    #
#   must be unique.                                                          #
#  The second column is either the remote repository's URL (for mirrored     #
#   repositories) or the local repository's absolution location.             #
#  The third column specifies whether the source is to be advertized via     #
#   SLP or not. ( either SLP or noSLP)                                       #
#  The fourth column is only used if the repository in question is a custom  #
#   repository. (A group of RPMs compiled by the administrator for any       #
#   number of different locations)                                           #
#                                                                            #
# All URLs needs to have a trailing slash for the process to work properly.  #
#                                                                            #
# Comments can be used but must be preceeded by a hash symbol (#)            #
# All Comments are to be preceded by a hash symbol (#) (Maximum of 78 chars) #
#                                                                            #
##############################################################################
##############################################################################


# OPW Auto YaST XML documents
autoyast	/repodata/rawdata/autoyast/	noSLP


# Customer Specific RPMs
Custom	/repodata/rawdata/Custom/	SLP	Custom


# Windows Source CDs (Based on http://unattended.sourceforge.net)
Windows	/repodata/rawdata/unattended-4.6/install/	noSLP


# SuSE Linux Enterprise Desktop 10 Source Directory
SLED10-DVD	/repodata/rawdata/SLED10/	noSLP


# SuSE Linux Enterprise Server 10 Source Directory
SLES10-DVD	/repodata/rawdata/SLES10/	noSLP


# SuSE Linux Enterprise Desktop 10 Source Directory
SLED10sp1-DVD	/repodata/rawdata/SLED10sp1/	SLP


# SuSE Linux Enterprise Server 10 Source Directory
SLES10sp1-DVD	/repodata/rawdata/SLES10sp1/	SLP


# Novell Client for SLE10
Novell-Client	/repodata/rawdata/novell-client-1.2-SLE10/	SLP


# Novell Client for SLE10
Novell-Client-2.0	/repodata/rawdata/novell-client-2.0-SLE10/	SLP


# Novell OES2
OES2	/repodata/rawdata/OES2-Beta5/	SLP


# nVidia SuSE Repository
#nVidia	http://download.nvidia.com/	noSLP


# SuSE 10.1 Update Repository
#  We want to download from the following site: http://download.suse.com/update/10.1/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp5.gwdg.de/
SuSE10.1-Update	ftp://ftp5.gwdg.de/pub/suse/update/10.1/	SLP


# SuSE OpenOffice.org for SLED 10 Repository
#  We want to download from the following site: http://software.opensuse.org/download/OpenOffice.org:/STABLE/SLED_10/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp5.gwdg.de/
OOo-SLED10	ftp://ftp5.gwdg.de/pub/opensuse/repositories/OpenOffice.org:/STABLE/SLED_10/	SLP


# SuSE OpenOffice.org Extras for SLED 10 Repository
#  We want to download from the following site: http://software.opensuse.org/download/OpenOffice.org:/EXTRAS/SLED_10/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp5.gwdg.de/
OOo-SLED10-Extras	ftp://ftp5.gwdg.de/pub/opensuse/repositories/OpenOffice.org:/EXTRAS/SLED_10/	SLP


# SuSE Mozilla for SLE 10 Repository
#  We want to download from the following site: http://software.opensuse.org/download/mozilla/SLE_10/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp5.gwdg.de/	
Mozilla-SLE10	ftp://ftp5.gwdg.de/pub/opensuse/repositories/mozilla/SLE_10/	SLP



# Packman's SuSE 10.1 Repository
Packman-SuSE10.1	ftp://packman.iu-bremen.de/suse/10.1/	SLP


# SuSE OSS Install Source for SuSE 10.1 Repository
#  We want to download from the following site: http://download.opensuse.org/distribution/SL-10.1/inst-source/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp.heanet.ie/
SuSE10.1-OSS	ftp://ftp.heanet.ie/mirrors/ftp.opensuse.org/opensuse/distribution/SL-10.1/inst-source/	SLP


# SuSE non-OSS Install Source for SuSE 10.1 Repository
#  We want to download from the following site: http://download.opensuse.org/distribution/SL-10.1/non-oss-inst-source/
#  but this site is redirects http requests to other sites so we need to 
#  download from an authorized mirror http://ftp.heanet.ie/
SuSE10.1-NONOSS	ftp://ftp.heanet.ie/mirrors/ftp.opensuse.org/opensuse/distribution/SL-10.1/non-oss-inst-source/	SLP

(:divend:)

The next step was to create the actual script. This script need to be able to perform six specific functions:

1. Mirror the sites from the internet
2. Remove links in the public directory to the data
3. Create links in the public directory to the data
4. Within the Custom repositories, read the RPM header information and create the necessary files and sign the repository
5. Change the ownership and mode of the files, so that they can be read by public users via FTP and HTTP.
6. Modify the SLP configuration files so the repositories will be advertised by SLP

As always, I pushed the boundaries and perhaps went overboard, but the my script is below. One of the important details I discovered during this project was that the Symbolic Links I orginally used in the public directory to point to the data did not work in any of the FTP servers I tried (ProFTPd, Pure-FTPd, vsFTPd) I am still not sure why exactly since it works in every other protocol, but the basic issue is that the Symbolic Links (Symlinks) are actually viewed through the FTP client as a shortcut pointing to the absolute position of the data directory but the root (as viewed through the FTP client) as been moved. For example: If I have a Symlink in the repository directory pointing to the /repodata/rawdata/SLED10 directory, the FTP client will try to access the physical /repository/repodata/rawdata/SLED10 which obviously does not exist!! This was extremely frustrating!! The only work around I could find was to use the bind mount command (mount --bind). This is basically a hardlink for directories. It made the code in my script a little more complex but in the end I did it. (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#!/bin/bash
##############################################################################
##############################################################################
#                                                                            #
# repository.sh                                                              #
#                                                                            #
# This script is designed to perform three functions:                        #
#   1. Mirror remote repositories locally                                    #
#   2. Create links from the data location to the visible directory (This    #
#      isolates the end users from the behind the scenes working of the      #
#      server)                                                               #
#   3. Update custom repositories using the contents of it RPMs              #
#                                                                            #
# The basic idea is to have two repository locations. The first is where the #
#  actual data resides.  The second is a directory which only contains links #
#  to the data sitting in the first.  This way the actual structure of the   #
#  data is hidden from the end-users so all they see is a list of            #
#  repositories for them to use.  The structure I have used is below:        #
#                                                                            #
# / (root directory)                                                         #
# |--repodata                                                                #
# |   |--rawdata                                                             #
# |   |   |--data copied from ISO file                                       #
# |   |   |--custom RPMs                                                     #
# |   |--mirrors                                                             #
# |       |--various ftp and http sites with their own structures            #
# |--repository                                                              #
#     |--a number of links to the data in the repodata directory (regardless #
#         of how deep the data resides in the tree)                          #
#                                                                            #
# There are a number of command line switches that can be used with this     #
#  script to see all the options use the -? or --help options.               #
#                                                                            #
#                                                                            #
# This script uses a config file which has the following format:             #
#  Three columns using TAB as the delimitation character                     #
#  The first column contains the local name of the repository.  This name    #
#   must be unique.                                                          #
#  The second column is either the remote repository's URL (for mirrored     #
#   repositories) or the local repository's absolution location.             #
#  The third column is only used if the repository in question is a custom   #
#   repository. (A group of RPMs compiled by the administrator for any       #
#   number of different locations)                                           #
#                                                                            #
# Comments can be used but must be preceeded by a hash symbol (#)            #
# All Comments are to be preceded by a hash symbol (#) (Maximum of 78 chars) #
#                                                                            #
##############################################################################
##############################################################################

# This is if you need to use a proxy, if there is no proxy, then comment this
#  line out.
default_http_proxy="http://proxy.customer.domain:80"

# These are the default parameters, change as needed.  The directories need 
#  the trailing slash (/)
default_configfile="/etc/repository/repo.sources"
default_mirrordirectory="/repodata/mirrors/"
default_linkdirectory="/repository/"
default_owner="ftp:ftp"
default_slpregd="/etc/slp.reg.d/"
default_slpprefix="http://repository.customer.domain/"
version="0.2.0"

##############################################################################
# Below is the code responsible for the mirroring of the remote repositories #
##############################################################################
function mirror_repository
{
	# This function requires wget be installed on the system.  If it is not exit!
	if [ ! -x /usr/bin/wget ]; then
		echo -e "\nError: wget needs to be installed, before this program can work properly!"
		display_help
		exit 1
	fi

	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning Mirror Operation" 

	# wget has a number of parameter options
	# wget has a pseudo-hidden parameter --mirror, which is the same as below
	wgetparameters="--recursive --timestamping --level=inf --no-remove-listing"
	# To that we restrict the number of tries and force the creation of directories
	wgetparameters=$wgetparameters" --tries=2 --force-directories"
	# We next force wget to only look at the directory we specify and nothing above it
	#  we also tell wget to only follow relative links (i.e. don't link to other sites)
	#  we also tell wget to ignore any robot infomation (some sites try to exclude robots)
	#  and lastly we tell wget to give us the minimum output (even with this the output file can become large)
	wgetparameters=$wgetparameters" --no-parent --relative --execute=robots=off --no-verbose"
	# Since we are not interested in power pc or foreign languages, we exclude those directories of download
	wgetparameters=$wgetparameters" --exclude-directories=*/ppc,*/ppc64,*/cs,*/de,*/es,*/fr,*/it,*/ja,*/pt_BR,*/zh_CN,*/zh_TW"
	# Finally we tell wget where to start the download of all these repositories.
	wgetparameters=$wgetparameters" --directory-prefix="$mirrordirectory

	# Go through every non-commented, non-blank line which has a url component and 
	#  select the first column (using tab as the delimiter)
	for name in `egrep -v ^\# $configfile | grep :// | cut -f 1`
	do 
		# Using the unique name as a guide find the appropriate line and select
		#  the second column. (using tab as the delimiter)
		url=`egrep "^$name[[:space:]]" $configfile | cut -f 2`
		# We add another parameter, telling wget to record the output in a log file.
		echo -e "\nProcessing $url" 
		echo "command: wget $wgetparameters $url" 
		wget $wgetparameters $url --append-output=$mirrordirectory$name.log
	done
	echo -e "\nMirroring done." 
}

##############################################################################
# Below is the code responsible for removing the mounts in the repository    #
#  directory to the respective data                                          #
##############################################################################
function umount_repository
{
	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning Unlinking Operation"

	# Find all the bind mount points and umount them.
	for mountpoint in `mount | grep "(rw,bind)" | cut -d " " -f 3`
	do
		echo -e "\nUnlinking $mountpoint"
		echo "command: umount $mountpoint"
		umount $mountpoint
	done

	# Find all the empty subdirectories in the LinkDirectory and delete them.
	echo -e "\nRemoving all empty directories in $linkdirectory"
	echo "command: rmdir \"$linkdirectory\"*" 
	rmdir "$linkdirectory"*  2> /dev/null

	echo -e "\nUnlinking done."
}

##############################################################################
# Below is the code responsible for creating the mounts in the repository    #
#  directory to the respective data                                          #
##############################################################################
function mount_repository
{
	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning Linking Operation"

	# Go through every non-commented, non-blank line and select the first column 
	#  (using tab as the delimiter)
	for name in `egrep -v ^\# $configfile | cut -f 1`
	do 
		# Using the unique name as a guide to find the appropriate line and 
		#  select the second column. (using tab as the delimiter)
		target=`egrep "^$name[[:space:]]" $configfile | cut -f 2`

		# If a URL component is present, replace it with the mirror directory 
		#  specified above
		temp="ftp://"
		target=${target/$temp/$mirrordirectory}
		temp="http://"
		target=${target/$temp/$mirrordirectory}
		temp="https://"
		target=${target/$temp/$mirrordirectory}

		echo -e "\nLinking $name from $target to $linkdirectory$name"

		# If the mount directory does not exist, then create it.		
		if [ ! -d "$linkdirectory$name" ]; then
			echo "Creating directory $linkdirectory$name"
			echo -e "command: mkdir \"$linkdirectory$name\""
			mkdir "$linkdirectory$name"
		else
			echo "Directory $linkdirectory$name already exists"
		fi

		# Only mount the directory if it is not already mounted!
		temp=`mount | grep "$linkdirectory$name type none (rw,bind)"`
		if [ "$temp" = "" ]; then
			echo "Mounting $target on $linkdirectory$name"
			echo -e "command mount -v --bind \"$target\" \"$linkdirectory$name\""
			mount -v --bind "$target" "$linkdirectory$name"
		else
			echo "$target is already mounted"
			echo $temp
		fi
	done

	echo -e "\nLinking done."

}

##############################################################################
# Below is the code responsible for creating the metadata with the custom    #
#  repositories                                                              #
##############################################################################
function create_repository
{
	# This function requires wget be installed on the system.  If it is not exit!
	if [ ! -x /usr/bin/createrepo ]; then
		echo "createrepo needs to be installed, before this program can work properly!"
		display_help
		exit 1
	fi

	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning Create Operation"


	# Go through every non-commented, non-blank line and select the first column 
	#  (using tab as the delimiter)
	for name in `egrep -v ^\# $configfile | cut -f 1`
	do 
		# Using the unique name as a guide to find the appropriate line and 
		#  select the third column. (using tab as the delimiter)
		custom=`egrep "^$name[[:space:]]" $configfile | cut -f 4`

		# Only if the fourth column is present, then process that repository
		if [[ "$custom" != "" ]]
		then
			# Using the unique name as a guide to find the appropriate line and 
			#  select the second column. (using tab as the delimiter)
			targetdirectory=`egrep "^$name[[:space:]]" $configfile | cut -f 2`

			# If a URL component is present, replace it with the mirror directory 
			#  specified above
			temp="ftp://"
			targetdirectory=${targetdirectory/$temp/$mirrordirectory}
			temp="http://"
			targetdirectory=${targetdirectory/$temp/$mirrordirectory}
			temp="https://"
			targetdirectory=${targetdirectory/$temp/$mirrordirectory}

			echo -e "\nCreating YUM repository files for $name"
			echo -e "command: createrepo --pretty --verbose --baseurl=$targetdirectory $targetdirectory"
			createrepo --pretty --verbose --baseurl=$targetdirectory $targetdirectory
		fi
	done

	echo -e "\nCreate done."

}


##############################################################################
# Below is the code responsible for changing the file ownership.             #
##############################################################################
function change_ownership
{
	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning Ownership Operation"
	echo -e "command: chown --recursive $owner $linkdirectory"

	chown -v --recursive $owner $linkdirectory > /dev/null

	# While maybe not the most secure, it will give everyone read access to the files and be allowed to open directories
	echo -e "command: chmod -c --recursive a+rx $linkdirectory"

	chmod -c --recursive a+r $linkdirectory 

	echo -e "\nOwnership done."
}


##############################################################################
# Below is the code responsible for updating the SLP information.            #
##############################################################################
function change_slp
{
	echo -e "\n--------------------------------------------------------------------------------"
	echo -e "Beginning SLP Operation"

	echo -e "\nStopping SLP services"
	rcslpd stop

	echo -e "\nRemove all the existing SLP registration files."
	echo -e "command: rm \"$default_slpregd\"repository.*.reg"
	rm "$default_slpregd"repository.*.reg 2> /dev/null

	for name in `egrep -v ^\# $configfile | cut -f 1`
	do
		# Using the unique name as a guide to find the appropriate line and 
		#  select the second column. (using tab as the delimiter)
		target=`egrep "^$name[[:space:]]" $configfile | cut -f 2`

		# Using the unique name as a guide to find the appropriate line and 
		#  select the third column. (using tab as the delimiter)
		custom=`egrep "^$name[[:space:]]" $configfile | cut -f 3`

		# Only if the third column is present, then process that repository
		if [[ "$custom" = "SLP" ]]; then
			regfile="$default_slpregd"repository.$name.reg
			echo -e "\nCreating registration file $regfile"

			# Set default SLP entries
			service="service:install.suse:$default_slpprefix$name,en,65535"
			defaultbase="i586"
			label="$name"
			description="$label ++${target##/*}++"
			machine="i386,i486,i586,i686"
			vendor=
			slpversion=

			# See if a .repo or content file is present in the specified repository and gather information from it.
			# If a URL component is present, replace it with the mirror directory 
			#  specified above
			temp="ftp://"
			target=${target/$temp/$mirrordirectory}
			temp="http://"
			target=${target/$temp/$mirrordirectory}
			temp="https://"
			target=${target/$temp/$mirrordirectory}
			tempfile="$target"content
			if [ -f "$tempfile" ]
			then
				# Retrieve the Description from the PRODUCT and DISTPRODUCT fields of the content file
				temp=`grep -i "^PRODUCT[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					description=$temp
				fi
				temp=`grep -i "^DISTPRODUCT[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					description=$description" ++"$temp"++"
				fi

				# Retrieve the Version from the VERSION field of the content file
				temp=`grep -i "^VERSION[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					slpversion=$temp
				fi

				# Retrieve the Vendor from the VENDOR field of the content file
				temp=`grep -i "^VENDOR[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					vendor=$temp
				fi

				# Retrieve the DefaultBase from the DEFAULTBASE field of the content file
				temp=`grep -i "^DEFAULTBASE[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					defaultbase=$temp
				fi

				# Retrieve the Label from the LABEL field of the content file
				temp=`grep -i "^LABEL[[:space:]]" $tempfile | cut -d " " -f 2`
				if [ -n "$temp" ]; then
					label=$temp
				fi
			else
				tempfile=`ls "$target"*.repo | head -n 1`
				if [ -f "$tempfile" ]; then
					# Retrieve the Label and Description from the name field of the .repo file
					temp=`grep -i "^name=" $tempfile | cut -d "=" -f 2`
					if [ -n "$temp" ]; then
						label=$temp
						description="$label ++${target##/*}++"
					fi
				fi
			fi

			echo "#############################################################################" > $regfile
			echo "#" >> $regfile
			echo "# OpenSLP registration file" >> $regfile
			echo "#" >> $regfile
			echo "# register $name" >> $regfile
			echo "#" >> $regfile
			echo -e "#############################################################################\n" >> $regfile
			echo "# $name Repository" >> $regfile
			echo "$service" >> $regfile
			echo "defaultbase=$defaultbase" >> $regfile
			echo "description=$description" >> $regfile
			echo "label=$label" >> $regfile
			echo "machine=$machine" >> $regfile
			if [ -n "$vendor" ]; then
				echo "vendor=$vendor" >> $regfile
			fi
			if [ -n "$slpversion" ]; then
				echo "version=$slpversion" >> $regfile
			fi
			echo -e "\n" >> $regfile
		fi
	done

	echo -e "Starting SLP services"
	rcslpd restart
	echo -e "\nSLP done."
}


##############################################################################
# Below is the code responsible for displaying the help screen.              #
##############################################################################
function display_help
{
	echo -e "\n    ${0##*/} [Optional Options] Manditory Options\n"
	echo -e "    Optional Options:"
	echo -e "     --http-proxy  = URL of the proxy to use"
	echo -e "     --config-file = location of the configuration file"
	echo -e "     --mirror-base = base directory of the local mirror repositories"
	echo -e "     --link-base   = base directory of the repository visible to the public"
	echo -e "     --owner       = user:group ownership of the files (default is ftp:ftp)"
	echo -e "     -d, --default = use the default values"
	echo -e "\n    Manditory Options: (at least one of the following)"
	echo -e "     -m, --mirror  = mirror repositories"
	echo -e "     -u, --unlink  = unlink repositories (umount)"
	echo -e "     -l, --link    = link repositories (mount)"
	echo -e "     -c, --create  = create repository metafiles"
	echo -e "     -o, --owner   = change the ownership of the files"
	echo -e "     -s, --slp     = update SLP information"
	echo -e "     -a, --all     = perform all the above operations"
	echo -e "                     and finally create the repositories"
	echo -e "     -h, --help    = show this help"
	echo -e "     -v, --version = output version\n"
	echo -e "    Please note that wget and createrepo need to be installed for this script"
	echo -e "     to run properly.\n\n"
	echo -e "    Examples:"
	echo -e "     To only mirror remote repositories through a proxy"
	echo -e "     ${0##*/} --http-proxy http://proxy.local.domain:80 --mirror\n"
	echo -e "     To specify a different config file and only create repository metafiles"
	echo -e "     ${0##*/} --config-file /repodata/first.conf --create\n"
	echo -e "     To create metadata and change ownership of files to root:users"
	echo -e "     ${0##*/} --create --owner root:users -o"
	echo -e "     To perform all actions (say for a cron job) using the defaults"
	echo -e "     ${0##*/} --default --all\n"
	exit 0
}

##############################################################################
# Below is the code responsible for displaying the version screen.           #
##############################################################################
function display_version
{
	echo -e "\nGNU ${0##*/} $version\n"
	echo -e "This program is distributed in the hope that it will be useful,"
	echo -e "but WITHOUT ANY WARRANTY; without even the implied warranty of"
	echo -e "MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the"
	echo -e "GNU General Public License for more details.\n"
	echo -e "Originally written by Bob Brandt <projects@brandt.ie>.\n"
	exit 0
}

##############################################################################
# Below is the code responsible for setting the default values specified in  #
#  the beginning of the file.                                                #
##############################################################################
function set_defaults
{
	# The http_proxy variable must be exported if the wget function is to 
	#  work properly.
	export http_proxy=$default_http_proxy
	configfile=$default_configfile
	mirrordirectory=$default_mirrordirectory
	linkdirectory=$default_linkdirectory
	owner=$default_owner
}



##############################################################################
# Everything below is the main code of the script, which processes the       #
#  parameters and runs the functions above.                                  #
##############################################################################

# Flags to be used below
run_mirror=0
run_unlink=0
run_link=0
run_create=0
run_chown=0
run_slp=0
use_default=0

# Begin by setting the default values
set_defaults

# Process all the parameters.  Since we do not know what order the parameters will 
#  be given, we need to process all the parameters before we act upon them.
while [ "$1" != "" ]; do
	case "$1" in
		"--http-proxy" )
			shift
			http_proxy="$1"
		;;
		"--config-file" )
			shift
			configfile="$1"
		;;
		"--mirror-base" )
			shift
			mirrordirectory="$1"
		;;
		"--link-base" )
			shift
			linkdirectory="$1"
		;;
		"--owner" )
			shift
			owner="$1"
		;;
		"/d" | "-d" | "--default" )
			use_default=1
		;;
		"/m" | "-m" | "--mirror" )
			run_mirror=1
		;;
		"/u" | "-u" | "--unlink" )
			run_unlink=1
		;;
		"/l" | "-l" | "--link" )
			run_link=1
		;;
		"/c" | "-c" | "--create" )
			run_create=1
		;;
		"/o" | "-o" | "--owner" )
			run_chown=1
		;;
		"/s" | "-s" | "--slp" )
			run_slp=1
		;;
		"/a" | "-a" | "--all" )
			run_mirror=1
			run_unlink=1
			run_link=1
			run_create=1
			run_chown=1
			run_slp=1
		;;
		"/?" | "/h" | "-?" | "-h" | "--help" )
			display_help
			exit 0
		;;
		"/v" | "-v" | "--version" )
			display_version
			exit 0
		;;
		* )
			echo "Options Error: option $1 not recognized."
			display_help
			exit 1
		;;
	esac
	shift
done

# Regardless of the other parameters, if the default option was set, set 
#  everything back to defaults
if [ "$use_default" = "1" ]; then
	set_defaults
fi

# Regardless of order, if selected, the mirror function will run before 
#  the link function which will run before the create function.
if [ "$run_unlink" = "1" ]; then
	umount_repository
fi
if [ "$run_mirror" = "1" ]; then
	mirror_repository
fi
if [ "$run_create" = "1" ]; then
	create_repository
fi
if [ "$run_link" = "1" ]; then
	mount_repository
fi
if [ "$run_chown" = "1" ]; then
	change_ownership
fi
if [ "$run_slp" = "1" ]; then
	change_slp
fi

exit 0

(:divend:)

However, because I could not use symbolic links, and I did not wish want the script to automatically modity the fstab file, each reboot "erases" the mount commands that I issued earlier. So I need a way to make sure that the script is run, linking the necessary directories, when the server is started. The obvious way to do this is to create a init.d wrapper script for the script above. Whenever I need to create a init.d script I look both at examples already in the /etc/init.d directory, and at the Cool Solutions page created by Kirk Coombs Creating Custom init Scripts (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#! /bin/sh
##############################################################################
##############################################################################
# Author: Bob Brandt projects@brandt.ie, based on random SuSE script files   #
#                                                                            #
# /etc/init.d/repod                                                          #
#                                                                            #
#  Because this is a basic script, there is no need for a rc symbolic link   #
#   in the /usr/sbin directory.                                              #
#                                                                            #
# This script is a simple startup wrapper for another script I wrote.  The   #
#  reason I need a wrapper is because the original script needs to be        #
#  launched at least once every time the server restarts. Admittedly this    #
#  script is script is a bit over the top for what needs to be done, but I   #
#  already has this script for other reasons and it was easier to just       #
#  modify this one then create a new one.                                    #
#                                                                            #
# Comments can be used but must be preceeded by a hash symbol (#)            #
# All Comments are to be preceded by a hash symbol (#) (Maximum of 78 chars) #
#                                                                            #
##############################################################################
##############################################################################
### BEGIN INIT INFO
# Provides:          repod
# Required-Start:    $network $named
# Required-Stop:
# Default-Start:     3 5
# Default-Stop:      0 1 2 4 6
# Description: Wrapper for repository.sh script file which maintaines repository mirrors
### END INIT INFO

# Change this script location as needed for different wrappers.
scriptlocation="/etc/repository/repository.sh"

# Determine the base and follow a runlevel link name.
base=${0##*/}
link=${base#*[SK][0-9][0-9]}

# Check to make sure the binaries, scripts and cofiguration files are available
test -x "$scriptlocation" || exit 1

# Load the rc.status script for this service
. /etc/rc.status

case "$1" in
  start)
    echo -n 'Starting linking repositories '
    startproc $scriptlocation --link
    rc_status -v
    ;;
  stop)
    echo -n "Shutting down repositories "
#    killproc -TERM $scriptlocation --unlink
    rc_status -v
    ;;
  restart)
    $0 stop
    $0 start
    rc_status
    ;;
#  try-restart)
#    $0 status
#    if test $? = 0; then
#	$0 restart
#    else
#	rc_reset
#    fi
#    rc_status
#    ;;
#  force-reload)
#    $0 stop; sleep 1  &&  $0 start
#    rc_status
#    ;;
#  reload)
#    echo -n "Reload service name "
#    killproc -HUP $scriptlocation
#    rc_status -v
#    ;;
  status)
    echo -n "Checking for servicename "
#    checkproc $scriptlocation
    rc_status -v
    ;;
  *)
#    echo "Usage: $base {start|stop|try-restart|restart|force-reload|reload|status}"
    echo "Usage: $base {start|stop|restart|status}"
    exit 1
esac
rc_exit

(:divend:) Once created you need to make sure it will be run on the various runtimes, you can either use the innserv /etc/init.d/repod command or the runlevel editor in YaST.

Need to setup the CRON Job While we need the script to run with the -l switch at the appropriate runtimes, we also want to run the script periodically with the -a switch. This means setting up a CRON job. This is very easy: Create a temporary text file with the something like: 0 18 * * * /etc/repository/repository.sh -a This will run the job every night at 6pm, if this is too often you could use the following: 0 18 * * 5 /etc/repository/repository.sh -a Which will run the job every Friday night at 6pm Then run the command (as root): crontab tempfile.name To make sure it took, you can use crontab -l to list all the jobs.

Need to setup the logrotate jobs Because we are appending to the log file, we may someday have a very large log file. To stop this from happening, we create a logrotate job. Now there is a standard CRON job that already runs on every Linux box. All we need to do is add our job to the list of logs that are already being rotated. The easiest way to do this is to create a text file in the /etc/logrotate.d directory. the standard job includes every file in this directory, so it make it very easy to separate job out. The job I used is below, feel free to change the specifics as you see fit. (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#
# /etc/logrotate.d/repository
#

/repodata/mirrors/*.log {
    compress
    dateext
    maxage 30
    rotate 5
    size=+2048k
    notifempty
    missingok
    copytruncate
}

(:divend:) To test to make sure that you did not screw up it is always wise to force logrotate to run at least once to see what happens. Use the command: logrotate -fv /etc/logrotate.d/repository

To publish the repositories via HTTP, I used the default Apache install and modified the default-server.conf file (which is called by an include in the httpd.conf file). It is very basic, but only allows users to read one single directory and it's contents. Exactly what we need! (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#
# /etc/apache2/default-server.conf
#

DocumentRoot "/repository"

<Directory "/repository">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

ServerName repository.company.domain
ServerAdmin admin@company.domain

(:divend:)

To publish the repositories via FTP I finally decided to use pure-FTPd (the default FTP server) and modified the pure-ftpd.conf as follows: (:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#
# /etc/pure-ftpd/pure-ftpd.conf
#

# Turn on compatibility hacks for broken clients
BrokenClientsCompatibility  No

# Fork in background
Daemonize                   yes

# Maximum number of sim clients with the same IP address
MaxClientsPerIP             3

# If you want to log all client commands, set this to "yes".
# This directive can be duplicated to also log server responses.
VerboseLog                  no

# Allow dot-files
AllowDotFiles               no

# List dot-files even when the client doesn't send "-a".
DisplayDotFiles             no

# Don't allow authenticated users - have a public anonymous FTP only.
AnonymousOnly               yes

# Disallow anonymous connections. Only allow authenticated users.
NoAnonymous                 no

# Syslog facility (auth, authpriv, daemon, ftp, security, user, local*)
# The default facility is "ftp". "none" disables logging.
SyslogFacility              ftp

# Don't resolve host names in log files. Logs are less verbose, but 
# it uses less bandwidth. Set this to "yes" on very busy servers or
# if you don't have a working DNS.
DontResolve                 yes

# Maximum idle time in minutes (default = 15 minutes)
MaxIdleTime                 15

# 'ls' recursion limits. The first argument is the maximum number of
# files to be displayed. The second one is the max subdirectories depth
LimitRecursion              2000 8

# Are anonymous users allowed to create new directories ?
AnonymousCanCreateDirs      no

# Disallow downloading of files owned by "ftp", ie.
# files that were uploaded but not validated by a local admin.
AntiWarez                   yes

# Users can't delete/write files beginning with a dot ('.')
# even if they own them. If TrustedGID is enabled, this group
# will have access to dot-files, though.
ProhibitDotFilesWrite       yes

# Prohibit *reading* of files beginning with a dot (.history, .ssh...)
ProhibitDotFilesRead        no

# Disallow anonymous users to upload new files (no = upload is allowed)
AnonymousCantUpload         yes

(:divend:)

To publish the repositories via SMB/CIFS I used the default Samba install and modified the smb.conf file. It is very basic, but again, only allows users to read one single directory and it's contents. Exactly what we need!

(:div style="border-style:ridge; border-width:2px; background-color:#ffffcc; margin-left:50px; overflow:auto; width:650px; height:250px;":)

#
# /etc/samba/smb.conf
#
[global]
	workgroup = DOMAINorWORKGROUPname
	map to guest = Bad User
	domain logons = No
	domain master = No
	netbios name = repository
	security = user
	wins server = WINSSERVERIP1 WINSSERVERIP2
	wins support = No
[repository]
	comment = Customer Linux Repositories
	guest ok = Yes
	inherit acls = No
	path = /repository/
	read only = Yes

(:divend:)

Need publish the repositories via NFS

Want to advertise via SLP