UNIX

I wanted to automate filesystem dumps on my servers running FreeBSD 7.2. After some searching I came across Vivek Gite’s FreeBSD Full / Incremental Tape Backup Shell Script which gave me a lot of ideas. Since I’m not using tape as the backup target I wanted to make a script specifically for that purpose while at the same time improve handling of some error conditions (such as, most importantly, checking for a missing level 0 dump before proceeding with an incremental dump) and add some new features such as autoincrement the dump level so that the dump level is not tied to specific day of the week.

Here’s my version of the script. While it bears some resemblance to Vivek’s script, it is largely rewritten. Read the script header for more information.

NOTE! In his comment James pointed out a possible bug in the script. The displayed script indeed had a problem: it was missing a backslash in front of the first dollar sign at:

eval “local fspath=\$${fsname}path”

This was caused by the script display plugin in WordPress that treated the backslash as an escape character (this has now been fixed). To be on the safe side, please download the script as a tarball. To further validate the integrity of the tarball, it should produce a md5 hash of 732ac44f11ba4484be4568e84929bb6a.

  | 
 copy code |
?
 
001
002
#!/bin/sh
003
 
004
# Autodump 1.5a (released 01 August 2009)
005
# Copyright (c) 2009 Ville Walveranta 
006
#
007
# A FreeBSD shell script to dump filesystems with full, and automatically 
008
# incremented incremental backups to a given directory location; this script
009
# was written with the intent of saving the filesystem dumps not onto a tape
010
# device but on another hard drive such as a different filesystem on the same 
011
# computer. The resulting dump files can be copied offsite with a separate 
012
# cron job.
013
#
014
# This script creates the necessary directory structure below the defined 
015
# 'BASEDIR' as well as the necessary log file. This script also ensures that
016
# the level 0 dump exists before creating an incremental dump; if it doesn't
017
# the script automatically erases the incremental files for the current week 
018
# (if any exist) and starts over with a level 0 dump. This way you can start 
019
# using the script on any day of the week and level 0 dump is automatically 
020
# created on the first run.
021
#
022
# When ran daily (such as from a cron job), the script creates level 0 dump
023
# on every Monday (beginning the ISO week), or Sunday (beginning of the U.S. 
024
# week) and an incremental dump on all the other days of each week. The dumps 
025
# are compressed with gzip and saved below the 'BASEDIR' to an automatically 
026
# created directory whose name is derived from the list given in 'FSNAMES'. 
027
# Each week's dumps are organized into subfolders with name YYYY-WW ('WW' 
028
# being the current week). By default three most recent weekly dumps 
029
# (level 0 + incrementals) are retained.
030
#
031
# The script maintains each weekly folder's date at the _beginning_ date
032
# of the dump (i.e. Monday or Sunday of the current week) at 00:00, not 
033
# at the most recent incremental's date/time.
034
#
035
# By default the root (/) and usr (/usr) filesystems are dumped. To add more  
036
# add a "friendly name" to the 'FSNAMES' list (it is used for the weekly folder
037
# names, for dump filenames, and to reference the corresponding mount point
038
# variable); then add the corresponding mount point variable (i.e. if you 
039
# add "var" to 'FSNAMES', then add a variable varpath=/var). The "path" 
040
# ending of the mount point variable name is required. 
041
#
042
# Since the number of incremental dumps is limited to nine (level 0 +
043
# incremental levels 1-9), the script will allow maximum of one dump 
044
# to be created per day. However, since the level incrementing is dynamic
045
# you can start the script on any day of the week, and run it on any
046
# number of days during the rest of the week and you'll always get
047
# level 0 plus the incremental dumps in sequential order. However, The 
048
# new weekly folder is always created on Monday or Sunday (as chosen by
049
# you). Note that the script determines whether "today's" dump exists 
050
# based on the modification date stamp of the most recent dump. Hence 
051
# it is a good idea to run this script in the early hours of each day 
052
# rather than in the very end of each day. Running the script, for 
053
# example, at 23:50 has the potential to push longer dump processes 
054
# over the midnight and so potentially cause the next day's dump to 
055
# be skipped.
056
#
057
# Written for FreeBSD 7.2 but should work on most BSD and *NIX systems with
058
# minor modifications.
059
# -------------------------------------------------------------------------
060
# Copyright (c) 2009 Ville Walveranta 
061
# <http://my.galagzee.com/2009/07/17/freebsd-dump-filesystem-shell-script>
062
# This script is licensed under GNU GPL version 2.0 or above, and is provided
063
# 'as-is' with no warranty which is to say that I'm not liable if it wipes out
064
# your hard drive clean or doesn't back up your precious data. However, to the 
065
# best or my knowledge it is working as expected -- I'm using it myself. :-)
066
# -------------------------------------------------------------------------
067
# This script was inspired by 
068
# FreeBSD Full / Incremental Tape Backup Shell Script
069
# by nixCraft project / Vivek Gite
070
# at <http://bash.cyberciti.biz/backup/freebsd-dump-filesystem-shell-script/>
071
# -------------------------------------------------------------------------
072
 
073
 
074
#### GLOBAL VARIABLES ###############################################
075
 
076
WEEKSTARTS=Mon      # Accepted values are "Mon" (ISO standard) or "Sun" (U.S.)
077
KEEPDUMPS=30        # in days; this is evaluated on the weekly level per start
078
                    # of the week, so '30' keeps 3-4 weekly dumps
079
BASEDIR=/bak/dumps
080
GLOBALDUMPOPTS=Lua  # add 'n' for wall notifications
081
LOGFILE=/var/log/dump.log
082
 
083
# to add more filesystems to be dumped add the dump name in 'FSNAMES'
084
# and add the corresponding mount point variable (dumpname+path=mountpoint)
085
FSNAMES="root usr"  # this is used for dump directory name 
086
                    # and to ID the path from a variable below
087
rootpath=/
088
usrpath=/usr
089
 
090
#####################################################################
091
 
092
DUMP=/sbin/dump
093
GZIP=/usr/bin/gzip
094
LOGGER=/usr/bin/logger
095
 
096
WEEKDAY=$(date +"%a")
097
DATE=$(date +"%Y%m%d")
098
HUMANDATE=$(date +"%d-%b-%Y")
099
HUMANDATE=`echo $HUMANDATE | tr '[:lower:]' '[:upper:]'`
100
HUMANTIME=$(date +"%H:%M (%Z)")
101
TODAYYR=$(date +"%Y")
102
TODAYMO=$(date +"%m")
103
TODAYDT=$(date +"%d")
104
 
105
# datestamp at midnight today
106
TODAYSTARTSTAMP=$(date -j +%s "${TODAYYR}${TODAYMO}${TODAYDT}0000")
107
 
108
# default lastdump to midnight today; it will be checked
109
# and and adjusted later
110
LASTDUMP=$TODAYSTARTSTAMP
111
 
112
# do not crete world-readable dumps!
113
umask 117
114
 
115
# make sure the logfile exists
116
if [ ! -e $LOGFILE ] ; then
117
   touch $LOGFILE
118
   chmod 660 $LOGFILE
119
fi
120
 
121
# make sure that entire week's incremental dumps are deposted
122
# in the same directory, even when a week spans new year
123
# NOTE: When the ending year has a partial 53rd week, there
124
# won't be a dump folder for the first week of the new year.
125
# The incremental dumps instead complete the 53rd week folder,
126
# even when the 1st week of the new year begins mid-week. 
127
# However, the dates of the incremental dump files in the 
128
# 53rd week folder correctly reflect the dates of the 
129
# beginning year.
130
adjust_date(){
131
   local dateoffset=$1
132
   local epochnow=$(date +%s)
133
   local offsetsecs=`expr $dateoffset "*" 86400`
134
   local newepoch=`expr $epochnow "-" $offsetsecs`
135
   local year=`date -r $newepoch +"%Y"`
136
 
137
   if [ "$WEEKSTARTS" = "Mon" ] ; then
138
      local week=`date -r $newepoch +"%W"`
139
   else
140
      local week=`date -r $newepoch +"%U"`
141
   fi
142
   NEWEPOCHISO=`date -r $newepoch +"%Y%m%d0000"`
143
 
144
   #system week starts from `0', there is no calendar week `0'
145
   week=`expr $week "+" 1`
146
   YWEEK=${year}-${week}
147
}
148
 
149
# determines the 'distance' from the level 0 dump in days
150
if [ "$WEEKSTARTS" = "Mon" ] ; then
151
   case $WEEKDAY in
152
      Mon) adjust_date 0;;
153
      Tue) adjust_date 1;;
154
      Wed) adjust_date 2;;
155
      Thu) adjust_date 3;;
156
      Fri) adjust_date 4;;
157
      Sat) adjust_date 5;;
158
      Sun) adjust_date 6;;
159
      *) ;;
160
   esac
161
else 
162
   case $WEEKDAY in
163
      Sun) adjust_date 0;;
164
      Mon) adjust_date 1;;
165
      Tue) adjust_date 2;;
166
      Wed) adjust_date 3;;
167
      Thu) adjust_date 4;;
168
      Fri) adjust_date 5;;
169
      Sat) adjust_date 6;;
170
      *) ;;
171
   esac
172
fi
173
 
174
mk_auto_dump(){
175
 
176
   local fsname=$1
177
 
178
   # get the current filesystem's path
179
   # as defined in the corresponding variable
180
   eval "local fspath=\$${fsname}path"
181
 
182
   # composite the dump path
183
   local dumppath=${BASEDIR}/${fsname}/${YWEEK}
184
 
185
   # make sure the dump directory for this week exists;
186
   # this automatically creates a new dump directory on 
187
   # every Monday or Sunday (as selected by 'WEEKSTARTS')
188
   [ ! -d $dumppath ] && mkdir -p $dumppath
189
 
190
   # get name of the last file in the current dump directory
191
   local lastfile=`ls -ltr $dumppath | grep -v "^d" | tail -n 1 | awk '{ print $9 }'`
192
 
193
   # assume that the 'lastfile', if it exists, was not created today
194
   local dumped_today=false
195
 
196
   # if a file exists, check its modification date; 
197
   # if it is at or after 00:00 today, set a flag to skip the dump
198
   if [ "$lastfile" != "" ] ; then
199
      local fq_lastfile=${dumppath}/$lastfile
200
      if [ -e $fq_lastfile ] ; then
201
         # get the last modification time for the most recently created dumpfile
202
         LASTDUMP=`stat -f %m $fq_lastfile`
203
         if [ $LASTDUMP -ge $TODAYSTARTSTAMP ] ; then
204
            local dumped_today=true
205
         fi
206
      fi
207
 
208
      # get the first and the last dump level for this directory
209
      local levelcommand="ls $dumppath | sed -e 's/^[[:digit:]]*\_//' | sed -e 's/\..*$//'"
210
      local firstlevel=`eval $levelcommand | head -n 1`
211
      local lastlevel=`eval $levelcommand | tail -n 1`
212
 
213
      # make sure level zero dump exists;
214
      # if it doesn't, start over
215
      if [ "$firstlevel" != "0" ] ; then
216
         # it doesn't matter if a previous dump exists from today
217
         # since we're starting over as level 0 dump is missing
218
         local dumped_today=false
219
         local dumplevel=0
220
         rm -f $dumppath/*.gz
221
      else
222
         # otherwise just increment the dump level
223
         # for levels 1-6, i.e. normally Tuesday thru Sunday
224
         local dumplevel=`expr $lastlevel "+" 1`
225
      fi
226
   else
227
      # no dump exists in this week's folder; reset level to '0'
228
      local dumplevel=0
229
   fi
230
 
231
   # skip the entire dump process if a dumpfile has
232
   # already been created for this filesystem today
233
   if [ "$dumped_today" = "false" ] ; then  
234
 
235
      # define the dump filename
236
      local dumpfn=${DATE}_${dumplevel}
237
 
238
      echo ---------------- >> $LOGFILE
239
      echo >> $LOGFILE
240
      echo BEGINNING LEVEL $dumplevel DUMP OF \'$fsname\' \(${fspath}\) FILESYSTEM ON $HUMANDATE AT $HUMANTIME >> $LOGFILE
241
      echo >> $LOGFILE
242
      echo Creating a snapshot of \'$fspath\'.. >> $LOGFILE
243
      # execute the dump
244
      $DUMP -$dumplevel -$GLOBALDUMPOPTS -f ${dumppath}/${dumpfn} $fspath >> $LOGFILE 2>&1
245
      local dumpresult=$?
246
 
247
      if [ "$dumpresult" != "0" ] ; then
248
         # log the dump result to syslog
249
         $LOGGER "$DUMP LEVEL $dumplevel DUMP OF $fsname (${fspath}) FAILED!"
250
 
251
         echo "*** DUMP FAILED - LEVEL $dumplevel DUMP of $fsname (${fspath}) ***" >> $LOGFILE
252
         echo >> $LOGFILE
253
      else
254
         # log the dump result to syslog
255
         $LOGGER "LEVEL $dumplevel DUMP of $fsname (${fspath}) COMPLETED SUCCESSFULLY!"
256
 
257
         echo >> $LOGFILE
258
         # compress the dump
259
         echo Compressing the dumpfile \'${dumpfn}\'.. >> $LOGFILE
260
         $GZIP -v ${dumppath}/${dumpfn} >> $LOGFILE 2>&1
261
         echo DONE >> $LOGFILE
262
         echo >> $LOGFILE
263
 
264
         # make sure dumps are not world readable (security risk!)
265
         echo Updating dumpfile \'${dumpfn}.gz\' permissions.. >> $LOGFILE
266
         chmod -v -v 440 ${dumppath}/${dumpfn}.gz >> $LOGFILE 2>&1
267
         echo DONE >> $LOGFILE
268
         echo >> $LOGFILE
269
 
270
         # reset current dump dir's timestamp to that of the level 0 dump
271
         touch -t ${NEWEPOCHISO} ${dumppath}
272
 
273
         # delete old dumps
274
         echo Deleting old \'$fsname\' dumpfiles.. >> $LOGFILE
275
         find $BASEDIR/$fsname -mtime +$KEEPDUMPS -maxdepth 1 -print -exec rm -rf {} \; >> $LOGFILE 2>&1
276
         echo DONE >> $LOGFILE
277
         echo >> $LOGFILE
278
      fi
279
   else
280
      local lastdump_readable=`date -j -r $LASTDUMP +"%H:%M"`
281
      local lastdump_readableZ=`date -j -r $LASTDUMP +"%Z"`
282
      local lastdumpmsg="Autodump for filesystem '$fsname' ($fspath) has already been executed today at $lastdump_readable ($lastdump_readableZ)."
283
      echo $lastdumpmsg
284
      $LOGGER $lastdumpmsg
285
   fi
286
}
287
 
288
 
289
# Dump filesystems defined in 'FSNAMES'
290
#
291
# Monday or Sunday (as selected by 'WEEKSTARTS') starts with 
292
# the level 0 dump, with incrementals created through the rest of 
293
# the week (autoincremented). If the level 0 dump is missing in 
294
# the current week's folder for filesystem currently being backed 
295
# up, it is created automatically instead of an incremental dump, 
296
# no matter what day of the week it is.
297
for f in $FSNAMES
298
do
299
   mk_auto_dump $f
300
done
301