DragonFlyBSD

Real World DragonFlyBSD Hammer DeDup figures from HiFX - Second Disk 20% ( 88 GB ) Disk Space reclaimed

2011-08-19T02:07:00.000-07:00

I De-Duped the second disk on the backup Server today.

Before De-Dup

Filesystem Size Used Avail Capacity Mounted on
Backup2 454G 335G 119G 74% /Backup2

After De-Dup

Filesystem Size Used Avail Capacity Mounted on
Backup2 454G 247G 207G 54% /Backup2

So 20% or 88 GB reclaimed without deleting anything :-)

No. of PFSes - 3 Slaves and 3 Masters

De-dup Simulate

==============================

======================================
dfly-bkpsrv# hammer dedup-simulate /Backup2
Dedup-simulate /Backup2: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 0
Dedup-simulate /Backup2 succeeded
Simulated dedup ratio = 1.26
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/Data
Dedup-simulate /Backup2/pfs/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup-simulate /Backup2/pfs/Data succeeded
Simulated dedup ratio = 1.37
You have new mail.
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/VersionControl
Dedup-simulate /Backup2/pfs/VersionControl: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 2
Dedup-simulate /Backup2/pfs/VersionControl succeeded
Simulated dedup ratio = 2.79
dfly-bkpsrv# hammer dedup-simulate /Backup2/pfs/test
Dedup-simulate /Backup2/pfs/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup-simulate /Backup2/pfs/test succeeded
Simulated dedup ratio = 0.00
dfly-bkpsrv# hammer dedup-simulate /Backup2/www-daily-bak/www-hot
Dedup-simulate /Backup2/www-daily-bak/www-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 5
Dedup-simulate /Backup2/www-daily-bak/www-hot succeeded
Simulated dedup ratio = 1.64
dfly-bkpsrv# hammer dedup-simulate /Backup2/mysql-daily-bak/mysql-hot
Dedup-simulate /Backup2/mysql-daily-bak/mysql-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 6
Dedup-simulate /Backup2/mysql-daily-bak/mysql-hot succeeded
Simulated dedup ratio = 5.99
====================================================================

Real De-dup

=========================================================
dfly-bkpsrv# hammer dedup /Backup2
Dedup /Backup2: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 0
Dedup /Backup2 succeeded
Dedup ratio = 1.25
580 MB referenced
462 MB allocated
2274 KB skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/Data
Dedup /Backup2/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup /Backup2/Data succeeded
Dedup ratio = 1.37
266 GB referenced
194 GB allocated
31 MB skipped
1960 CRC collisions
0 SHA collisions
20 bigblock underflows
You have new mail.

dfly-bkpsrv# hammer dedup /Backup2/VersionControl
Dedup /Backup2/VersionControl: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 2
Dedup /Backup2/VersionControl succeeded
Dedup ratio = 2.75
160 MB referenced
58 MB allocated
861 KB skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows
You have new mail.
dfly-bkpsrv# hammer dedup /Backup2/test
Dedup /Backup2/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup /Backup2/test succeeded
Dedup ratio = 0.00
0 B referenced
0 B allocated
0 B skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/www-daily-bak/www-hot
Dedup /Backup2/www-daily-bak/www-hot: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 5
Dedup /Backup2/www-daily-bak/www-hot succeeded
Dedup ratio = 1.64
5599 MB referenced
3420 MB allocated
12 MB skipped
4 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup2/mysql-daily-bak/mysql-hot
Dedup /Backup2/mysql-daily-bak/mysql-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 6
Dedup /Backup2/mysql-daily-bak/mysql-hot succeeded
Dedup ratio = 6.28
49 GB referenced
8079 MB allocated
0 B skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

Real World DragonFlyBSD Hammer DeDup figures from HiFX - Reclaiming more than 1/4th ( 30% ) Disk Space from an Almost Full Drive

2011-08-19T01:56:00.000-07:00

Finally I got free after a long busy season to work on my DragonFlyBSD
Backup Servers.
One of the Backup Server has around 10 years of Company Archives.

Short Sumary before dedup of firtst Hard Disk

Filesystem Size Used Avail Capacity Mounted on
Backup1 454G 451G 2.8G 99% /Backup1

Short Sumary after dedup of firtst Hard Disk

Filesystem Size Used Avail Capacity Mounted on
/Backup1/pfs/@@-1:00001 454G 313G 141G 69% /Backup1/Data

Reclaimed 138 GB i.e 30% of Disk space without deleting anything or
considerably affecting the perfomance of the Server.

Full Story:

The first backups server was Debian Sarge, then Debian Etch and then
OpenBSD with RAIDFRAME mirrors because it was the only Unix/Linux that
would even detect the 120 GB hard disks we had back then.
Later I turned to DragonFlyBSD due to HAMMER ( No fsck, No RAID Parity
chceks and Easy FS Snapshots )
So this Dragonfly backup server has around 10 years old backups of

1) Web files of Projects ( html, php, images etc )

2) SQL dumps both zipped and unzipped .Hammer snapshots gave me the
luxury to do

http://www.dragonflybsd.org/docs/real_time_backup_server_for_microsoft_windows__44___linux__44___bsd_and_mac_os_x_clients/

But now we have SQL dumps of induvidual databses taken every hour and
made available to the developers using snapshots in the same manner
:-)

3) MS Word, Excell Doc files - Company documents and User backups

4) PSD files and such from Designers which takes a larg space.

5) Git, SVN repositories backup

6) Virtual Machine images ( mostly qcow2 )

7) Configuration files of several servers and other details backuped
daily/hourly os some times every 15 minutes and maintained with coarse
grained snapshots without pruning.

8) Several Softwares and CD ISO images

9) Video/Audio files such as mp3,avi.flv,mpg and so on.

The OS version currently is

DragonFly v2.11.0.247.gda17d9-

DEVELOPMENT

Processor is

AMD Athlon(tm) 64 Processor 3400+ (2193.63-MHz 686-class CPU)

Memory is

real memory = 2113336320 (2015 MB)
avail memory = 2029342720 (1935 MB)

with four 500GB SATA Disks mirroring PFS from each other and also from
another Dragonfly Backup Server on a differrent floor using
'mirror-stream' started at boot using cron with an entry similar to

@reboot /sbin/hammer mirror-stream /Backup1/Data /Backup2/Data &

I have never reinstalled the OS but kept following the development
version from July 2009 so that is two years of rolling release which
is a great advantage in itself :-)

The first Disk is mounted as /Backup1 and seems to be a good Candidate
for dedup because it is almost full.

======================================================================================
Filesystem Size Used Avail Capacity Mounted on

Backup1 454G 451G 2.8G 99% /Backup1
/Backup1/pfs/@@-1:00001 454G 451G 2.8G 99% /Backup1/Data
/Backup1/pfs/@@-1:00009 454G 451G 2.8G 99% /Backup1/pkgsrc
/Backup1/pfs/@@-1:00002 454G 451G 2.8G 99% /Backup1/VersionControl
/Backup1/pfs/@@-1:00003 454G 451G 2.8G 99% /Backup1/test
/Backup1/pfs/@@-1:00005 454G 451G 2.8G 99%
/Backup1/www-5mbak/www-hot
/Backup1/pfs/@@-1:00006 454G 451G 2.8G 99%
/Backup1/mysql-1hbak/mysql-hot
/Backup1/pfs/@@-1:00007 454G 451G 2.8G 99%
/Backup1/project-docs-bak/project-docs
=======================================================================================

Full Details below.

=========================================================

Label Backup1
No. Volumes 1
FSID e182...............................................
HAMMER Version 4
Big block information
Total 58140
Used 57713 (99.27%)
Reserved 69 (0.12%)
Free 358 (0.62%)
Space information
No. Inodes 11350364
Total size 454G (487713669120 bytes)
Used 451G (99.27%)
Reserved 552M (0.12%)
Free 2.8G (0.62%)
PFS information
PFS ID Mode Snaps Mounted on
0 MASTER 0 /Backup1
1 MASTER 0 /Backup1/Data
2 MASTER 0 /Backup1/VersionControl
3 MASTER 0 /Backup1/test
5 MASTER 0 /Backup1/www-5mbak/www-hot
6 MASTER 0 /Backup1/mysql-1hbak/mysql-hot
7 MASTER 0 /Backup1/project-docs-bak/project-docs
9 MASTER 0 /Backup1/pkgsrc
==========================================================

De Duping Steps Taken:
----------------------------------

1) Version Upgrading from 4 to 6.

=================================
dfly-bkpsrv# hammer version-upgrade /Backup1 5
hammer version-upgrade: succeeded
dfly-bkpsrv# hammer version-upgrade /Backup1 6
hammer version-upgrade: succeeded
=================================

2) Simulating using 'dedup-simulate' to get an idea.

=====================================================================================

dfly-bkpsrv# hammer dedup-simulate /Backup1
Dedup-simulate /Backup1: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 0
Dedup-simulate /Backup1 succeeded
Simulated dedup ratio = 1.07

dfly-bkpsrv# hammer dedup-simulate /Backup1/Data
Dedup-simulate /Backup1/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup-simulate /Backup1/Data succeeded
Simulated dedup ratio = 1.34

dfly-bkpsrv# hammer dedup-simulate /Backup1/pkgsrc
Dedup-simulate /Backup1/pkgsrc: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 9
Dedup-simulate /Backup1/pkgsrc succeeded
Simulated dedup ratio = 1.10

dfly-bkpsrv# hammer dedup-simulate /Backup1/VersionControl
Dedup-simulate /Backup1/VersionControl: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 2
Dedup-simulate /Backup1/VersionControl succeeded
Simulated dedup ratio = 2.79

dfly-bkpsrv# hammer dedup-simulate /Backup1/test
Dedup-simulate /Backup1/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup-simulate /Backup1/test succeeded
Simulated dedup ratio = 0.00

dfly-bkpsrv# hammer dedup-simulate /Backup1/www-5mbak/www-hot
Dedup-simulate /Backup1/www-5mbak/www-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 5
Dedup-simulate /Backup1/www-5mbak/www-hot succeeded
Simulated dedup ratio = 1.39

dfly-bkpsrv# hammer dedup-simulate /Backup1/mysql-1hbak/mysql-hot
Dedup-simulate /Backup1/mysql-1hbak/mysql-hot: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 6
Dedup-simulate /Backup1/mysql-1hbak/mysql-hot succeeded
Simulated dedup ratio = 13.78

dfly-bkpsrv# hammer dedup-simulate /Backup1/project-docs-bak/project-docs
Dedup-simulate /Backup1/project-docs-bak/project-docs: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 7
Dedup-simulate /Backup1/project-docs-bak/project-docs succeeded
Simulated dedup ratio = 1.15

===================================================================================================

3) Real 'de-dup' of the Mother File System and all PFSes

=======================================================================

dfly-bkpsrv# hammer dedup /Backup1
Dedup /Backup1: objspace 8000000000000000:0000 7fffffffffffffff:ffff pfs_id 0
Dedup /Backup1 succeeded
Dedup ratio = 1.07
625 MB referenced
585 MB allocated
224 KB skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/Data
Dedup /Backup1/Data: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 1
Dedup /Backup1/Data succeeded
Dedup ratio = 1.34
259 GB referenced
193 GB allocated
40 MB skipped
1944 CRC collisions
0 SHA collisions
20 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/pkgsrc
Dedup /Backup1/pkgsrc: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 9
Dedup /Backup1/pkgsrc succeeded
Dedup ratio = 1.10
1687 MB referenced
1539 MB allocated
1718 KB skipped
3 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/VersionControl
Dedup /Backup1/VersionControl: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 2
Dedup /Backup1/VersionControl succeeded
Dedup ratio = 2.75
160 MB referenced
58 MB allocated
853 KB skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/test
Dedup /Backup1/test: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 3
Dedup /Backup1/test succeeded
Dedup ratio = 0.00
0 B referenced
0 B allocated
0 B skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/www-5mbak/www-hot
Dedup /Backup1/www-5mbak/www-hot: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 5
Dedup /Backup1/www-5mbak/www-hot succeeded
Dedup ratio = 1.39
50 GB referenced
36 GB allocated
53 MB skipped
167 CRC collisions
0 SHA collisions
0 bigblock underflows

Dedup /Backup1/mysql-1hbak/mysql-hot: objspace 8000000000000000:0000
7fffffffffffffff:ffff pfs_id 6
Dedup /Backup1/mysql-1hbak/mysql-hot succeeded
Dedup ratio = 13.78
117 GB referenced
8747 MB allocated
0 B skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows

dfly-bkpsrv# hammer dedup /Backup1/project-docs-bak/project-docs
Dedup /Backup1/project-docs-bak/project-docs: objspace
8000000000000000:0000 7fffffffffffffff:ffff pfs_id 7
Dedup /Backup1/project-docs-bak/project-docs succeeded
Dedup ratio = 1.15
247 MB referenced
215 MB allocated
102 KB skipped
0 CRC collisions
0 SHA collisions
0 bigblock underflows
=================================================================================================

Now after de-duping all PFSes on First Disk a 'df -h' gives this details

Filesystem Size Used Avail Capacity Mounted on
/Backup1/pfs/@@-1:00001 454G 313G 141G 69% /Backup1/Data

Before de-duping it was

Filesystem Size Used Avail Capacity Mounted on
Backup1 454G 451G 2.8G 99% /Backup1

So that is reclaiming 30% of Disk space amounting to 138 GB :-)

Carefull configuring designing PFSes and snapshots can save a lot of Disk space.
But de-dup can still save more :-)

In order to 'de-dup' the file system automatically every day using
'hammer cleanup' in the periodic script I have put some thing like
this in the configuration files for PFSes.

=============================================
dfly-bkpsrv# hammer config /Backup1/VersionControl/
snapshots 1d 1000d
prune 1d 15m
rebalance 1d 5m
reblock 1d 60m
recopy 30d 60m
dedup 1d 30m
==============================================

A million thanks to Matt and team for DragonFly, Hammer, de-dup,
vkernel and a lot of other gooddies comming up :-D

Dealing with packages when its PKGNAME changes

2010-07-01T21:34:00.000-07:00

cd to the new location for the renamed pkg and use

#bmake replace OLDNAME=previousName.

This way, the pkg_delete phase removes the pkg with its previous name and dependencies are adjusted. Then pkg_rr can restart the whole process on correct basis.

Installing package that cannot be installed by pkg_chk

2009-08-27T07:04:00.000-07:00

For example to install "pkg_install".

Run make update directly in pkgtools/pkg_install.

If that doesn't work for example "bootstrap-mk-files"

Try bmake USE_DESTDIR=yes package
then pkg_add -u /usr/pkgsrc/packages/All/

bootstrap-mk-files-20090807.tgz

Or the simpler equivalent of "bmake USE_DESTDIR=yes replace". Setting
USE_DESTDIR=yes with pkg_rr should just work.

Using differrent Volumes grouped to a single hammer file system

2009-07-21T06:56:00.000-07:00

commit 90ecab35525ea3637a59763d75df6712ff91728a
Author: Michael Neumann 
Date:   Tue Jul 14 15:55:02 2009 +0200

  Expand HAMMER filesystem step 1/2

  Format the volume and increase other volumes' vol_count field.
  The space of the new volume is not yet given to the filesystem (that's step 2).

  Example:

   newfs_hammer -L TEST /dev/da0
   mount_hammer /dev/da0 /hammer
   hammer expand /hammer /dev/da1
   umount /hammer
   mount_hammer /dev/da0:/dev/da1 /hammer

   # the following fails (wrong number of volumes)
   mount_hammer /dev/da0 /hammer

Summary of changes:
sbin/hammer/cmd_expand.c       |   51 +++++++++++-
sys/vfs/hammer/hammer_expand.c |  189 +++++++++++++++++++++++++++++++++++++++-
sys/vfs/hammer/hammer_ioctl.h  |    3 +
3 files changed, 239 insertions(+), 4 deletions(-)

http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/90ecab35525ea3637a59763d75df6712ff91728a


--
DragonFly BSD source repository

HammerFS Backup Howto

2009-07-03T01:29:00.000-07:00

 There are several ways to backup a HAMMER filesystem.  Many ways, in
   fact.  By default HAMMER makes daily snapshots and keeps 60 days worth
   around in "/snapshots/snap*/.".  Backing a HAMMER filesystem can be
   done using rdist or cpdup from a snapshot or it can be done using the
   mirroring directives from the hammer utility.

   When using cpdup or rdist you typically just use the same destination
   each time (and not use any hardlink or other tricks), then snapshot the
   HAMMER fs on the destination.  You also typically use a snapshot as a
   source so the data doesn't change out from under the running cpdup or
   rdist.

   When using the mirroring directives one can set up a cron job to run
   a batch every so often or one can set up a streaming mirror in loop
   using the mirror-stream directive in the HAMMER utility, along with
   bandwidth limiting options, to maintain a near-real-time backup to
   a remote HAMMER filesystem.

   Backups to HAMMER filesystems on another machine or off-site can be
   managed independantly.  I usually do a daily to a backup box on my LAN
   and a weekly off-site from the LAN backup box to the off-site backup
   box.

   In anycase, it takes a little script writing and you want to use the
   lockf(1) utility to prevent cron jobs from building up.

     -Matt
     Matthew Dillon