Bareos Performance erhöhen mit LZ4 statt GZIP

Nach dem Austausch von Datenträgern (Backup To Disk) wunderte ich mich über die nach wie vor geringe Schreib-Geschwindigkeit bei Bareos-Backups, die meistens bei 12 Mbyte/s lag, würden die Datenträger doch locker mindestens das 10fache an Geschwindigkeit entgegennehmen können.

Auf der Suche nach den Ursachen schaltete ich die "Software Compression" ab und siehe da, die Performance stieg drastisch an. Zurück zu GZIP: Wieder alles langsam, obwohl sich die CPU langweilte, während das Backup lief.

Auf der Suche nach Lösungen fand ich in der Bareos-Dokumentation weitere Komprimierungsoptionen und probierte LZ4 aus, die Ergebnisse seht ihr folgend in der Grafik.

Da solche Tests ja immer sehr von der Umgebung abhängen, führte ich Tests mit Bareos 17 und Bareos 19 auf verschiedender Hardware aus, es kam jeweils Backup To Disk (Bei XEON mit RAID1) zum Einsatz, die Cataloge unterscheiden sich bezüglich der Größe stark.

Ergebnis eines Backups mit LZ4

11-May 23:02 debian2.fritz.box-dir JobId 1940: shell command: run BeforeJob "sudo /etc/bareos/scripts/start-stop-vms stop"
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Try to cleanly shut down all running KVM domains...
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Domain awxdeb is being shutdown
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: 
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Domain filesrv2 is being shutdown
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: 
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Domain git is being shutdown
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: 
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Domain printsrv2 is being shutdown
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: 
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: Domain puppetmaster1 is being shutdown
11-May 23:02 debian2.fritz.box-dir JobId 1940: BeforeJob: 
11-May 23:02 debian2.fritz.box-dir JobId 1940: Start Backup JobId 1940, Job=debian2.fritz.box-libvirt-images.2020-05-11_23.02.33_37
11-May 23:02 debian2.fritz.box-dir JobId 1940: Connected Storage daemon at debian2.fritz.box:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-02-01-03:00:18-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-03-01-03:00:16-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-04-01-03:06:45-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-05-01-03:00:15-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:29:03-Incremental-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 23:02 debian2.fritz.box-dir JobId 1940: Created new Volume "debian2.fritz.box-libvirt-images-2020-05-11-23:02:49-Incremental-Backup" in catalog.
11-May 23:02 debian2.fritz.box-dir JobId 1940: Using Device "debian2.fritz.box-libvirt-imagesStorage" to write.
11-May 23:02 debian2.fritz.box-dir JobId 1940: Probing client protocol... (result will be saved until config reload)
11-May 23:02 debian2.fritz.box-dir JobId 1940: Connected Client: debian2.fritz.box-fd at debian2.fritz.box:9102, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 23:02 debian2.fritz.box-dir JobId 1940:    Handshake: Immediate TLS 11-May 23:02 debian2.fritz.box-dir JobId 1940:  Encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 23:02 debian2.fritz.box-fd JobId 1940: Connected Storage daemon at debian2.fritz.box:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 23:02 debian2.fritz.box-fd JobId 1940: Extended attribute support is enabled
11-May 23:02 debian2.fritz.box-sd JobId 1940: Labeled new Volume "debian2.fritz.box-libvirt-images-2020-05-11-23:02:49-Incremental-Backup" on device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images).
11-May 23:02 debian2.fritz.box-fd JobId 1940: ACL support is enabled
11-May 23:02 debian2.fritz.box-sd JobId 1940: Wrote label to prelabeled Volume "debian2.fritz.box-libvirt-images-2020-05-11-23:02:49-Incremental-Backup" on device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images)
11-May 23:02 debian2.fritz.box-dir JobId 1940: Volume used once. Marking Volume "debian2.fritz.box-libvirt-images-2020-05-11-23:02:49-Incremental-Backup" as Used.
11-May 23:08 debian2.fritz.box-sd JobId 1940: Releasing device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images).
11-May 23:08 debian2.fritz.box-sd JobId 1940: Elapsed time=00:05:46, Transfer rate=84.55 M Bytes/second
11-May 23:08 debian2.fritz.box-dir JobId 1940: Insert of attributes batch table with 5 entries start
11-May 23:08 debian2.fritz.box-dir JobId 1940: Insert of attributes batch table done
11-May 23:08 debian2.fritz.box-dir JobId 1940: Bareos debian2.fritz.box-dir 19.2.7 (16Apr20):
  Build OS:               Linux-3.10.0-1062.18.1.el7.x86_64 debian Debian GNU/Linux 9.12 (stretch)
  JobId:                  1940
  Job:                    debian2.fritz.box-libvirt-images.2020-05-11_23.02.33_37
  Backup Level:           Incremental, since=2020-05-11 22:52:17
  Client:                 "debian2.fritz.box-fd" 19.2.7 (16Apr20) Linux-3.10.0-1062.18.1.el7.x86_64,debian,Debian GNU/Linux 9.12 (stretch)
  FileSet:                "debian2.fritz.box-libvirt-images" 2019-11-01 03:00:00
  Pool:                   "debian2.fritz.box-libvirt-images" (From Job resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "debian2.fritz.box-libvirt-imagesFile" (From Job resource)
  Scheduled time:         11-May-2020 23:02:32
  Start time:             11-May-2020 23:02:49
  End time:               11-May-2020 23:08:35
  Elapsed time:           5 mins 46 secs
  Priority:               10
  FD Files Written:       5
  SD Files Written:       5
  FD Bytes Written:       29,257,405,153 (29.25 GB)
  SD Bytes Written:       29,257,405,782 (29.25 GB)
  Rate:                   84559.0 KB/s
  Software Compression:   54.6 % (lz4)
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         debian2.fritz.box-libvirt-images-2020-05-11-23:02:49-Incremental-Backup
  Volume Session Id:      61
  Volume Session Time:    1588772034
  Last Volume Bytes:      29,285,544,915 (29.28 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Bareos binary info:     bareos.org build: Get official binaries and vendor support on bareos.com
  Termination:            Backup OK

11-May 23:08 debian2.fritz.box-dir JobId 1940: shell command: run AfterJob "sudo /etc/bareos/scripts/start-stop-vms start"
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: Domain awxdeb started
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: 
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: Domain filesrv2 started
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: 
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: Domain git started
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: 
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: Domain printsrv2 started
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: 
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: Domain puppetmaster1 started
11-May 23:08 debian2.fritz.box-dir JobId 1940: AfterJob: 

Ergebnis eines Backups mit GZIP

11-May 22:16 debian2.fritz.box-dir JobId 1936: shell command: run BeforeJob "sudo /etc/bareos/scripts/start-stop-vms stop"
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Try to cleanly shut down all running KVM domains...
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain puppetmaster1 is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain awxdeb is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain git is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain filesrv2 is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain printsrv2 is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: Domain centos7kackbuechse1 is being shutdown
11-May 22:16 debian2.fritz.box-dir JobId 1936: BeforeJob: 
11-May 22:16 debian2.fritz.box-dir JobId 1936: Start Backup JobId 1936, Job=debian2.fritz.box-libvirt-images.2020-05-11_22.16.01_21
11-May 22:16 debian2.fritz.box-dir JobId 1936: Connected Storage daemon at debian2.fritz.box:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume "debian2.fritz.box-libvirt-images-2020-01-01-03:00:08-Full-Backup" has Volume Retention of 11232000 sec. and has 1 jobs that will be pruned
11-May 22:16 debian2.fritz.box-dir JobId 1936: Purging the following JobIds: 762
11-May 22:16 debian2.fritz.box-dir JobId 1936: There are no more Jobs associated with Volume "debian2.fritz.box-libvirt-images-2020-01-01-03:00:08-Full-Backup". Marking it purged.
11-May 22:16 debian2.fritz.box-dir JobId 1936: All records pruned from Volume "debian2.fritz.box-libvirt-images-2020-01-01-03:00:08-Full-Backup"; marking it "Purged"
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume "debian2.fritz.box-libvirt-images-2020-02-01-03:00:18-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume "debian2.fritz.box-libvirt-images-2020-03-01-03:00:16-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume "debian2.fritz.box-libvirt-images-2020-04-01-03:06:45-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume "debian2.fritz.box-libvirt-images-2020-05-01-03:00:15-Full-Backup" has Volume Retention of 11232000 sec. and has 0 jobs that will be pruned
11-May 22:16 debian2.fritz.box-dir JobId 1936: Created new Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup" in catalog.
11-May 22:16 debian2.fritz.box-dir JobId 1936: Using Device "debian2.fritz.box-libvirt-imagesStorage" to write.
11-May 22:16 debian2.fritz.box-dir JobId 1936: Connected Client: debian2.fritz.box-fd at debian2.fritz.box:9102, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 22:16 debian2.fritz.box-dir JobId 1936:  Handshake: Immediate TLS 11-May 22:16 debian2.fritz.box-dir JobId 1936:  Encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 22:16 debian2.fritz.box-fd JobId 1936: Connected Storage daemon at debian2.fritz.box:9103, encryption: ECDHE-PSK-CHACHA20-POLY1305
11-May 22:16 debian2.fritz.box-fd JobId 1936: Extended attribute support is enabled
11-May 22:16 debian2.fritz.box-fd JobId 1936: ACL support is enabled
11-May 22:16 debian2.fritz.box-sd JobId 1936: Labeled new Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup" on device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images).
11-May 22:16 debian2.fritz.box-sd JobId 1936: Wrote label to prelabeled Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup" on device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images)
11-May 22:16 debian2.fritz.box-dir JobId 1936: Volume used once. Marking Volume "debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup" as Used.
11-May 22:49 debian2.fritz.box-sd JobId 1936: Releasing device "debian2.fritz.box-libvirt-imagesStorage" (/backup_partition/bareos/debian2-libvirt-images).
11-May 22:49 debian2.fritz.box-sd JobId 1936: Elapsed time=00:33:28, Transfer rate=12.36 M Bytes/second
11-May 22:49 debian2.fritz.box-dir JobId 1936: Insert of attributes batch table with 6 entries start
11-May 22:49 debian2.fritz.box-dir JobId 1936: Insert of attributes batch table done
11-May 22:49 debian2.fritz.box-dir JobId 1936: Bareos debian2.fritz.box-dir 19.2.7 (16Apr20):
  Build OS:               Linux-3.10.0-1062.18.1.el7.x86_64 debian Debian GNU/Linux 9.12 (stretch)
  JobId:                  1936
  Job:                    debian2.fritz.box-libvirt-images.2020-05-11_22.16.01_21
  Backup Level:           Incremental, since=2020-05-01 03:00:17
  Client:                 "debian2.fritz.box-fd" 19.2.7 (16Apr20) Linux-3.10.0-1062.18.1.el7.x86_64,debian,Debian GNU/Linux 9.12 (stretch)
  FileSet:                "debian2.fritz.box-libvirt-images" 2019-11-01 03:00:00
  Pool:                   "debian2.fritz.box-libvirt-images" (From Job resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "debian2.fritz.box-libvirt-imagesFile" (From Job resource)
  Scheduled time:         11-May-2020 22:16:00
  Start time:             11-May-2020 22:16:17
  End time:               11-May-2020 22:49:45
  Elapsed time:           33 mins 28 secs
  Priority:               10
  FD Files Written:       6
  SD Files Written:       6
  FD Bytes Written:       24,837,545,704 (24.83 GB)
  SD Bytes Written:       24,837,546,469 (24.83 GB)
  Rate:                   12369.3 KB/s
  Software Compression:   66.0 % (gzip)
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         debian2.fritz.box-libvirt-images-2020-05-11-22:16:17-Incremental-Backup
  Volume Session Id:      57
  Volume Session Time:    1588772034
  Last Volume Bytes:      24,864,791,662 (24.86 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Bareos binary info:     bareos.org build: Get official binaries and vendor support on bareos.com
  Termination:            Backup OK

11-May 22:49 debian2.fritz.box-dir JobId 1936: shell command: run AfterJob "sudo /etc/bareos/scripts/start-stop-vms start"
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: Domain awxdeb started
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: 
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: Domain filesrv2 started
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: 
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: Domain git started
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: 
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: Domain printsrv2 started
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: 
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: Domain puppetmaster1 started
11-May 22:49 debian2.fritz.box-dir JobId 1936: AfterJob: 

Beispiel für eine Konfiguration mit LZ4 als Software-Komprimierungs-Methode

FileSet {
  Name = "debian2.fritz.box-libvirt-images"
  Include {
    Options {
    compression = LZ4
    recurse = yes
    signature = SHA1
    onefs = no
    }
    File = /var/lib/libvirt/images
    }
    Exclude {
    }
}

Benutzer von Bacula könnten die selben Auswirkungen bemerken.