flifloo/BackupPC_archive_host_B2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
67 commits 1 branch 0 tags 101 KiB
Commit graph

67 commits

This branch
This branch
All branches
Author SHA1 Message Date
Florian Charlaix
5461dec55c Use account_id instead of access_key for bucket names and trim it trim to 50 2022-10-19 13:38:03 +02:00
flifloo
4cbaf0c52f Update backup manager script 2022-10-18 00:02:58 +02:00
flifloo
dca2dbdad8 Rewrite a lot of BackupPC_archiveHost_s3 to b2 2022-04-08 18:20:55 +02:00
Ryan Tucker
e9ee689844 pyflakes / PEP8 compliance 2013-02-06 19:00:32 -05:00
Ryan Tucker
4f4574b727 Reorganize script to be less hacky 
A class for caching, less shady delete logic, etc.
 2013-02-06 18:02:07 -05:00
Ryan Tucker
59eedb580b Start the most needed backups first 
Uses a somewhat-hackish scoring algorithm to decide which backup
to start first, but it's better than alphabetical order!
 2013-02-06 16:21:45 -05:00
Ryan Tucker
5b4d78fea3 Don't encrypt COMPLETE files nor choke if they are 
May fix:
https://github.com/rtucker/backuppc-archive-s3/issues/2
 2012-11-12 16:52:02 -05:00
Ryan Tucker
c2ddefb316 Update link to Boto source 
Fixes issue #1: https://github.com/rtucker/backuppc-archive-s3/issues/1
 2012-05-12 16:08:45 -04:00
Ryan Tucker
a7c9100b7a autostart: Use ArchiveSplit from BackupPC's conf 
This is probably a bug; I ought to report it if it isn't
already fixed.
 2011-10-02 11:48:50 -04:00
Ryan Tucker
91296f1775 Clarifying license stuff. 2011-10-02 11:36:20 -04:00
Ryan Tucker
7ff0a277a6 Adding BackupPC_archiveStart from BackupPC source 
This is the unedited version from my copy of BackupPC.
I'm planning to edit this in order to fix a couple bugs...

-rwxr-xr-x 1 root root 4036 2011-06-30 04:41 /usr/share/backuppc/bin/BackupPC_archiveStart

backuppc:
  Installed: 3.2.0-3ubuntu4~lucid1
  Candidate: 3.2.0-3ubuntu4~lucid1
  Version table:
 *** 3.2.0-3ubuntu4~lucid1 0
        500 http://archive.ubuntu.com/ubuntu/ lucid-backports/main Packages
        100 /var/lib/dpkg/status
     3.1.0-9ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid/main Packages
 2011-10-02 11:30:34 -04:00
Ryan Tucker
abbbbba970 backup-manager: add ability to start archives 
This will allow things to run completely on autopilot, creating
archives on S3 when necessary.
 2011-09-29 11:22:54 -04:00
Ryan Tucker
b5533a843f Fix args for unlink_worker subprocess 
unlink_worker was not starting and the main process was
dying, due to args being the queue itself, not a tuple
containing the queue.
 2011-09-29 09:09:01 -04:00
Ryan Tucker
d0ea15beff Move unlinks to unlink_worker; handle OSErrors 2011-09-24 18:27:42 -04:00
Ryan Tucker
e74e89ede1 unlink_worker: don't verify the remote file 
send_file does this now, and odds are good that if it verified
once, it's going to verify a second time...
 2011-09-24 18:18:18 -04:00
Ryan Tucker
e8f7ce4cf9 minor tweaks to exception handling and logging 2011-09-23 21:09:41 -04:00
Ryan Tucker
871613a6ea Various adjustments and bug fixes 
Fix: force integers to strs in Popen arguments

Refactored verify_file out of unlink_worker
If a file already exists on S3 and is identical, don't re-upload
Add some delays, etc, to allow things to settle
Put a STOP in the gpg_queue for each process
 2011-09-22 22:21:37 -04:00
Ryan Tucker
7bda017245 Update documentation to reflect changes 2011-09-22 10:57:50 -04:00
Ryan Tucker
5df1ce7594 Make logging and exceptions more consistent 
Mostly changed the format, added raises where things
are unrecoverable, and using a wider range of levels
than just "info".  Most output is now debug.
 2011-09-22 10:44:54 -04:00
Ryan Tucker
e4fd5ca8ac Fix multiprocessing bugs, add unlink_worker 
This is still under some live development, for sure :-)

Among other minor (yet significant) fixes, I've added another
worker process to handle unlinking.  This verifies that the file
was transmitted properly (or at least with the right md5sum)
before unlinking it.
 2011-09-22 10:43:28 -04:00
Ryan Tucker
3892037351 Use SSL to talk to S3, and tweak send_file 
Note: We're now dependent upon Boto 2.0.
 2011-09-21 20:39:15 -04:00
Ryan Tucker
7bd57300cf Clean up process calls, renice to 10 where available 2011-09-21 20:37:59 -04:00
Ryan Tucker
3eeda045bd Logging, etc tweaks 2011-09-21 20:35:10 -04:00
Ryan Tucker
d07636a5ff Merge branch 'master' of github.com:rtucker/backuppc-archive-s3 2011-09-20 22:35:54 -04:00
Ryan Tucker
a888eab0c3 Remove SlowFile class 
This functionality is best provided by the upstream router and/or
underlying system.
 2011-09-20 22:26:19 -04:00
Ryan Tucker
7fa6dd0115 Adapt to use a multiprocessing queue 
This will let it GPG things in the background while S3ing things in the
foreground, hopefully helping throughput!
 2011-09-20 22:24:56 -04:00
Ryan Tucker
3af02240a4 Change "logging" to "logger" throughout 2011-09-20 22:22:56 -04:00
Ryan Tucker
c4cb3c8a7f Wrap main routine in if __name__ == '__main__' 2011-09-20 16:55:23 -04:00
Ryan Tucker
c3730f8719 markdown formatting fixes 2011-06-10 21:14:53 -04:00
Ryan Tucker
96e8cf12e4 add something that looks like documentation 2011-06-10 21:04:08 -04:00
Ryan Tucker
fb2aaeae0f make the backup-manager.py --help more useful 2011-06-10 20:55:59 -04:00
Ryan Tucker
45360b0b1a fixing hardcoded path (wow...) 2011-06-10 20:26:58 -04:00
Ryan Tucker
6360655bf8 Change exception to use comma format 
The "except foo as e" form, used on line 350, doesn't work with
older versions of Python; this change should allow it to work
with Python 2.x in general.

Thanks to Aaron Ciarlotta for the heads up!
 2011-01-25 22:47:51 -05:00
Ryan Tucker
01cc4abbf6 setting storage-class to REDUCED_REDUNDANCY 2010-05-19 08:52:37 -04:00
Ryan Tucker
a752c3c1dd fixing a logic bug (i think) 2010-03-18 09:09:39 -04:00
Ryan Tucker
7b1d0356e0 add --keep option to keep the n most recent backups when deleting by age 2010-03-16 11:26:49 -04:00
Ryan Tucker
2b5b57b888 respect the test mode when wanting to delete the COMPLETE file (oops) 2010-03-16 11:26:32 -04:00
Ryan Tucker
7b14ba9488 tweaking the congestion handler a bit 2010-03-16 11:25:50 -04:00
Ryan Tucker
a9262b201a oops, used the wrong variable name 2010-03-09 17:23:22 -05:00
Ryan Tucker
1fea10bc03 more accurate backup age handling; also nuking the .COMPLETE file on a delete 2010-02-18 15:34:58 -05:00
Ryan Tucker
9edeab8662 improve handling of interrupted uploads 2010-02-03 11:58:51 -05:00
Ryan Tucker
b471ad45e8 another tweak to the congestion mitigator 2010-01-24 10:29:19 -05:00
Ryan Tucker
f246975ce4 more delayfactor tweaks; also fixing a filehead problem on restarts 2010-01-20 22:54:52 -05:00
Ryan Tucker
e09b42797a oops, old values for targetspeed/lastblocktargettime are bad 2010-01-18 17:58:44 -05:00
Ryan Tucker
f87dbdafc9 some tweaks to speed rollback logic 2010-01-18 17:06:05 -05:00
Ryan Tucker
55e1a5cb4a trapping socket.error; retry old files on a restart instead of starting anew 2010-01-09 15:00:51 -05:00
Ryan Tucker
2581d85058 fix wrong filename for complete semaphore 2010-01-06 22:43:16 -05:00
Ryan Tucker
0694a2e8c7 stringio does not have a fastread... 2010-01-06 09:03:08 -05:00
Ryan Tucker
dfa9eff725 typo in exception handling; think i know the problem 2010-01-05 23:06:20 -05:00
Ryan Tucker
db7cea8a7e plan c for writing metadata 2010-01-05 17:42:14 -05:00