flifloo/BackupPC_archive_host_B2
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
54 commits 1 branch 0 tags 101 KiB
Commit graph

41 commits

Author SHA1 Message Date
Ryan Tucker
d0ea15beff Move unlinks to unlink_worker; handle OSErrors 2011-09-24 18:27:42 -04:00
Ryan Tucker
e74e89ede1 unlink_worker: don't verify the remote file 
send_file does this now, and odds are good that if it verified
once, it's going to verify a second time...
 2011-09-24 18:18:18 -04:00
Ryan Tucker
e8f7ce4cf9 minor tweaks to exception handling and logging 2011-09-23 21:09:41 -04:00
Ryan Tucker
871613a6ea Various adjustments and bug fixes 
Fix: force integers to strs in Popen arguments

Refactored verify_file out of unlink_worker
If a file already exists on S3 and is identical, don't re-upload
Add some delays, etc, to allow things to settle
Put a STOP in the gpg_queue for each process
 2011-09-22 22:21:37 -04:00
Ryan Tucker
5df1ce7594 Make logging and exceptions more consistent 
Mostly changed the format, added raises where things
are unrecoverable, and using a wider range of levels
than just "info".  Most output is now debug.
 2011-09-22 10:44:54 -04:00
Ryan Tucker
e4fd5ca8ac Fix multiprocessing bugs, add unlink_worker 
This is still under some live development, for sure :-)

Among other minor (yet significant) fixes, I've added another
worker process to handle unlinking.  This verifies that the file
was transmitted properly (or at least with the right md5sum)
before unlinking it.
 2011-09-22 10:43:28 -04:00
Ryan Tucker
3892037351 Use SSL to talk to S3, and tweak send_file 
Note: We're now dependent upon Boto 2.0.
 2011-09-21 20:39:15 -04:00
Ryan Tucker
7bd57300cf Clean up process calls, renice to 10 where available 2011-09-21 20:37:59 -04:00
Ryan Tucker
3eeda045bd Logging, etc tweaks 2011-09-21 20:35:10 -04:00
Ryan Tucker
a888eab0c3 Remove SlowFile class 
This functionality is best provided by the upstream router and/or
underlying system.
 2011-09-20 22:26:19 -04:00
Ryan Tucker
7fa6dd0115 Adapt to use a multiprocessing queue 
This will let it GPG things in the background while S3ing things in the
foreground, hopefully helping throughput!
 2011-09-20 22:24:56 -04:00
Ryan Tucker
3af02240a4 Change "logging" to "logger" throughout 2011-09-20 22:22:56 -04:00
Ryan Tucker
c4cb3c8a7f Wrap main routine in if __name__ == '__main__' 2011-09-20 16:55:23 -04:00
Ryan Tucker
45360b0b1a fixing hardcoded path (wow...) 2011-06-10 20:26:58 -04:00
Ryan Tucker
6360655bf8 Change exception to use comma format 
The "except foo as e" form, used on line 350, doesn't work with
older versions of Python; this change should allow it to work
with Python 2.x in general.

Thanks to Aaron Ciarlotta for the heads up!
 2011-01-25 22:47:51 -05:00
Ryan Tucker
01cc4abbf6 setting storage-class to REDUCED_REDUNDANCY 2010-05-19 08:52:37 -04:00
Ryan Tucker
7b14ba9488 tweaking the congestion handler a bit 2010-03-16 11:25:50 -04:00
Ryan Tucker
9edeab8662 improve handling of interrupted uploads 2010-02-03 11:58:51 -05:00
Ryan Tucker
b471ad45e8 another tweak to the congestion mitigator 2010-01-24 10:29:19 -05:00
Ryan Tucker
f246975ce4 more delayfactor tweaks; also fixing a filehead problem on restarts 2010-01-20 22:54:52 -05:00
Ryan Tucker
e09b42797a oops, old values for targetspeed/lastblocktargettime are bad 2010-01-18 17:58:44 -05:00
Ryan Tucker
f87dbdafc9 some tweaks to speed rollback logic 2010-01-18 17:06:05 -05:00
Ryan Tucker
55e1a5cb4a trapping socket.error; retry old files on a restart instead of starting anew 2010-01-09 15:00:51 -05:00
Ryan Tucker
2581d85058 fix wrong filename for complete semaphore 2010-01-06 22:43:16 -05:00
Ryan Tucker
0694a2e8c7 stringio does not have a fastread... 2010-01-06 09:03:08 -05:00
Ryan Tucker
dfa9eff725 typo in exception handling; think i know the problem 2010-01-05 23:06:20 -05:00
Ryan Tucker
db7cea8a7e plan c for writing metadata 2010-01-05 17:42:14 -05:00
Ryan Tucker
d1256d08cc dupe file checking was not; adding more basename action to send_file 2010-01-04 21:04:18 -05:00
Ryan Tucker
5a5f190929 call update_metadata() to send the updated metadata 2010-01-04 16:09:34 -05:00
Ryan Tucker
1b1ad9445e adding code to indicate when the backup isn't finalized 2010-01-04 15:59:34 -05:00
Ryan Tucker
cc73e77a3c more tweaks to the rate limiting code 2010-01-02 19:32:16 -05:00
Ryan Tucker
484922d4ef oops... removing bug in debugging code 2010-01-02 13:05:23 -05:00
Ryan Tucker
c6fc1f1d80 adding some more debugging output 2010-01-01 20:11:36 -05:00
Ryan Tucker
0b8a7b89d2 trying some adjustments to make it more responsive to ratty comms 2010-01-01 14:01:35 -05:00
Ryan Tucker
e80b23e036 fixing an infinite-pause error; sending some output to stdout to avoid timeouts 2010-01-01 13:59:54 -05:00
Ryan Tucker
5fe7494f15 some tweaks: easier retry, better maths 2009-12-30 09:35:15 -05:00
Ryan Tucker
f942a2763f increasing delayfactor punishment a bit; adding speed control 2009-12-29 18:53:14 -05:00
Ryan Tucker
dd0a141d5f adding rate limiting and retry logic 2009-12-29 14:41:50 -05:00
Ryan Tucker
07777a8628 moving bucket opening towards beginning 2009-12-29 08:48:39 -05:00
Ryan Tucker
c7a6d2a310 not using SSL for a little bit, due to stability issues -- payload still encrypted 2009-12-29 08:47:29 -05:00
Ryan Tucker
69c9fc4163 initial commit 2009-12-27 23:04:01 -05:00