Ryan Tucker
e74e89ede1
unlink_worker: don't verify the remote file
...
send_file does this now, and odds are good that if it verified
once, it's going to verify a second time...
2011-09-24 18:18:18 -04:00
Ryan Tucker
e8f7ce4cf9
minor tweaks to exception handling and logging
2011-09-23 21:09:41 -04:00
Ryan Tucker
871613a6ea
Various adjustments and bug fixes
...
Fix: force integers to strs in Popen arguments
Refactored verify_file out of unlink_worker
If a file already exists on S3 and is identical, don't re-upload
Add some delays, etc, to allow things to settle
Put a STOP in the gpg_queue for each process
2011-09-22 22:21:37 -04:00
Ryan Tucker
5df1ce7594
Make logging and exceptions more consistent
...
Mostly changed the format, added raises where things
are unrecoverable, and using a wider range of levels
than just "info". Most output is now debug.
2011-09-22 10:44:54 -04:00
Ryan Tucker
e4fd5ca8ac
Fix multiprocessing bugs, add unlink_worker
...
This is still under some live development, for sure :-)
Among other minor (yet significant) fixes, I've added another
worker process to handle unlinking. This verifies that the file
was transmitted properly (or at least with the right md5sum)
before unlinking it.
2011-09-22 10:43:28 -04:00
Ryan Tucker
3892037351
Use SSL to talk to S3, and tweak send_file
...
Note: We're now dependent upon Boto 2.0.
2011-09-21 20:39:15 -04:00
Ryan Tucker
7bd57300cf
Clean up process calls, renice to 10 where available
2011-09-21 20:37:59 -04:00
Ryan Tucker
3eeda045bd
Logging, etc tweaks
2011-09-21 20:35:10 -04:00
Ryan Tucker
a888eab0c3
Remove SlowFile class
...
This functionality is best provided by the upstream router and/or
underlying system.
2011-09-20 22:26:19 -04:00
Ryan Tucker
7fa6dd0115
Adapt to use a multiprocessing queue
...
This will let it GPG things in the background while S3ing things in the
foreground, hopefully helping throughput!
2011-09-20 22:24:56 -04:00
Ryan Tucker
3af02240a4
Change "logging" to "logger" throughout
2011-09-20 22:22:56 -04:00
Ryan Tucker
c4cb3c8a7f
Wrap main routine in if __name__ == '__main__'
2011-09-20 16:55:23 -04:00
Ryan Tucker
45360b0b1a
fixing hardcoded path (wow...)
2011-06-10 20:26:58 -04:00
Ryan Tucker
6360655bf8
Change exception to use comma format
...
The "except foo as e" form, used on line 350, doesn't work with
older versions of Python; this change should allow it to work
with Python 2.x in general.
Thanks to Aaron Ciarlotta for the heads up!
2011-01-25 22:47:51 -05:00
Ryan Tucker
01cc4abbf6
setting storage-class to REDUCED_REDUNDANCY
2010-05-19 08:52:37 -04:00
Ryan Tucker
7b14ba9488
tweaking the congestion handler a bit
2010-03-16 11:25:50 -04:00
Ryan Tucker
9edeab8662
improve handling of interrupted uploads
2010-02-03 11:58:51 -05:00
Ryan Tucker
b471ad45e8
another tweak to the congestion mitigator
2010-01-24 10:29:19 -05:00
Ryan Tucker
f246975ce4
more delayfactor tweaks; also fixing a filehead problem on restarts
2010-01-20 22:54:52 -05:00
Ryan Tucker
e09b42797a
oops, old values for targetspeed/lastblocktargettime are bad
2010-01-18 17:58:44 -05:00
Ryan Tucker
f87dbdafc9
some tweaks to speed rollback logic
2010-01-18 17:06:05 -05:00
Ryan Tucker
55e1a5cb4a
trapping socket.error; retry old files on a restart instead of starting anew
2010-01-09 15:00:51 -05:00
Ryan Tucker
2581d85058
fix wrong filename for complete semaphore
2010-01-06 22:43:16 -05:00
Ryan Tucker
0694a2e8c7
stringio does not have a fastread...
2010-01-06 09:03:08 -05:00
Ryan Tucker
dfa9eff725
typo in exception handling; think i know the problem
2010-01-05 23:06:20 -05:00
Ryan Tucker
db7cea8a7e
plan c for writing metadata
2010-01-05 17:42:14 -05:00
Ryan Tucker
d1256d08cc
dupe file checking was not; adding more basename action to send_file
2010-01-04 21:04:18 -05:00
Ryan Tucker
5a5f190929
call update_metadata() to send the updated metadata
2010-01-04 16:09:34 -05:00
Ryan Tucker
1b1ad9445e
adding code to indicate when the backup isn't finalized
2010-01-04 15:59:34 -05:00
Ryan Tucker
cc73e77a3c
more tweaks to the rate limiting code
2010-01-02 19:32:16 -05:00
Ryan Tucker
484922d4ef
oops... removing bug in debugging code
2010-01-02 13:05:23 -05:00
Ryan Tucker
c6fc1f1d80
adding some more debugging output
2010-01-01 20:11:36 -05:00
Ryan Tucker
0b8a7b89d2
trying some adjustments to make it more responsive to ratty comms
2010-01-01 14:01:35 -05:00
Ryan Tucker
e80b23e036
fixing an infinite-pause error; sending some output to stdout to avoid timeouts
2010-01-01 13:59:54 -05:00
Ryan Tucker
5fe7494f15
some tweaks: easier retry, better maths
2009-12-30 09:35:15 -05:00
Ryan Tucker
f942a2763f
increasing delayfactor punishment a bit; adding speed control
2009-12-29 18:53:14 -05:00
Ryan Tucker
dd0a141d5f
adding rate limiting and retry logic
2009-12-29 14:41:50 -05:00
Ryan Tucker
07777a8628
moving bucket opening towards beginning
2009-12-29 08:48:39 -05:00
Ryan Tucker
c7a6d2a310
not using SSL for a little bit, due to stability issues -- payload still encrypted
2009-12-29 08:47:29 -05:00
Ryan Tucker
69c9fc4163
initial commit
2009-12-27 23:04:01 -05:00
