JustAnotherArchivist
|
f5c3eb42b3
|
WIP attempt to remove warcio
|
3 年之前 |
JustAnotherArchivist
|
cb0d11284e
|
Write only successful retrievals (i.e. ones that don't cause an exception) to WARC
|
4 年之前 |
JustAnotherArchivist
|
1214409a0b
|
Flush big responses to a temporary file instead of trying to keep everything in-memory
|
4 年之前 |
JustAnotherArchivist
|
37dbcfad21
|
Don't write responses to WARC that triggered an exception
For example, if the connection breaks while retrieving a response but after the headers have been parsed, the response body would be incomplete.
|
4 年之前 |
JustAnotherArchivist
|
f038cf91db
|
Fix unfound distribution handling
|
4 年之前 |
JustAnotherArchivist
|
a5dfd5c805
|
Write spec file + its dependencies and command line to meta WARC
|
4 年之前 |
JustAnotherArchivist
|
e99e2304c9
|
Write meta WARC with log file
|
4 年之前 |
JustAnotherArchivist
|
85d78cee13
|
Add warcinfo record with version information on Python, system, and dependencies
|
4 年之前 |
JustAnotherArchivist
|
6fafd32685
|
Error when the retries are exceeded
|
4 年之前 |
JustAnotherArchivist
|
8647d6b396
|
Use f-strings instead of str.format
|
4 年之前 |
JustAnotherArchivist
|
85f6f7bd82
|
Make qwarc.utils.handle_response_limit_error_retries more useful by passing the deferring handler as an argument
|
5 年之前 |
JustAnotherArchivist
|
2d52e78d85
|
Fix reference to aiohttp.CientError
|
5 年之前 |
JustAnotherArchivist
|
e892a6b6a7
|
Initial commit
|
5 年之前 |