qwarc

Commit graph

Autor	SHA1	Nachricht	Datum
JustAnotherArchivist	5675118877	Rename id to id_ to avoid clash with builtin	vor 3 Jahren
JustAnotherArchivist	a1e693739e	Replace DB locking with an async context manager	vor 3 Jahren
JustAnotherArchivist	15203bd991	Handle redirect traps/loops	vor 3 Jahren
JustAnotherArchivist	f8f5258197	Track redirect depth	vor 3 Jahren
JustAnotherArchivist	a3d6fb35f8	Turn response handlers into kwarg-only functions for easier extendability without breaking existing code	vor 3 Jahren
JustAnotherArchivist	6cc4adb901	Remove stray TODO The DB creation operates with a DB lock, so that code can't run while another process is filling the DB; it would block on obtaining the lock a few lines prior instead.	vor 3 Jahren
JustAnotherArchivist	c5604ef965	Simplify header merging	vor 3 Jahren
JustAnotherArchivist	59ae1183d2	Add fromResponse parameter for URL completion and automatic Referer header	vor 3 Jahren
JustAnotherArchivist	2324216016	Add baseUrl and evaluate incomplete URLs relative to it	vor 3 Jahren
JustAnotherArchivist	b30ccf8bf8	Move response/exception history to ClientResponse.qhistory It is rarely necessary to access the history, and the tuple return value clutters the spec file code. As a consequence, it's no longer possible to return None if an error occurred without losing the history. To replace that, this also introduces a DummyClientResponse, which is kind of ClientResponse-like, has the same qhistory attribute, and evaluates to False when cast to bool (such that the intuitive `if response` works as expected).	vor 3 Jahren
JustAnotherArchivist	e69527c715	Add defaultResponseHandler on the Item level	vor 3 Jahren
JustAnotherArchivist	03336e4988	Add item to response handler arguments (e.g. for logging)	vor 3 Jahren
JustAnotherArchivist	6bdcfe71f0	Refactor database creation and item generation: call `Item.generate()` on every qwarc run and dedupe its output, allowing the addition of further items by modifying the spec file	vor 3 Jahren
JustAnotherArchivist	c878241f24	Switch from concurrent.futures.CancelledError to asyncio.CancelledError Since Python 3.8, the latter does not inherit from the former anymore.	vor 3 Jahren
JustAnotherArchivist	749158b97a	Use the Future's result directly rather than awaiting again The asyncio documentation does not specify whether awaiting a Future multiple times is supported or not: https://bugs.python.org/issue41275	vor 3 Jahren
JustAnotherArchivist	a85e80ffa2	Configurable request timeout	vor 3 Jahren
JustAnotherArchivist	429ac94689	Make it possible to override and remove headers	vor 3 Jahren
JustAnotherArchivist	e40be54578	Document verify_ssl parameter	vor 3 Jahren
JustAnotherArchivist	d3437bde19	Move default headers to qwarc.const	vor 3 Jahren
JustAnotherArchivist	1678075a89	Log traceback on exceptions raised from an item	vor 4 Jahren
JustAnotherArchivist	b1a1c03f7e	Handle STOP file and high memory usage before full disk to allow stopping while the disk is above the limit	vor 4 Jahren
JustAnotherArchivist	dd44d9b174	Adjust logging levels: log individual request failures only at WARNING and cancelled tasks at ERROR level	vor 4 Jahren
JustAnotherArchivist	91035d769c	Catch exceptions in Item.process and mark the items as errors instead of crashing	vor 4 Jahren
JustAnotherArchivist	69984765b3	Fix taskType typo silencing cancellation warnings	vor 4 Jahren
JustAnotherArchivist	c263ad0b03	Return ClientResponse object from fetch only if the retrieval was successful If an exception was raised and caught, the object is still present in the history.	vor 4 Jahren
JustAnotherArchivist	cb0d11284e	Write only successful retrievals (i.e. ones that don't cause an exception) to WARC	vor 4 Jahren
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	vor 4 Jahren
JustAnotherArchivist	08d96b37c5	Support deep/multiple inheritance from Item	vor 4 Jahren
JustAnotherArchivist	9d8de13775	Add Item.flush_subitems to flush the new subitems to the database while the item is still being processed This also renames add_item to add_subitem for clarity.	vor 4 Jahren
JustAnotherArchivist	50b936b18c	Refactor QWARC class to keep relevant variables in instance attributes instead of local variables	vor 4 Jahren
JustAnotherArchivist	c5d8d93166	Remove stray whitespace	vor 4 Jahren
JustAnotherArchivist	7d53577522	Add parameter for disabling SSL/TLS certificate validation	vor 4 Jahren
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	vor 4 Jahren
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	vor 4 Jahren
JustAnotherArchivist	d751844626	Fix starting another item before stopping on STOP file or memory limit exceedance	vor 4 Jahren
JustAnotherArchivist	2b0778f9b5	Remove leftovers from initial code rewrite	vor 4 Jahren
JustAnotherArchivist	ab22966fef	Add to log which item a message is coming from	vor 4 Jahren
JustAnotherArchivist	6fafd32685	Error when the retries are exceeded	vor 4 Jahren
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	vor 4 Jahren
JustAnotherArchivist	5008e6e8cd	Deduplicate items	vor 4 Jahren
JustAnotherArchivist	46c95e2157	Disable decoding the response content chardet can be very slow (https://github.com/chardet/chardet/issues/29 https://github.com/psf/requests/issues/2359) and the decoding may be unnecessary if it's binary content.	vor 4 Jahren
JustAnotherArchivist	ad22a2327a	Support adding headers to individual requests	vor 5 Jahren
JustAnotherArchivist	67076f964c	Add support for POST requests	vor 5 Jahren
JustAnotherArchivist	c1574a06c9	Fix sleep task type	vor 5 Jahren
JustAnotherArchivist	e0ca88c807	Fix reference to get_rss	vor 5 Jahren
JustAnotherArchivist	8a8935810d	Fix references to memory and disk space check methods	vor 5 Jahren
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	vor 5 Jahren
JustAnotherArchivist	e892a6b6a7	Initial commit	vor 5 Jahren

48 Commits (56751188774506e307e1c91ac1a304358240adfe)