qwarc

コミットグラフ

作成者	SHA1	メッセージ	日付
JustAnotherArchivist	f5c3eb42b3	WIP attempt to remove warcio	3年前
JustAnotherArchivist	820384fe1e	Stop deduping small responses For small responses, the additional headers for the revisit outweigh the payload truncation savings. The chosen limit of 100 bytes is completely arbitrary and not backed by any real-world data.	4年前
JustAnotherArchivist	461cedbbde	Avoid temporary files created by warcio due to not knowing the record payload length	4年前
JustAnotherArchivist	1214409a0b	Flush big responses to a temporary file instead of trying to keep everything in-memory	4年前
JustAnotherArchivist	93df9cd18d	Get rid of the temporary extra log file and read the plain file instead	4年前
JustAnotherArchivist	08c3d55376	Add comment on block digest workaround (cf. `f14a664b`)	4年前
JustAnotherArchivist	413435b7fb	Work around warcio not writing the correct WARC-Profile header for revisit records on WARC/1.1 https://github.com/webrecorder/warcio/issues/94	4年前
JustAnotherArchivist	8ee9b20718	Remove WARC-Target-URI header from warcinfo record WARC 1.1 specification, section 5.14: "A ‘warcinfo’ record shall not have a WARC-Target-URI field."	4年前
JustAnotherArchivist	f14a664b1c	Work around warcio not writing a block digest for warcinfo records (https://github.com/webrecorder/warcio/issues/87 ) The length has to be set manually because otherwise warcio will automatically remove the header again.	4年前
JustAnotherArchivist	bd14ab3901	Fix crash due to closing the log handler on reaching the max WARC size	4年前
JustAnotherArchivist	08117630b0	Remove warcinfo record in each data WARC and refer to the process's warcinfo record in the meta WARC instead	4年前
JustAnotherArchivist	26aab15605	urn:X-qwarc instead of urn:qwarc	4年前
JustAnotherArchivist	50d46ad51c	Use log filename in the target URI of the log resource record	4年前
JustAnotherArchivist	e093211496	Set content type for resource records	4年前
JustAnotherArchivist	ae46b53401	Always write a WARC-Warcinfo-ID header	4年前
JustAnotherArchivist	23fcdd4026	Write microsecond dates for request and response records	4年前
JustAnotherArchivist	3030ad10ab	Mark private API accordingly	4年前
JustAnotherArchivist	e0b4104d21	Remove log handler before writing log record since that requires closing the stream	4年前
JustAnotherArchivist	6cfd352f68	Write WARC/1.1 files	4年前
JustAnotherArchivist	e1ad5c232e	Write warcinfo and resource records in meta WARC on firing up qwarc rather than at the end	4年前
JustAnotherArchivist	a5dfd5c805	Write spec file + its dependencies and command line to meta WARC	4年前
JustAnotherArchivist	e99e2304c9	Write meta WARC with log file	4年前
JustAnotherArchivist	85d78cee13	Add warcinfo record with version information on Python, system, and dependencies	4年前
JustAnotherArchivist	9cff6bd5c1	Only open a WARC file when necessary to avoid producing empty WARCs at the end	4年前
JustAnotherArchivist	8647d6b396	Use f-strings instead of str.format	4年前
JustAnotherArchivist	be5673cfbf	Add record deduplication within a process	5年前
JustAnotherArchivist	e892a6b6a7	Initial commit	5年前

27 コミット (f5c3eb42b3e93c54d6f5312bd3c747617ffb2424)