選択できるのは25トピックまでです。 トピックは、先頭が英数字で、英数字とダッシュ('-')を使用した35文字以内のものにしてください。
arkiver 1b3f680951 Flush written dictionary. 3年前
.gitignore Fix processing of GZ WARCs. 4年前
README.md Add 'pack' option to pack files. 11年前
megawarc Flush written dictionary. 3年前
megawarc-fix Check for trailing zeroes in gzips. 11年前
ordereddict.py OrderedDict for Python 2.6. 11年前

README.md

Megawarc

megawarc is useful if you have .tar full of .warc.gz files and you really want one big .warc.gz. With megawarc you get your .warc.gz, but you can still restore the original .tar.

The megawarc tool looks for .warc.gz in the .tar file and creates three files, the megawarc:

  • FILE.warc.gz is the concatenated .warc.gz
  • FILE.tar contains any non-warc files from the .tar
  • FILE.json.gz contains metadata

You need the JSON file to reconstruct the original .tar from the .warc.gz and .tar files. The JSON file has the location of every file from the original .tar file.

Metadata format

One line with a JSON object per file in the .tar.

{
   "target": {
     "container": "warc" or "tar", // where is this file?
     "offset": number,             // where in the tar/warc does this file start?
                                   // for files in the tar this includes the tar header, which is
                                   // copied to the tar.
     "size": size                  // where does this file end?
                                   // for files in the tar, this includes the padding to 512 bytes
   },
   "src_offsets": {
     "entry": number,              // where is this file in the original tar?
     "data": number,               // where does the data start? entry+512
     "next_entry": number          // where does the next tar entry start
   },
   "header_fields": {
     ...                           // parsed fields from the tar header
   },
   "header_string": string         // the tar header for this entry
}

Usage

megawarc convert FILE

Converts the tar file (containing .warc.gz files) to a megawarc. It creates FILE.warc.gz, FILE.tar and FILE.json.gz from FILE.

megawarc pack FILE INFILE_1 [[INFILE_2] ...]

Creates a megawarc with basename FILE and recursively adds the given files and directories to it, as if they were in a tar file. It creates FILE.warc.gz, FILE.tar and FILE.json.gz.

megawarc restore FILE

Converts the megawarc back to the original tar. It reads FILE.warc.gz, FILE.tar and FILE.json.gz to make FILE.