소스 검색

Proper script for tracking size of uploaded data

master
JustAnotherArchivist 5 년 전
부모
커밋
321067819c
1개의 변경된 파일14개의 추가작업 그리고 1개의 파일을 삭제
  1. +14
    -1
      ia-upload-progress

+ 14
- 1
ia-upload-progress 파일 보기

@@ -1,3 +1,16 @@
#!/bin/bash
# Check how much of an upload made it into the item yet (vs. how much is stuck in the S3 queue if you also watch the upload process)
echo "Uploaded $(ia metadata $identifier | grep -Po '("source"\s*:\s*"original",[^}]*"size"\s*:\s*"\K\d+|"size"\s*:\s*"\K\d+(?="\s*,[^}]*"source"\s*:\s*"original"))' | awk '{sum+=$1} END {printf "%.2f GiB\n",sum/1024/1024/1024}') of $(du -bc *.warc.gz | tail -1 | cut -f1 | awk '{printf "%.2f GiB", $1/1024/1024/1024}')"
# Usage: ia-upload-progress IA_IDENTIFIER [FILES]
# If no files are specified, all WARCs in the current directory are used.
identifier=$1
shift
files=("$@")
if [[ ${#files[@]} -eq 0 ]]
then
files=(*.warc.gz)
fi
scriptpath="$(cd "$(dirname "$0")"; pwd -P)"

uploaded=$(ia metadata "${identifier}" | grep -Po '("source"\s*:\s*"original",[^}]*"size"\s*:\s*"\K\d+|"size"\s*:\s*"\K\d+(?="\s*,[^}]*"source"\s*:\s*"original"))' | awk '{sum+=$1} END {printf "%.0f\n", sum}')
localSize=$(du -bc "${files[@]}" | tail -1 | cut -f1)
echo "Uploaded $("${scriptpath}/format-size" <<<"${uploaded}") of $("${scriptpath}/format-size" <<<"${localSize}") ($(awk '{ printf "%.1f", 100 * ($1 / $2); }' <<<"${uploaded} ${localSize}") %)"

불러오는 중...
취소
저장