Check WARC fixity for web AIPs? about general-aip

[amhanson9]

A script that I've since deleted from web-aip worked like this to compare the bag manifest to the WARC MD5 immediately after unzipping.

if os.path.exists("warc_unzip_log.csv"):

# Dataframe with WARC filename and MD5 from the unzipping log.
# Removes ".gz" from the end of the WARC name and removes extra columns so it matches the bag manifest.
log_df = pd.read_csv("warc_unzip_log.csv")
log_df["WARC"] = log_df["WARC"].str.replace(".warc.gz", ".warc", regex=False)
log_df.drop(["AIP", "Zip_MD5", "Fixity_Comparison", "Unzipping_Result"], inplace=True, axis=1)

# Dataframe that combines the WARc rows from the md5 manifests from every bag.
# Removes the path from the WARC filename so it matches what is in the warc unzip log.
bag_df = pd.DataFrame(columns=["Unzip_MD5", "WARC"])
for root, dirs, files in os.walk(f"aips_{date_end}"):
    if "manifest-md5.txt" in files:
        manifest_path = os.path.join(root, "manifest-md5.txt")
        manifest_df = pd.read_csv(manifest_path, names=["Unzip_MD5", "Extra_Space", "WARC"], sep=" ")
        manifest_df.drop(["Extra_Space"], inplace=True, axis=1)
        bag_df = pd.concat([bag_df, manifest_df], ignore_index=True)
bag_df = bag_df[bag_df["WARC"].str.endswith(".warc")]
bag_df["WARC"] = bag_df["WARC"].str.replace("data/objects/", "")

# Compares the two dataframes. If they don't match, saves an error log.
# It will be moved into the AIPs directory in the next step
df = log_df.merge(bag_df, indicator=True, how="outer")
if len(df[df["_merge"] != "both"]) > 0:
    df.to_csv(f"warc_md5_differences.csv", index=False)
else:
    print("All WARC fixity was unchanged")