Support for robotsflag field based on OpenWayback behaviour about outbackcdx

[ato]

Sure, would be happy to accept a pull request that implements it.

There's no problem with it. The index format includes a pre-record (per CDX line) version number. So create a new ddcodd

[ato]

Gah. Sorry. 'close issue' is too near the text box on mobile.

... So create a new Capture.decodeValueV2() method for a version 2 record format that supports the robots field and update Capture.encodeValue() to write the new format. Then the index server will happily read both new and old records and you can even mix them in the one index while incrementally reindexing to fill in the robots field data.

It was marked as todo simply because I didn't have any CDX files on hand with that field populated and wasn't sure what the data format was or what exactly it was used for in Wayback.