untokenize returns all tokens up to a given one about token-utils

[aroberge]

token_utils is designed to be able to 1) recreate exactly the original source from a list of tokens; 2) easily allow the substitution of the string content of one or more tokens by other strings, recreating a transformed source with all the previous spacing between previous tokens as they were in the original source.

Single tokens are objects that carry information including the entire content of the line where they came from. The untokenize function makes use of this content to know where to insert the string content of a given token, thus preserving the spacing between tokens.

For token_utils, untokenizing and printing a single token amounts to simply doing print(token.string).

Alternatively, to untokenize a single token as index 4, like you do, an overcomplicated way of doing it would be:

new_tokens = []
for index, tok in tokens:
    if index == 4:
        new_tokens.append(tok)
    else:
        new_tokens.append('')
print(token_utils.untokenize(new_tokens))

However, instead of using token_utils, Python's own tokenize module would likely be a more appropriate way of doing such an operation.