ENH: Randomised row selection with read_csv() about pandas

[rhshadrach]

I'm not sure if there will be much interest in this feature.

Agreed @Aloqeely - CSV reading in pandas is already very complex and I think we would like to simplify it where possible. This feature seems too niche to me to have it supported directly in pandas.

If I have a large dataset with 100K rows, and I want to select 1000 rows at random, I first need to load the whole dataset into my RAM

How does one determine which 1000 rows to choose without knowing the total number of rows? Wouldn't that require reading the entire file anyways?

[mroeschke]

Thanks for the request but it seems like there's not much interest in this feature so closing

[twoertwein]

I think you can use skiprows for this (I did not test it)

def selectFirstNRandom(size: int):
    selected_count = 0
    def skip(index: int):
        if selected_count >= size:
            return True
        skip_row = random.random() > 0.5
        if not skip_row:
            selected_count += 1
         return skip_row
    return skip

df = pd.read_csv('your_file.txt', skiprows=selectFirstNRandom(100))

[Aloqeely]

I'm not sure if there will be much interest in this feature.

@twoertwein's skiprows solution probably works, but I think using chunksize and iterator is cleaner, it's less random though.

it = pd.read_csv("test.csv", chunksize=1000, iterator=True)

chunk1 = it.get_chunk()  # returns DataFrame with first 1000 lines
chunk2 = it.get_chunk()  # returns DataFrame with the next thousand lines

For more information on the chunksize and iterator parameters, you can see the IO Tools user guide