Garbage Collector List destructor about cparse

[VinGarcia]

Uhmm, this seems problematic 🤔

Can you give me a sample code that reproduces the problem? In the meanwhile, I will read the code for the list destructor to see if I find any flaws in it.

Thanks for the feedback.

i can't really i had to change my approach for a more realistic one regarding the data i handle (>50Gb). Maybe the issue was coming from something else.

[VinGarcia]

I see, sorry I didn't have time to investigate it more closely last week. I hope this new approach works alright =].

[VinGarcia]

Also grazba, there is a concept that might be useful for you. Have you heard of generators?

This concept exists in many programming languages and it is a little hard to grasp at first but is very useful. One of the scenarios where it is helpful is when you are iterating over a really big array.

A short description of the concept is a function that return one value on each iteration of a for loop and the programming language hides this from the final user, e.g. in Python 3.6:

# A very simple example on how to implement a custom generator in Python:
def myRange():
    for i in ...:
         // Load some big data from somewhere with a complex logic...
          yield loaded_data;

for (data in myRange()):
    // work with the yielded data one at a time.

In other words, a generator is basically an iterator that gets each data right before the iteration with a custom logic. I have implemented a hard-coded generator in JSpy for the range() function by extending the iterable class from cparse and implement its behavior using some instance variables to keep track of the state of the generator.

I have been doing some tests, and the issue is time i tested again with some big data (smaller than the other one) and it takes forever to clear the list calling the packToken destructor for each element, it would be great if i could just drop the memory... so the problem is really about the data type being packToken the compiler can't clear it in O(1) and has to call the destructor even if my data is float/double.

For now i will consider doing something else for handling data this big , but maybe it would be interesting to add a way to store native types (or maybe there is and i'm not aware of it) for performance.

[VinGarcia]

Uhmm, the pack Token is a dynamic pointer that can be any type, thus it can't be passed by value (although i simulate it by cloning each Token) and stored in an array as you wanted.

However a simple solution would be to implement a new type called "double_array" since and then implement operators for it such as the index operator [] and some type specific functions like .len()

This would be the best approach I think and would be lighter to load, and delete.

[VinGarcia]

Just to be more clear, the list I am suggesting could be implemented internally as an std::vector, return only packTokens of type real accept only packTokens that are convertible to the double and throw an error otherwise.

yeah i totally got your point, i just wanted to explain the issue i was having regarding performance so that someone else i the same situation could benefit from the info. i'm using temporary files to handle my data as it is the only reliable way to deal with big data , the TokenList works fine (but yeah it could be faster if i implement the list) for smaller data.

thanks again for everything man 👍

[VinGarcia]

Thank you for all the support =], should I close this issue?