Cache mechanism is not working related to metadata.json in s3 about spark-nlp

[mehmetbutgul]

Hi @maziyarpanahi,
Thanks for your comments.
After your comments, I made a PR for the issue.
I implemented your last idea.
That said, the very best solution is to check metadata.json, if it was updated, we MUST download it. If the file hasn't changed, we shall skip it in that session/application cycle.
I agree with you. This idea seems to be the best solution.
PR --> #14224

[maziyarpanahi]

Many thanks @mehmetbutgul - I left it to Danilo to review it, I make sure to include it in the tomorrow's release. Thanks agian for your contribution. 🚀

[maziyarpanahi]

Do you have any PR to suggest some caching mechanism? It has to be:

• time base (expires after a duration)
• session base (if the session dies and we are in a new session, we MUST download a new metadata.json)
• there must be a force_download Boolean for users to override, either enable caching or disable it to be in the default behavior now. (only the code suggests we cache the metadata.json, we have never mentioned this in our docs or anywhere else. As far as anybody knows, if we make a change in metadata.json you'll see it immediately!)

That said, the very best solution is to check metadata.json, if it was updated, we MUST download it. If the file hasn't changed, we shall skip it in that session/application cycle.