Incoherent Result Labeling between GPTZero.sg and Github Code about detectgpt

[slgphantom]

I tried the same text on GPTZero.sg with another machine, and the result aligns this time...

Any how, let me also share the result of using v1 (GPTZero) of the GPT2PPL model of the same text here:

Perplexity 10
Perplexity per line 14.0
Burstiness 20
(OrderedDict([('Perplexity', 10),
              ('Perplexity per line', 14.0),
              ('Burstiness', 20),
              ('label', 0)]),
 'The Text is generated by AI.')

[BurhanUlTayyab]

The results are same on every end. I think the fluctuation might occur because of different Pytorch / Transformer versions.

For GPTZerov1, you can watch this video to understand how it works: https://www.youtube.com/watch?v=x9H-aY5sCDA (we've added one more feature where perplexity per line is also calculated and aggregated).

For DetectGPT: We use Log Prob function from a LLM to detect text are similar to what it can produce. Also done line by line.

[BurhanUlTayyab]

Additionally, it would be helpful if the threshold could be set as an input argument of the model so that users can customize it.
Sure, we can add that as an input parameter for our next release.

[slgphantom]

Thank you for the prompt response!

I noticed that the results are fluctuating. It might be helpful if you could test it on your device as well.

Could you please provide more information about the relationship between the confidence level, threshold, and labeling? As someone who is not well-versed in the internal calculations, I believe that the text is only labeled as "human" when the confidence level is above the threshold (0.7). However, this does not seem to be the case. It is frustrating to see that other methods are still reporting the text as being written by AI, such as GPTZero in v1 and DetectGPT on both your and the OG's website.

It would also be highly beneficial if the module could return an object that includes the confidence, label, and highlighted text (AI generated) so that users can better interpret the results and further apply them in their work.