PySpark without DataProc about text-dedup

[ChenghaoMou]

The Spark version might require some tinkering for your dataset.

In the error message above, the loading code is trying to load parquet files instead of txt files:

File "/home/scheible/git/github/text-dedup/text_dedup/minhash_spark.py", line 403, in <module>
    df: DataFrame = spark.read.option("mergeSchema", "true").parquet(args.input)

You should change this part to load text files properly. For example https://spark.apache.org/docs/latest/sql-data-sources-text.html

What you can do also is to convert your text-file-based dataset to a parque dataset first before using the script.

[scheiblr]

Thanks for the quick answer. The data loading was resolved quickly. However, I struggle now with the saving format. Currently it is saved as parquet files. Is it possible to change that behavior in such a way, that one text file will be the result containing the dedup corpus?

[ChenghaoMou]

that one text file will be the result containing the dedup corpus

It is possible. You can change the save part to something like this:

(
	df # Assume only one column of text
	.coalesce(1) # One file instead of parts. Make sure it fits in memory
	.write
	.format("text")
	.option("header", "false")
	.mode("append")
	.save("output.txt")
)

[scheiblr]

Nice, thank you very much! That lead to a working solution!

God bless you!