Improve createDataFrame annotations about pyspark-stubs

[zero323]

Partially addressed by #119

[redlickigrzegorz]

@zero323 According to the documentation, it is possible to create a DataFrame with the usage of the proper schema and the data which are a list e.g. List[tuple].
I have prepared the example of the code which is not proper against mypy but it is proper against pyspark and it works properly, at least in my opinion 😅

data = [('Alice', 1)]
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True)
])

# a properly created DataFrame, unfortunately with lost "IntegerType"
spark.createDataFrame(data, ['name', 'age'])

# a properly created DataFrame unfortunately `mypy` is telling about incompatible used type
spark.createDataFrame(data, schema)

The exact error produced by mypy is as followed:

error: Argument 1 to "createDataFrame" of "SparkSession" has incompatible type "List[Tuple[str, int]]"; expected "Union[RDD[Union[datetime, date, bool, int, float, str, Decimal]], List[Union[datetime, date, bool, int, float, str, Decimal]]]"

Could you tell me if your issue is addressing this thing too?
In the project where we are using your library, we are creating the DataFrame manually mostly for the unit tests. It is done this way because it is easier to have the test data as a simple Python list.

I can also prepare the pull request to resolve my specific problem but I know that it can be not enough for everything that you have in your mind. That's why I would like to consult it before 😉

The versions which I have used for this test:

• Python 3.5.1
• pyspark==2.4.3
• pyspark-stubs==2.4.0.post5

[zero323]

@redlickigrzegorz That's something I'd consider bug, rather than target of this particular enhancement proposal.

There are at least two problems here - the overload that has been partially matched is to wide, and there is no overload that is intended to match this case.

If you want to work on that I'd suggest three things:

• Narrowing down the overload that has been matched here by changing schema to AtomicType. We wan't to see something around

  error: No overload variant of "createDataFrame" of "SparkSession" matches argument types "List[Tuple[str, int]]", "StructType"
  ...
  

  with your failing case.
• Adding another overload that targets this specific case ((Union[RDD[Union[List,Tuple]], Iterable[Union[List,Tuple]]], StructType))
• Adding new data driven test case that confirms that all definitions work (that's probably the most consuming part here).

[zero323]

Resolved by #268, #342