raveendratal / ravi_azureadbadf Goto Github PK

Contributors

Stargazers

Forkers

hercules261188 nani122211 akshay777asi depangi manikantaraogopalam parvatisharma rohithsingam boinapalli prakashbaaskar baaskarprakash talha915 faizateeq dattatreyarapelli tlnreddy1 aishwarya-bhatt girijadas9 hiteshgupta457 rajasekar-ai mkohli27 maimoonaiqbal2000 premgopinath pushpavallin nagabhushanam07 royalt1234 harshalrasal udaymewada satyahadev stanleycruvinel starofazurecloud tillu1997 harivasireddy1 sarikayamehmet jeevanu94 sreenivasa-xor nnaveensingh95 venkatprasadk yaseenmohammad spukkala shankartalluri chella107 rajashree12bagal nagaraju100 madhu-k-sr geetasuhas2012 abinash15th venkataramanr07 jash625 leomahesh veera4103 chinmoydash se008 sangam-ux jaibarathibalakrishnan ramyavin vaquarkhan blpraveen eshaik2 dravindranath praveen1590 venkatranga45 bijuthottathil swapnabalaji bosedib ishank296 luanpedro meruguvikram nimit572 anupam0004 eclases prashanthgowda121 sureshdevaraa sonusagarde ravirex sahithineelam-sn shekpaul narvadeshtripathi shantanu-bh bprakash97 kirtisri04 qinggfengg satishmuppala1989 saepratthap sachinkasbe111 vsatish452 sudharosh surajkaruvel prateek0009 soumyakanta-garabadu madhusamal himanshuwarkad sujeetkumarmehta arjunrajaazure renu967 swiz0617 priti835 9-tails-fox harikalyani25 shlh pankajdhome venket-abhishek

def master_array(df):
    array_cols = [c[0] for c in df.dtypes if c[1][:5]=="array"]
    while len(array_cols)>0:
        for c in array_cols:
            df = df.withColumn(c,explode_outer(c))
        df = child_struct(df)
        array_cols = [c[0] for c in df.dtypes if c[1][:5]=="array"]
    return df

def master_array(df):

    find_nested_data = True

    while find_nested_data:

        array_cols = [c[0] for c in df.dtypes if (c[1])[:5] == 'array']

        for c in array_cols:
            df = df.withColumn(c, explode_outer(c))

        df = child_struct(df)
        array_cols = [c[0] for c in df.dtypes if (c[1])[:5] == 'array']

        if len(array_cols) == 0:
            find_nested_data = False

    return df