aws wrangler - s3.to_csv stripes 00:00:00 in timestamps about aws-sdk-pandas

[jaidisido]

Can you share a dummy code example of this behaviour? The below is working fine for me:

import awswrangler as wr
import pandas as pd

df = pd.DataFrame({
    "datetime": pd.date_range("2018-01-01", periods=2, freq="h"),
    "str": ["a", "b"]
})
print(df)
print(df.dtypes)

df.to_csv("./test_2769.csv", index=False)

path = "s3://bucket/test_2769.py"
wr.s3.to_csv(df, path, index=False)
wr.s3.download(path, "./test_s3_2769.csv")

             datetime str
0 2018-01-01 00:00:00   a
1 2018-01-01 01:00:00   b
datetime    datetime64[ns]
str                 object
dtype: object

The downloaded csv file does contain the 00:00:00

datetime,str
2018-01-01 00:00:00,a
2018-01-01 01:00:00,b

[kennethng]

Hi - thank you for your help on this!
This is within a lambda function:

        cur.execute("select * from test")
        offer_reports = cur.fetchall()
        data = pd.DataFrame(test)
        if not data.empty:
            wr.s3.to_csv(data, f's3://cdc-target/test.csv',index=False)

I tried using df.to_csv() but getting an write error within the lambda function when trying to use that function.

[jaidisido]

Your code feeds a test variable to pd.DataFrame but it is referenced nowhere else so hard to infer anything from it. In any case, here are two suggestions:

1. Ensure that the column is indeed of type datetime64
2. pandas to_csv has a date_format argument that you can leverage to enforce how datetime objects are formatted (e.g. '%d/%m/%Y %H:%M:%S')

[kennethng]

Hi @jaidisido - apologies, I was trying to remove any syntax from my code, my code actually is the following:

        cur.execute("select * from reports")
        reports = cur.fetchall()
        data = pd.DataFrame(reports)
        if not data.empty:
             wr.s3.to_csv(data, f's3://cdc/reports.csv',index=False)
       

with this code, for some reason it is stripping out 00:00:00, if i add 1 second to it, then I have no issues exporting out datetime.

I tried using to_csv in the lambda function but running into permission issues and haven't been able to resolve the permission issues yet.