Time is not inserting with microsecond precision about gorm-bulk-insert

[bombsimon]

I've created #20 which fixes this (and also fixes broken tests). Maybe @t-tiger will merge it if it looks OK. :)

[bombsimon]

Could you elaborate where you get createdAt from? You say you're using Create() which isn't a part of this package but rather gorm itself. It looks like createdAt is a string you're trying to parse but I don't see how it's related to this package or where you got it from. Also, time.RFC3339 doesn't include nanoseconds, time.RFC3339Nano does, see time.Time godoc

If you're using gorm.Model to add your CreatedAt fields they will be of type time.Time which means that you shouldn't parse them after reading them from the database.

Given type MyType

type MyType struct {
    gorm.Model
    Value string
}

Running the auto migrations and inserting values

db.AutoMigrate(&MyType{})

toInsert := []interface{}{
    MyType{Value: "first"},
    MyType{Value: "second"},
    MyType{Value: "third"},
}

gormbulk.BulkInsert(db, toInsert, 3000)

I can fetch the first value and access the time as time.Time

var result MyType

db.First(&result)

fmt.Println(result.CreatedAt.Format(time.RFC3339Nano))

[bombsimon]

Oh, ok!

Yeah, this is becase gorm-bulk-insert sets the CreatedAt and UpdatedAt fields manually (even with if the column has a value) in a loop. This means that they're not all set at the same time.

One could either leave the field as is if it's blank:

diff --git a/bulk_insert.go b/bulk_insert.go
index fb49a8b..777ec9a 100644
--- a/bulk_insert.go
+++ b/bulk_insert.go
@@ -103,7 +104,7 @@ func extractMapValue(value interface{}, excludeColumns []string) (map[string]int

                if !containString(excludeColumns, field.Struct.Name) && field.StructField.Relationship == nil && !hasForeignKey &&
                        !field.IsIgnored && !fieldIsAutoIncrement(field) {
-                       if field.Struct.Name == "CreatedAt" || field.Struct.Name == "UpdatedAt" {
+                       if (field.Struct.Name == "CreatedAt" || field.Struct.Name == "UpdatedAt") && field.IsBlank {
                                attrs[field.DBName] = time.Now()
                        } else if field.StructField.HasDefaultValue && field.IsBlank {
                                // If default value presents and field is empty, assign a default value

Or set a time and re-use for all fields, like

diff --git a/bulk_insert.go b/bulk_insert.go
index fb49a8b..992b1fa 100644
--- a/bulk_insert.go
+++ b/bulk_insert.go
@@ -30,7 +31,9 @@ func insertObjSet(db *gorm.DB, objects []interface{}, excludeColumns ...string)
                return nil
        }

-       firstAttrs, err := extractMapValue(objects[0], excludeColumns)
+       nowTime := time.Now()
+
+       firstAttrs, err := extractMapValue(objects[0], excludeColumns, nowTime)
        if err != nil {
                return err
        }
@@ -49,7 +52,7 @@ func insertObjSet(db *gorm.DB, objects []interface{}, excludeColumns ...string)
        }

        for _, obj := range objects {
-               objAttrs, err := extractMapValue(obj, excludeColumns)
+               objAttrs, err := extractMapValue(obj, excludeColumns, nowTime)
                if err != nil {
                        return err
                }
@@ -85,7 +88,7 @@ func insertObjSet(db *gorm.DB, objects []interface{}, excludeColumns ...string)
 }

 // Obtain columns and values required for insert from interface
-func extractMapValue(value interface{}, excludeColumns []string) (map[string]interface{}, error) {
+func extractMapValue(value interface{}, excludeColumns []string, nowTime time.Time) (map[string]interface{}, error) {
        rv := reflect.ValueOf(value)
        if rv.Kind() == reflect.Ptr {
                rv = rv.Elem()
@@ -104,7 +107,7 @@ func extractMapValue(value interface{}, excludeColumns []string) (map[string]int
                if !containString(excludeColumns, field.Struct.Name) && field.StructField.Relationship == nil && !hasForeignKey &&
                        !field.IsIgnored && !fieldIsAutoIncrement(field) {
                        if field.Struct.Name == "CreatedAt" || field.Struct.Name == "UpdatedAt" {
-                               attrs[field.DBName] = time.Now()
+                               attrs[field.DBName] = nowTime
                        } else if field.StructField.HasDefaultValue && field.IsBlank {
                                // If default value presents and field is empty, assign a default value
                                if val, ok := field.TagSettingsGet("DEFAULT"); ok {

[t-tiger]

I also tried bulk insert with PostgreSQL. If the column type of PostgreSQL is set as timestamp and execute bulk insert, it certainly contains even with microsecond. Sample code is as bellow.

type post struct {
	ID        int
	Body      string
	CreatedAt time.Time
}

var posts []interface{}
for i := 0; i < 10; i++ {
	posts = append(posts, &post{
		Body:      strconv.Itoa(i + 1),
		CreatedAt: time.Now(),
	})
}
if err := gormbulk.BulkInsert(db1, posts, 2000); err != nil {
	panic(err)
}

Not all PostgreSQL patterns have been verified, so it may be a problem in your case. If you can tell me what type you are using, I'll verify it.

[holyxiaoxin]

Sorry for the late response.

My postgres created_at is setup with timestamptz(26,6).

func NewRecord(name string, createdAt string) {
        createdAtTime, err := time.Parse(time.RFC3339, createdAt)
	if err != nil {
		panic(err)
	}
        return Record {
            Name: name,
            CreatedAt: createdAtTime,
        }
}
func insertBulk() {
	var newRecords []interface{}
        createdAtString := res.CreatedAt
        fmt.Println(createdAtString) // 2019-11-25T04:31:07.805929Z
	for _, item := range res.data {
		newRecord := NewRecord(
                         item["name"],
			createdAtString,
		)

		newRecords = append(newRecords, newRecord)
                 // db.Create(&newRecord), this saves the whole array with microsecond precision
	}
        fmt.Printf("%+v", newRecords)
        // [{Name:TestName1 CreatedAt:2019-11-25 04:31:07.805929 +0000 UTC} {Name:TestName2 CreatedAt:2019-11-25 04:31:07.805929 +0000 UTC}]
        err := gormbulk.BulkInsert(db, newRecords, 3000)
	if err != nil {
		panic(err)
	}
}

Somehow it's dropping the microsecond precision from the record and appending it with an incrementing precision.

With native Gorm Create: created_at with multiple inserts keeps microsecond precision:

record1 created_at: 2019-11-21 12:18:12.490279
record2 created_at: 2019-11-21 12:18:12.490279
record3 created_at: 2019-11-21 12:18:12.490279

With Gorm Bulk Insert:

record1 created_at: 2019-11-21 12:18:12.490289
record2 created_at: 2019-11-21 12:18:12.490292
record3 created_at: 2019-11-21 12:18:12.490301

[holyxiaoxin]

Thanks, I'm closing this issue.

[AlwxSin]

@t-tiger Can you add field.IsBlank check, please? In my case, CreatedAt field is populated on the other side, and if I rename it, it will be very confusing.

[bombsimon]

@AlwxSin Usually don't want to promote my own things in others projects but I've created a repository for the same purpose but with a different approach supporting this, see gorm-bulk if you're interested.

[AlwxSin]

@bombsimon thank you

[t-tiger]

@AlwxSin
Thanks to PR #20, it is now (tag v1.1.1) possible to set the time manually, avoiding automatic setting. Thank you for the great efforts of @bombsimon.