Coder Social home page Coder Social logo

Comments (2)

JP95Git avatar JP95Git commented on June 15, 2024
[user@host programs]$ /path/to/pgloader/bin/pgloader --client-min-messages=notice database.load
2024-02-22T10:43:40.001437+01:00 NOTICE Starting pgloader, log system is ready.
2024-02-22T10:43:40.007689+01:00 LOG pgloader version "3.6.af8c3c1"
[...]
2024-02-23T02:34:37.652045+01:00 LOG report summary reset
                        table name     errors       read   imported      bytes      total time       read      write
----------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
                   fetch meta data          0        211        211                     0.494s
                    Create Schemas          0          0          0                     0.001s
                  Create SQL Types          0          0          0                     0.000s
                     Create tables          0        100        100                     0.360s
                    Set Table OIDs          0         50         50                     0.002s
----------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
[...]
----------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
           COPY Threads Completion          0          8          8              14h37m57.750s
                    Create Indexes          0        111        111                4h47m9.918s
            Index Build Completion          0        111        111                 48m26.715s
                   Reset Sequences          0         27         27                     0.870s
                      Primary Keys          0         49         49                     1.044s
               Create Foreign Keys          0         50         50                 24m11.139s
                   Create Triggers          0          0          0                     0.000s
                  Install Comments          0          0          0                     0.000s
                        after load          0         23         23                     0.364s
----------------------------------  ---------  ---------  ---------  ---------  --------------  ---------  ---------
                 Total import time          ✓  4449186294  4449186294   227.9 GB   20h37m47.805s

Output from the virtual server at my customer.

Duration based on the log lines = ~ 15.5 hours
Duration reported by pgLoader = 20.5 hours

from pgloader.

hvanoch avatar hvanoch commented on June 15, 2024

We are having the same issues. On our end it seems that the "Create Indexes" part makes no sense. As an example, this is a summary which only migrates the schema. Real time it took was 14 sec:

            table name     errors       rows      bytes      total time
----------------------  ---------  ---------  ---------  --------------
       fetch meta data          0       3405                     0.293s
        Create Schemas          0          0                     0.437s
      Create SQL Types          0          0                     0.004s
         Create tables          0       1062                     3.530s
        Set Table OIDs          0        531                     0.047s
----------------------  ---------  ---------  ---------  --------------
----------------------  ---------  ---------  ---------  --------------
Index Build Completion          0       1906                     8.285s
        Create Indexes          0       1906                  1m38.173s
       Reset Sequences          0        310                     0.144s
          Primary Keys          0        521                     0.282s
   Create Foreign Keys          0        968                     0.785s
       Create Triggers          0          0                     0.000s
      Install Comments          0        664                     0.274s
            after load          0          2                     0.013s
----------------------  ---------  ---------  ---------  --------------
     Total import time          ✓          0                  1m47.957s

from pgloader.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.