Warnings on test/scripts/test.sh about mobilitydb

[cvvergara]

Will start working on this PR once #141 is been merged, otherwise I can not verify the progress of the work done

[cvvergara]

I think I found why sometimes the tests take way tooooooo long

[cvvergara]

@estebanzimanyi
I am trying to start the server manually

PGCTL="${BIN_DIR}/pg_ctl -w -D ${DBDIR} -l ${WORKDIR}/log/postgres.log -o -k -o ${WORKDIR}/lock -o -h -o ''"

For me:

DBDIR= /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db
BIN_DIR= /usr/lib/postgresql/12/bin
WORKDIR= /home/vicky/mobilitydb/mobilitydb/cvvergara/build

substituting

/usr/lib/postgresql/12/bin/pg_ctl -w -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db -l /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/log/postgres.log -o -k -o /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/lock -o -h -o '' start

I get

/usr/lib/postgresql/12/bin/postgres: option requires an argument -- 'h'
Try "postgres --help" for more information.

what I can do manually

/usr/lib/postgresql/12/bin/pg_ctl -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db status

and

/usr/lib/postgresql/12/bin/pg_ctl -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db stop

I am stuck

shellcheck test/scripts/test.sh 

In test/scripts/test.sh line 23:
PGCTL="${BIN_DIR}/pg_ctl -w -D ${DBDIR} -l ${WORKDIR}/log/postgres.log -o -k -o ${WORKDIR}/lock -o -h -o ''"
                                                                                          ^---------------^ SC2089: Quotes/backslashes will be treated literally. Use an array.


In test/scripts/test.sh line 47:
  if $PGCTL start 2>&1 | tee "$WORKDIR"/log/pg_start.log; then
     ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 62:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 95:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 128:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.

If you can tell me how to you manage to start the server manually it would be of great help

[estebanzimanyi]

After running make test

[...]
111/111 Test #111: 97_tnpoint_tempspatialrels_tbl ....   Passed    1.55 sec

100% tests passed, 0 tests failed out of 111

Total Test time (real) = 466.62 sec

I am able to do as follows directly on bash

esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ BIN_DIR=/usr/local/pgsql/13/bin
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ DBDIR=/home/esteban/src/PR/MobilityDB/build/tmptest/db
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ WORKDIR=/home/esteban/src/PR/MobilityDB/build/tmptest
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ PGCTL="${BIN_DIR}/pg_ctl -w -D ${DBDIR} -l ${WORKDIR}/log/postgres.log -o -k -o ${WORKDIR}/lock -o -h -o ''"
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ echo $BIN_DIR
/usr/local/pgsql/13/bin
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ echo $DBDIR
/home/esteban/src/PR/MobilityDB/build/tmptest/db
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ echo $WORKDIR
/home/esteban/src/PR/MobilityDB/build/tmptest
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ echo $PGCTL
/usr/local/pgsql/13/bin/pg_ctl -w -D /home/esteban/src/PR/MobilityDB/build/tmptest/db -l /home/esteban/src/PR/MobilityDB/build/tmptest/log/postgres.log -o -k -o /home/esteban/src/PR/MobilityDB/build/tmptest/lock -o -h -o ''
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ $PGCTL status
pg_ctl: server is running (PID: 3993)
/usr/local/pgsql/13/bin/postgres "-D" "/home/esteban/src/PR/MobilityDB/build/tmptest/db" "-k" "/home/esteban/src/PR/MobilityDB/build/tmptest/lock" "-h" ""
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ $PGCTL stop
waiting for server to shut down.... done
server stopped
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ $PGCTL start
waiting for server to start.... done
server started
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$

[cvvergara]

$PGCTL start

Can you start that server without the script like with a command like this one:

/usr/lib/postgresql/12/bin/pg_ctl -w -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db -l /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/log/postgres.log -o -k -o /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/lock -o -h -o '' start

wait untill the make test finishes
then do something similar to this:

/usr/lib/postgresql/12/bin/pg_ctl -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db status

you will see that the server is running
then do something similar to this:

/usr/lib/postgresql/12/bin/pg_ctl -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db stop
/usr/lib/postgresql/12/bin/pg_ctl -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db status

you will see that the server has been stopped
then, what I need is to get the exact command to start the server, I have this that is the expansion of '$PGCTL`:

/usr/lib/postgresql/12/bin/pg_ctl -w -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db -l /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/log/postgres.log -o -k -o /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/lock -o -h -o '' start

But its working, its not starting that server
I want you to see if you can start it with a command.

[cvvergara]

I need the exact command to have remove that last warning
as you can see here, lots of warnings were removed by creating a function to start and to stop the server. (and of course by reading documentation to have the minimum needed)
Process is:

1. figure out how to get the status manually
2. remove the unnecessary flags
3. create the function & use it (lots of warning are gone)
4. figure out how to stop the server manually
5. remove the unnecessary flags
6. create the function & use it (lots of warning are gone)
7. figure out how to start the server manually
8. remove the unnecessary flags
9. create the function & use it
   I am stuck on step 7.
   I've never started a server using pg_ctl, which looks more complicated.
   I cant get it to start :-( so I havent reach to step 9, se there are still this warnings

shellcheck test/scripts/test.sh 

In test/scripts/test.sh line 23:
PGCTL="${BIN_DIR}/pg_ctl -w -D ${DBDIR} -l ${WORKDIR}/log/postgres.log -o -k -o ${WORKDIR}/lock -o -h -o ''"
                                                                                          ^---------------^ SC2089: Quotes/backslashes will be treated literally. Use an array.


In test/scripts/test.sh line 47:
  if $PGCTL start 2>&1 | tee "$WORKDIR"/log/pg_start.log; then
     ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 62:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 95:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.


In test/scripts/test.sh line 128:
    $PGCTL start
    ^----^ SC2090: Quotes/backslashes in this variable will not be respected.




https://github.com/MobilityDB/MobilityDB/pull/146/files#diff-e9a9aac610a169810c0c02074ed62145c1c30bfc459520f6a535417b84fa41eeR15

[estebanzimanyi]

The first step should be initdb, which is done when do e.g., when make test.
After that it is possible to do the commands as follows

100% tests passed, 0 tests failed out of 111

Total Test time (real) = 557.83 sec
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ /usr/local/pgsql/13/bin/pg_ctl -w -D /home/esteban/src/PR/MobilityDB/build/tmptest/db -l /home/esteban/src/PR/MobilityDB/build/tmptest/log/postgres.log -o -k -o /home/esteban/src/PR/MobilityDB/build/tmptest/lock -o -h -o '' status
pg_ctl: server is running (PID: 8907)
/usr/local/pgsql/13/bin/postgres "-D" "/home/esteban/src/PR/MobilityDB/build/tmptest/db" "-k" "/home/esteban/src/PR/MobilityDB/build/tmptest/lock" "-h" ""
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ /usr/local/pgsql/13/bin/pg_ctl -w -D /home/esteban/src/PR/MobilityDB/build/tmptest/db -l /home/esteban/src/PR/MobilityDB/build/tmptest/log/postgres.log -o -k -o /home/esteban/src/PR/MobilityDB/build/tmptest/lock -o -h -o '' stop
waiting for server to shut down.... done
server stopped
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$ /usr/local/pgsql/13/bin/pg_ctl -w -D /home/esteban/src/PR/MobilityDB/build/tmptest/db -l /home/esteban/src/PR/MobilityDB/build/tmptest/log/postgres.log -o -k -o /home/esteban/src/PR/MobilityDB/build/tmptest/lock -o -h -o '' start
waiting for server to start.... stopped waiting
pg_ctl: could not start server
Examine the log output.
esteban@DESKTOP-9FBND60:~/src/PR/MobilityDB/build$

I am investigating why the last one is not working

[estebanzimanyi]

However, as already stated before (see PR #129) a function may not be a solution, because it does not allow to put the output of the process in the log files.

[estebanzimanyi]

Original declaration

PGCTL="${BIN_DIR}/pg_ctl -w -D $DBDIR -l $WORKDIR/log/postgres.log -o -k -o $WORKDIR/lock -o -h -o ''"

Streamlining with information from https://www.postgresql.org/docs/13/app-pg-ctl.html

pg_ctl status [-D datadir]

/usr/local/pgsql/13/bin/pg_ctl -D /home/esteban/src/PR/MobilityDB/build/tmptest/db status

pg_ctl stop [-D datadir] [-m s[mart] | f[ast] | i[mmediate] ] [-W] [-t seconds] [-s]

/usr/local/pgsql/13/bin/pg_ctl -D /home/esteban/src/PR/MobilityDB/build/tmptest/db stop

pg_ctl start [-D datadir] [-l filename] [-W] [-t seconds] [-s] [-o options] [-p path] [-c]

/usr/local/pgsql/13/bin/pg_ctl -w -D /home/esteban/src/PR/MobilityDB/build/tmptest/db -l /home/esteban/src/PR/MobilityDB/build/tmptest/log/postgres.log start

I have tested on my machine after the database for test is initialized (e.g., after make test) and they work.

[cvvergara]

After running:

usr/lib/postgresql/12/bin/pg_ctl -w -D /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db -l /home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/log/postgres.log start

I am getting

2021-08-10 13:49:57.810 CDT [24228] LOG:  starting PostgreSQL 12.7 (Ubuntu 12.7-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2021-08-10 13:49:57.842 CDT [24228] LOG:  could not bind IPv4 address "127.0.0.1": Address already in use
2021-08-10 13:49:57.842 CDT [24228] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2021-08-10 13:49:57.842 CDT [24228] WARNING:  could not create listen socket for "localhost"
2021-08-10 13:49:57.842 CDT [24228] FATAL:  could not create any TCP/IP sockets
2021-08-10 13:49:57.843 CDT [24228] LOG:  database system is shut down

[cvvergara]

However, as already stated before (see PR #129) a function may not be a solution, because it does not allow to put the output of the process in the log files.

I am getting this on the pg_start.log:

more tmptest/log/pg_start.log 
waiting for server to start.... done
server started
pg_ctl: server is running (PID: 25347)
/usr/lib/postgresql/12/bin/postgres "-D" "/home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/db" "-k" "/home/vicky/mobilitydb/mobilitydb/cvvergara/build/tmptest/lock" "-h" ""

when I am using a function: https://github.com/cvvergara/MobilityDB/blob/issue-144/test/scripts/test.sh#L50
it is showing the status

On that PR it didn't work, as the $PGCTL I converted to a function and used also for start and status and stop
But then again I could not determine that there where other things wrong that we catched the other day.

Probably for this current point in time, for the $PGCTL used for starting the server, just leave it like that, and change the test_shell a little to ignore those warnings on that file in particular.