Compiler front-end not handling prompted for submit file input about hi-tech-z80-c

[tsupplis]

Is it stock CP/M 3?

[agn453]

While I build things using an emulator on my mac mini or Raspberry Pi (zxcc, RunCPM and under SIMH AltairZ80), I verify and test on real hardware too.

Mainly I use a Z280 running banked-memory CP/M 3 with Simeon Cran's ZPM3 enhanced BDOS replacement (The hardware is Bill Shen's Z280RC in RC2014 format). This is where I noticed this issue with submit file input.

I'll switch to Digital Research's CP/M 3's stock standard BNKBDOS3 and RESBDOS3 (Y2K versions) and report back.

[tsupplis]

So the problem I have is with zxcc. SIMH works perfectly ...
I have no real hardware with CP/M-3 though. (SC-131 and RC2014 with CP/M 2.2 only).
Are you using this?
https://www.retrobrewcomputers.org/doku.php?id=builderpages:plasmo:z280rc

[tsupplis]

I confirm same problem on simh cp/m-3 ...

[agn453]

Yes that's the Z280RC. I wrote the banked memory CP/M 3 BIOS for it.

Swapping to DRI's BDOS show's the same symptoms.

I'll now install a copy of HI-TECH C on my Z80 MBC2 (which will be slow) and test it there. If needs be I can also fire up my almost 40 year old S-100 bus system - but it only has 8-inch floppies which may not be working (last fired up about 14 years ago to make disk images of all my old floppies!).

[agn453]

It works properly under CP/M 2.2 on a real Z80

A>type test.sub
; CP/M 2.2 uses XSUB for program input
xsub
c309-6
-o -v test.c

A>submit test.sub
                                                                                
A>XSUB                                                                        
                                                                                
A>C309-6                                                                        
Hi-Tech C Compiler (CP/M-80) V3.09-6                                            
Copyright (C) 1984-87 HI-TECH SOFTWARE                                          
c> -O -V TEST.C                                                               
                                                                                
0:A:CPP -DCPM -DHI_TECH_C -Dz80 -I0:A: TEST.C $CTMP1.$$$                      
0:A:P1 $CTMP1.$$$ $CTMP2.$$$ $CTMP3.$$$                                         
0:A:CGEN $CTMP2.$$$ $CTMP1.$$$                                                  
0:A:OPTIM $CTMP1.$$$ $CTMP2.$$$                                                 
0:A:ZAS -J -N -oTEST.OBJ $CTMP2.$$$                                           
ERA $CTMP1.$$$                                                                  
ERA $CTMP2.$$$                                                                  
ERA $CTMP3.$$$                                                                  
0:A:LINQ -Z -Ptext=0,data,bss -C100H -OTEST.COM 0:A:CRTCPM.OBJ TEST.OBJ 0:AB
ERA TEST.OBJ                                                                  
ERA $$EXEC.$$$                                                                  
                                                                                
(xsub active)                                                                   
A>

(sorry iI posted this to the wrong issue before)

[tsupplis]

So quick look at it is the gets call in getargs after the prompt does not get anything ... it looks like the initialisation of the input being the console does not take into account input set by submit... (function nextch line 206).
FYI I also tried to GENCOM C.COM with PIPEMGR. To no avail. It does not work either ...

Will carry on investigating this week end if you want.

I also validated that a normal program generated with the same libc as c.com works properly ...

#include <stdio.h>

int main() {
   char buf[246];
   gets(buf);
   printf("(%s)\n",buf);
   return 0;
}

J>type test.sub
rc
< -v hello.c
<

J>test

J>rc
 -v hello.c
( -v hello.c)

J>

So the scope of the issue is the front end of getargs ...

[agn453]

The fact that the sign-on banner (written to stderr) loses the same number of characters that should be input to the c> prompt is suspicious. I've confirmed that increasing the command line reduces the banner by the same number of characters.

I'm going to have lunch then look deeper...

[tsupplis]

this is happening with all programs compiled with c.com contrary to what I thought. the more is written to stderr, the more is not taken into account in the input.

[tsupplis]

Ok progress...
Reversed write.c (plus U_RSX in case) and it works.
what creates the issue is at line 90:

            if (fc->fsize < fc->rwp) 
                fc->fsize = fc->rwp;

Removing this fsize adjustment resolves the problem
Looking at the history and reason for the fix is part of the pipemgr fix ...

Further digging needs to be I suspect ... I think further testing needs to be done . Interesting though is the fact that I did not have these two line in my pipemgr integration.

I will also check whether it resolve my problem with the size adjustment...

Interestingly ... the last problem I had (Issue #6) that was preventing c to work under cc is also gone. Everything works (i.e. the changes to stat, open, close I could not take ...). It is what I perceived as a bug in the cpm3 emulation of xxcc ....

the following changes were causing grief and removing the lines above makes it work. The changes were

in open.c commenting section around line 30:

 /*      if (d)               // Byte count represents UNUSED bytes in last
             d = 0x80 - d;    // sector ... Changed 9 Apr 2014 jrs */

in stat.c commenting section around line 57:

/*
            Changed to interpret last record byte count as the number of
            UNUSED bytes.  Slightly simpler, historically correct and
            consistent with other tools that I use.  [jrs 2014-04-08]

            if (d)
                d=0x80-d;
*/

in close.c around line 16 changing

    fc->nr = (fc->fsize & 0x7f);  /* Set exact file size */

by

    fc->nr = (-fc->fsize & 0x7f);  /* Set exact file size */

So if I remove those 2 lines in write.c, I am now able to run with exactly the same version as you, i.e. zxcc-htc and your project at 100% in line and only differ by cosmetics...

Can you double check that on your you zxcc env?

2 birds with one stone I hope.

[agn453]

Sorry, I had to attend to family commitments yesterday - so I didn't get to look into this further until now.

The changes for the exact file sizes under CP/M 3 were done by Jon Saxton (jrs) and are explained in my README.md - simplifying file size calculations with backwards compatibility to CP/M 2.2. These then needed to be back-ported into John Elliot's original PIPEMGR (and the Van Nuys utilities).

The code you found in the write() routine is suspicious - over-writing the read-write pointer. I began by looking for a missing call to fflush() which should be used when mixing reads and writes to the same stream.

[tsupplis]

Yes, Actually now the changes of Jon Saxton are working in zxcc perfectly once I remove those 2 lines in write.c. did not before which is the problem I was experiencing with the c front end.
It also fixes our submit input problem.
I tried extra flushes to no avail before reverting write and finding the suspicious two lines.

[agn453]

I'm just confirming the change to write() works on CP/M 3 before pushing it. Thanks for spotting this before I did!

[tsupplis]

Super cool!!! I am finally fully aligned with your code base 😉
I just need to do clean up to be verbatim the same.
PS: I created two small issues 🙁 sorry for not PR'ing but I am struggling with the text attributes in .gitattributes. cloning modifying everything.

[agn453]

I saw your .gitattributes issue. I use a mac and found it easier to just keep the sources with CR-LF line endings (which also is the way SIMH AltairZ80 exports files with the W command - and the reason I kept the ugly upper-case file names). Judicious use of dos2unix/unix2dos and the file commands fixes things - and I think there's a git setting to do this too.

[agn453]

I'm just confirming the change to write() works on CP/M 3 before pushing it. Thanks for spotting this before I did!

Sorry I was called away.

This change does not fix this issue.

Looking more closely at the code, there's two long words in the file control block for keeping the read write pointer (fc->rwp) and the file length (fc->fsize) in bytes. All this code in the write() routine is doing is updating the length of the file in bytes when the file is expanded.

[tsupplis]

so I can confirm that the 2 extra lines compared to the HTC original

• make submit input work in my test
• fix the usage of Saxton changes in a zxcc environment.

Do you have any idea/suggestion to pursue the testing and see side effects of removing them? I am a bit blind on this one as all my tests work.

[agn453]

I only see this issue under CP/M 3 on my Z280RC and under SIMH AltairZ80. I'll go back and look at diffs and try doing a paper walk-through the I/O in getargs().

[tsupplis]

Ok the problem reappeared for me too on SIMH. So I unfortunately lost the fix that was working at some stage.
The 2 lines removed from write are fixing for. sure the compatibility problem with zxcc but I am like you back to square one on submit input.
the symptom is for each character (including the banner) output to CON, we loose one submit input character I wonder if my success was just a red hairing as a result of using TED which has weird behaviour with spaces.
In any case, it is not fixed.

[tsupplis]

I am so frustrated to having succeeded in making it work yesterday, I went through the same motion again but with a couple of tests I kept ...

Added here a little litmus test.

A simple submit:

t.com
<-v    hello.c
<
t.com l
<a b c

and a simple program

The output on sim with CP/M-3

J>t.sub

J>t.com
epvr e f i xh
 ltleos.tc>

0; (test)
exiting ...

J>t.com l
 p rbe fci xd
 e f
(e f
)
exiting ...

J>

So

1. both tests fail: the problem is not in getargs specifically, using fgets to get the input from submit broken too. that removes the main guilt of getargs probably
2. notice the prefix that 'gobbles up' the number of inputs from submit

For summary, the variant with fgets is a simple as that:

            char * b;
            b=fgets(buf,256,stdin); 
            b[256]=0;
            fprintf(stderr, "(%s)\n",buf);

The problem seems to be in the management of the inputs/outputs (read, write, fcb) ...

I don't know enough about the sharing of FCBs and Context, but it looks like the context/FCB of stdin and stdout are shared which does not make sense to me ...

t.zip

Follow up:

1. The underlying action is completely implemented by fgetc in fgetc.as, looking at that to get clued in further. the only interaction it has with the channel is through filebuf.c which has not changed. in term calls read. the two functions manipulate only the stdin structure as expected so work on different FILE descriptors.
2. I again as I did yesterday reverse the write.c and read.c to their originals and here is the results:

J>t.sub

J>t.com
-v    hello.c

J>WARNING: PROGRAM INPUT IGNORED
t.com l
a b c d e f

It work perfectly....

3. I tried the two independently. write.c is the one making the difference.
4. I align the style to get the minimum diff:

3,4d2
< extern char _piped;    /* PIPEMGR loaded? */
< 
14d11
<     char    RSXPB[4];
21,23d17
<     RSXPB[0]=0x7D;
<     RSXPB[1]=0;
<     RSXPB[3]=0;
43,54d36
<     case U_ERR:
<          RSXPB[0]=0x7A;
<     case U_RSX:
<         while(nbytes--) {
<              _sigchk();
<              RSXPB[2]= *buf++;
<              if (_piped) 
<                 bdos(CPMRSX,RSXPB);
<              else        
<                 bdos(CPMWCON,RSXPB[2]);
<         }
<         return count;    
84,85d65
< /*            if (fc->fsize < fc->rwp) 
<                 fc->fsize = fc->rwp;*/

(ignore the 2 last lines, they are an issue with zxcc working or not)

5. The only big difference between the two is U_ERR but in the previous release, U_ERR was not used, U_CON was instead ... so it works because it goes simply cuts away the U_ERR content .... 🙁
6. I redo the test with stdout instead of stderr ... failure. So write.c is not the guilty guy ... or not alone at lease ..., searching for other readers of CON: .... so here is the list:

getch.c:	return bdos(CPMRCON) & 0xFF;
read.c:        bdos(CPMRCOB, buffer);
signal.c:	c = bdos(CPMRCON);

• getch.c is out of the loop
• read.c has no surprise
• signal.c ..... _sigchk is called everywhere in read and ... in write ....

7. so I changed quickly a sigchk into a nop function and it works !!!!!!!
8. let's check where _sigchk is called:

grep sigchk *.c *.as
read.c:            _sigchk();
signal.c:_sigchk()
write.c:            _sigchk();
write.c:            _sigchk();
write.c:             _sigchk();
write.c:            _sigchk();

in particular, when we do console write, creating a competition between CP/M Functions 10 and Function 1, which in a normal case never happens, the buffered console read being expected to be exclusive but in the case of the submit input, this is exactly happening.

    case U_ERR:
         RSXPB[0]=0x7A;
    case U_RSX:
        while(nbytes--) {
             _sigchk();
             RSXPB[2]= *buf++;
             if (_piped) 
                bdos(CPMRSX,RSXPB);
             else        
                bdos(CPMWCON,RSXPB[2]);
        }
        return count;    

9. A proposed solution is to document it and to surround input interactions with proper signal handling around writes to the console when a buffered read is acquired.

signal(SIGINT,SIG_IGN);
signal(SIGINT,SIG_DFL);

in getargs.c for the prompt:

static char nxtch()
{
    if (interactive && *str == '\\' && str[1] == 0)
    {
        if (!bp)
            bp = alloc(256);
        if (isatty(fileno(stdin))) {
            signal(SIGINT,SIG_IGN);
            fprintf(stderr, "%s> ", name);
            signal(SIGINT,SIG_DFL);
        }
        gets(bp);
        str = bp;
    }
    if (*str)
        return *str++;
    return 0;
}

and c.c for the banner

    if (isatty(fileno(stdin))) {
        signal(SIGINT,SIG_IGN);
        fprintf(stderr, "HI-TECH C COMPILER (CP/M-80) V@@TAG@@\n");
        fprintf(stderr, "Copyright (C) 1984-2021 HI-TECH SOFTWARE\n");
#if	EDUC
        fprintf(stderr, "Licensed for Educational purposes only\n");
#endif	EDUC
        signal(SIGINT,SIG_DFL);
    }
	if(argc == 1) {
		argv = _getargs((char *)0, PROMPT);
    }

10. trying it now. it works. but I found another bug on the way with the compiler this time, not being able to properly convert integers as function pointers (signal_t).

PS: I found an unrelated small bug on getargs I am also chasing. After getargs is executed there is a missing carriage return or the EOL is displayed as a LF, hence my fprintf(stderr,"\n") after getargs ....

[tsupplis]

Working solution description:

1. First of all a change to the small test:

t.zip

2. a small alteration of signal.h and signal.c to define signal_t as a void* instead of a function type. P1 does not like that.

in signal.h:

typedef void* signal_t;

in signal.c:

	if(where == SIG_DFL)
		exit(0);
	((int(*)())where)(0);

signal.zip

3. a protection of the printf in getargs.c

        if (isatty(fileno(stdin))) {
            signal_t prev=signal(SIGINT,SIG_IGN);
            fprintf(stderr, "%s> ", name);
            signal(SIGINT,prev);
        }

getargs.zip

4. a change to the c.c drive (careful this is mine but the change is only on the banner)

    if (isatty(fileno(stdin))) {
        signal_t prev=signal(SIGINT,SIG_IGN);
        fprintf(stderr, "HI-TECH C COMPILER (CP/M-80) V@@TAG@@\n");
        fprintf(stderr, "Copyright (C) 1984-2021 HI-TECH SOFTWARE\n");
#if	EDUC
        fprintf(stderr, "Licensed for Educational purposes only\n");
#endif	EDUC
        signal(SIGINT,prev);
    }

Working really well now.

last small test with the c front end ... success 😀

J>type tt.sub
c
<-v hello.c
J>tt

J>c
HI-TECH C COMPILER (CP/M-80) V3.09-P009.1-dev
Copyright (C) 1984-2021 HI-TECH SOFTWARE
c> -v hello.c
0:CPP -DCPM -DHI_TECH_C -D_HTC_C -Dz80 -I HELLO.C $CTMP1.$$$
0:P1 $CTMP1.$$$ $CTMP2.$$$ $CTMP3.$$$
0:CGEN $CTMP2.$$$ $CTMP1.$$$
0:ZAS -N -oHELLO.OBJ $CTMP1.$$$
ERA $CTMP1.$$$
ERA $CTMP2.$$$
ERA $CTMP3.$$$
ERA $CTMP5.$$$
0:LINK -Z -Ptext=0,data,bss -C100h -OHELLO.COM CRT0.OBJ HELLO.OBJ LIBC.LIB
ERA HELLO.OBJ
ERA $$EXEC.$$$

J>

[tsupplis]

And last, I think we could explore the usage of
(BDOS function 109 (C_MODE) - Set or get console mode)
for CP/M 3 an PCP/M at lease

But I suppose it is another story. The number of case where the sigchk can interfere with a submit like scenario are legions: Examples of that any I/Os happening before or during console interactions, functions like getenv for example (reading files so calling _sigchk)

[agn453]

You have been busy while I was sleeping.

I also narrowed down some issues with gets() too and the problems of mixing of input and output to a stream.
The manual warns about such mixing and suggests (page 120 of the doc/HTCZ80.TXT file)

   When using one of the  read/write  modes  (with  a  '+'
   character  in the string), although they permits reading and
   writing on the same stream, it  is  not  possible  to  arbi-
   trarily  mix  input  and output calls to the same stream. At
   any given time a stream opened with a "+" mode  will  be  in
   either  an  input  or  output  state.  The state may only be
   changed when the associated buffer is empty, which  is  only
   guaranteed  immediately  after  a call to fflush() or one of
   the file positioning  functions  fseek()  or  rewind().  The
   buffer will also be empty after encountering EOF while read-
   ing a binary stream, but it is recommended that an  explicit
   call  to  fflush()  be  used  to ensure this situation. Thus
   after reading from a stream  you  should  call  fflush()  or
   fseek()  before attempting to write on that stream, and vice
   versa.

Your solution seems fine - but there may be other traps lurking in other areas of the code.

Also regarding the sensitivity to message lengths in the start-up code I noticed the x86 detection code in the start-up modules is broken (doesn't work when a compiler produced .COM file runs under MS-DOS as it should). I'll fix this as a separate issue and review/merge your recent pull request shortly after I have my breakfast!

Tony

[agn453]

Finally fixed this issue (fingers crossed)!