'\n' compiles to EBCDIC NEL (X'15'), not LF (X'25') — the asymmetry behind the ecosystem's translation bugs

[mgrossmann]

Moved here from mvslovers/crent370#26. It was filed against crent370 but it is a compiler question — the byte comes out of cc370, not out of the library.

Still true today

char nl = '\n';
char s[] = "a\n";

$ cc370 -O1 -S nl.c
NL       EQU   *
         DC    X'15'
S        EQU   *
         DC    C'a'
         DC    X'15'
         DC    X'0'

X'15' is NEL. IBM convention, and the source of a long tail of trouble.

The asymmetry

IBM-1047 (the z/OS USS / Zowe default) does not round-trip:

ASCII LF  (0x0A) -> atoe -> EBCDIC NEL (0x15)
EBCDIC NEL(0x15) -> etoa -> ASCII LF  (0x0A)    round trip OK

EBCDIC LF (0x25) -> etoa -> ASCII NEL (0x85)    not LF
ASCII NEL (0x85) -> atoe -> EBCDIC NEL (0x15)   not 0x25

CP037 is symmetric:

ASCII LF  (0x0A) <-> EBCDIC LF  (0x25)
ASCII NEL (0x85) <-> EBCDIC NEL (0x15)

So a program that writes '\n' and a tool chain that translates CP037 disagree
about what a line end is, and every component in the ecosystem has grown its own
patch over that seam.

Why it belongs to the compiler

Changing it in the library is not possible: the byte is baked into the object at
compile time, in string literals as much as in character constants. Only the
backend that emits DC X'15' can emit DC X'25'.

What has to be weighed before changing it

• Everything already compiled. Load modules in the field carry X'15'. A
  library reading data written by an older program sees the other byte.
• libc370's own I/O. esc_print(), the record writers, the DD line handling
  — anything comparing against a newline literal changes meaning together with
  the compiler, which is good, but only if compiler and library are rebuilt as
  one.
• z/OS convention. X'15' is what IBM compilers emit. Diverging makes cc370
  correct in CP037 and unusual on z/OS.
• A flag instead of a switch. -fnewline=lf|nel would let the ecosystem move
  deliberately instead of all at once, at the cost of one more dimension in
  which two objects can disagree.

Filed as an investigation, not a change request: the first deliverable is a
survey of who compares against '\n' in the mvslovers projects, and what breaks
in each direction.

Original discussion and the ecosystem-wide analysis: mvslovers/crent370#26.