REXX EXECIO vs LINEIN — when is EXECIO faster?

[Anushka Sharma]

Use case:
Reading ~1M lines and parsing.
Is EXECIO with STEM more memory-efficient than LINEIN?
Any sample loop patterns?

[enrico-sorichetti]

it depends on the parameters used
ADDRESS TSO "EXECIO * DISKR MYFILE (STEM MYRECORDS."
will read ALL the records , and a large file might hog the mvs memory an termiate the read 
LINEIN will read one record at the time

P.S:
EXECIO will be faster when reading all the records in one invocation

[Anushka Sharma]

Makes sense, thanks! So in practice, would you suggest mixing both — say, using EXECIO in chunks (like 10K lines at a time) to balance speed and memory?
Or is LINEIN still the safer route for really large files?

[enrico-sorichetti]

I would use "EXECIO in chunks"
if You make the chunk size a variable to be set by an arg or statically at the beginning of the script
you will have a tunable environment by simply changing one statement
something like ...

Code: Select all

    parse arg chunksize
    if chunksize = "" then ,
       chunksize =  "a numeric value for the chunksize"
    if DATATYPE(chunksize) /= "NUM" then do
       say "ERROR, not numeric chunksize"
       exit
    end 

[enrico-sorichetti]

here is the proof of concept ...
written for open object rexx running on APPLE, 
but it will run as i on any system supporting open object rexx

Code: Select all

#! /usr/bin/env rexx

Trace "O"
numeric digits 16

xiodata = "execio.data"
parse arg blksize
if blksize = "" then ,
   blksize = 1
if DATATYPE(blksize) \= "NUM" then do
   say "ERROR, not numeric blksize"
   exit
end

-- outer loop
do blknmbr  = 1
  address hostemu "execio "blksize" diskr '"xiodata"' (stem xiobuff."
  if ( RC \= 0 ) then do
    say "EXECIO ERROR, RC="RC
    address hostemu "execio 0 diskr '"xiodata"' (finis"
    leave
  end

  if xiobuff.0 = 0 then do
    address hostemu "execio 0 diskr '"xiodata"' (finis"
    leave
  end

  -- inner loop
  say
  do recnmbr = 1 to xiobuff.0
    say align(blknmbr) align(recnmbr) xiobuff.recnmbr
  end
end

exit

align: procedure
  arg n
  return right(n, 3)

::requires "hostemu" LIBRARY

 

[Anushka Sharma]

Thanks, Enrico — super helpful.I tried the chunked approach locally and it looks promising. Before I finalize, wanted to validate a few things with the experts here:

1. For ~1M lines, is EXECIO in chunks (say 10K–50K lines per read) generally the sweet spot? Or do you tune chunk size based on LRECL/avg line length?
2. With STEM reads, is it okay to re‑use the same STEM per batch (overwrite stem. every time) or do you recommend dropping/clearing it explicitly after processing to avoid memory bloat?
3. Any simple rule for when LINEIN is safer — e.g., when lines are very long/irregular or when address space is tight?
4. On z/OS TSO: any knobs that actually move the needle — e.g., BUFNO, DYNALLOC/ALLOC FI() with BLKSIZE, or dataset attributes that help EXECIO throughput?

Below is the chunk pattern I’m testing (STEM overwrite per batch). If I’m missing something obvious, please point it out.

Code: Select all

/* REXX */
address TSO
parse arg dsname chunksize .
if dsname = "" then do
  say "Usage: execio_chunk.rex '<dsname>' <chunksize>"
  exit 8
end
if chunksize = "" | datatype(chunksize,'W') = 0 then chunksize = 20000
"ALLOC FI(INFILE) DA('"dsname"') SHR"     /* or OLD if needed */
if RC <> 0 then do
  say "ALLOC failed RC="RC
  exit RC
end
tot = 0
do while 1 = 1
  /* read a chunk into STEM */
  "EXECIO "chunksize" DISKR INFILE (STEM REC."
  if RC <> 0 then do
    say "EXECIO read failed RC="RC
    leave
  end
  if REC.0 = 0 then leave    /* EOF */
  /* process this batch */
  do i = 1 to REC.0
    line = REC.i
    /* parse your fields here */
    /* parse var line col1 1 col2 10 col3 +10 ... */
    /* ...do work... */
  end
  tot = tot + REC.0
  /* optional: free stem storage hint (overwrite next loop anyway) */
  /* do i = 1 to REC.0; REC.i = ""; end; REC.0 = 0 */
end
"EXECIO 0 DISKR INFILE (FINIS"
"FREE FI(INFILE)"
say "Total lines processed:" tot
exit 0

For comparison, here’s the simple LINEIN() loop I use when I want minimal footprint (yes, slower but steady):

Code: Select all

/* REXX (CMS/ooRexx compatible style) */
call linein 'MYFILE', 1, 1     /* position to start if needed */
tot = 0
do while lines('MYFILE') > 0
  L = linein('MYFILE')
  if L = "" & rc <> 0 then leave
  /* process L */
  tot = tot + 1
end
call lineout 'MYFILE'          /* close */
say "Total lines processed:" tot
exit 0
``

Plan of action (please correct me if off):

• Use EXECIO in chunks as default for big reads, keep chunk size tunable via arg, and watch storage + elapsed.
• Keep STEM stable (overwrite each batch), avoid building giant global arrays.
• Fall back to LINEIN when memory headroom is tight or when per‑line streaming logic dominates.

Would love your thoughts on the above and any chunk size heuristics you’ve seen work well in production.

Thanks!

[enrico-sorichetti]

topic locked, it is just spinning around with useless speculations