Using QShell and CL to work with Stream files in the IFS

It turns out that there is no simple way of generating a list of files currently residing in a folder on the IBM i IFS. Simple, in this case, would be a command like DSPLNK OUTPUT(*FILE). An API does exist, but a combination of not enough time and too much lazy proved to be quite a disincentive for me going down that route.

The issue was that we recieve a number of very big files and, to save a bit of bandwidth, these files were being zipped before being sent to the IBM i. Dropping the file into the IFS and unzipping it was easy enough but then I found myself with an archive folder containing one of more files. While I can set the name of the folder into which the files should be extracted, I have no way of determining beforehand what the file names will be.

Here’s a solution:

/* -------------------------------------------------------------------------- */
/* Program     : EXTRACTZIP                                                   */
/* Description : Retrieve and unpack a zipped archive                         */
/* Written by  : Paul Pritchard                                               */
/* Date        : 27/05/2015                                                   */
/* -------------------------------------------------------------------------- */
pgm
dcl &library *char 10 value('MYLIB')
dcl &fromfile *char 50
dcl &tombr *char 50 value('/qsys.lib/qtemp.lib/IMPORTP.file/IMPORTP.mbr')
dcl &dltfile *char 50
dclf EXTRACTP

/* Retrieve and the zipped file and unzip it.                                 */
/* I won't bore you with the details here, but the the production program is  */
/* retriving the ZIP file from an FTP server and using PKZIP to unzip it.     */
/* The extract directory is /home/EXPATPAUL/EXTRACTFLR which now contains one */
/* or more stream files                                                       */

/* Retrieve a list of extracted files                                         */
/* First, I use QShell to list the files in EXTRACTFLR. The output of this is */
/* redirected to ExtractedFiles.TXT.                                          */
/* In order to use this information, I copy the ExtractedFiles.TXT stream     */
/* to an ad-hoc physical file (QTEMP/EXTRACTP)                                */
qsh cmd('ls /home/EXPATPAUL/EXTRACTFLR/ > /home/EXPATPAUL/ExtractedFiles.TXT')
crtpf file(QTEMP/EXTRACTP) rcdlen(20)
cpyfrmstmf fromstmf('/home/EXPATPAUL/ExtractedFiles.TXT') +
           tombr('/qsys.lib/qtemp.lib/EXTRACTP.file/EXTRACTP.mbr') +
           mbropt(*REPLACE)

/* And now I can use QTEMP/EXTRACTP to drive my way through EXTRACTFLR and    */
/* copy each of the files in the archive into the IMPORTP physical file.      */
dowhile '1'
    rcvf
    monmsg msgid(CPF0864) exec(LEAVE)

    /* Copy the next sream file from the archive                              */
    chgvar &fromfile value('/home/EXPATPAUL/EXTRACTFLR/' *tcat &EXTRACTP)
    cpyfrmstmf fromstmf(&fromfile) tombr(&tombr) mbropt(*add) +
               STMFCCSID(*PCASCII)

    /* and then delete the stream file                                        */
    chgvar &dltfile value('rm /home/EXPATPAUL/EXTRACTFLR/' *tcat &EXTRACTP)
    qsh cmd(&dltfile)

    enddo

/* Clean up and exit                                                          */    
qsh cmd('rm /home/EXPATPAUL/ExtractedFiles.TXT')
dltf qtemp/EXTRACTP

endpgm

It should go without saying that some of the names have been changed and that the above program should be treated as a sample only.

Being able to move information between the QShell/IFS and traditional i5/OS environments is both useful and (in my experience) increasingly important. Although it does take a bit of thinking about, it isn’t difficult which is why I find that the oft-seen solution of “buy this tool” is both disappointing and (often) overkill.

Keeping Qshell sessions alive

Not a lot of people realise this, but the IBM i has a POSIX compliant shell environment, known as Qshell. It’s relatively basic (compared to both BASH and the native i command line) but it can be quite handy when I need (for example) to grep a source file.

One thing that has always annoyed me about Qshell, however, is that it doesn’t retain any history between sessions. Given that my workflow will involve starting at the i command line, performing a task in Qshell, and then returning to the command line, the lack of a history lead either to unnecessary typing or copying and pasting commands into and out of a text editor.

Today I noticed that the F12 key can be used to disconnect a Qshell session without actually ending it. And when I next enter the QSH command, I find myself back in the same session with my history intact.

This isn’t going to help with finding commands I typed yesterday, but it will allow me to avoid unnecessary retyping within the same day.

Footnote

Why use grep to search a source file rather than the more usual FNDSTRPDM command?

Incompetent contractors is the short answer. Incompetent contractors who introduced an unknown number of divide by zero errors is the slightly longer answer.

In RPG, the division operator is / and the comment symbol is //. I could use FNDSTRPDM to search for all the slashes and then manually scroll past all the comment lines. Or I could shortcut this process with the following piped grep:

grep -in '/' /qsys.lib/sourcelib.lib/qrpglesrc.file/program.mbr | grep -iv '//'

I’m lazy. I grep.

Fun with QShell

One thing that a lot of people fail to realise is that the IBM i includes a command environment based on POSIX and X/Open standards. This environment is known as QShell and consistes of two parts:

  • The shell interpreter (qsh) reads commands from an input source, interprets each command, and then runs the command using the services of the operating system.
  • The ulilities – external programs that provide additional functions.

You can use the QShell interactively, but the fun begins when you start integrating it with CL.

The below example is used in an interface that retrieves an FTP file from one location, processes it, and then sends the results on to another location. We wanted to retain the FTP logs and, for reasons best left unexamined, the powers that be decreed that these logs should accumulate in a database file. And they should have a timestamp.

Capturing the FTP log is a straightforward case of overriding the Output file to the log file/member. Including a timestamp is where the Qshell fun comes in.

/* Subroutine: Initialise FTPLOG Member                                     */
Subr SrFTPLOG                                                                   
   AddEnvVar EnvVar(QIBM_QSH_CMD_OUTPUT) Value(NONE) Replace(*YES)              
   Clrpfm FTPLOGFILE FTPLOG                                                     

   RtvMbrD File(FTPLOGFILE) Mbr(FTPLOG) RtnLib(&Library)                        
   ChgVar &SessionLib Value('db2 "Create Alias TMPFTPLOG for ' *Cat            + 
                            &Library *TCat '.FTPLOGFILE(FTPLOG)"')              

   RtvSysVal QDATETIME &Timestamp                                               
   ChgVar &SessionTop Value('db2 "Insert into TMPFTPLOG Values(''New Session ' +
                            *Cat %Sst(&Timestamp 1 4) *Cat '-'                 +
                            *Cat %Sst(&Timestamp 5 2) *Cat '-'                 +
                            *Cat %Sst(&Timestamp 7 2) *Cat ' '                 +
                            *Cat %Sst(&Timestamp 9 2) *Cat ':'                 +
                            *Cat %Sst(&Timestamp 11 2) *Cat ':'                +
                            *Cat %Sst(&Timestamp 13 2) *Cat ':'                +
                            *Cat %Sst(&Timestamp 15 6) *Cat ''')"')             
   Qsh Cmd(&SessionLib)                                                         
   Qsh Cmd(&SessionTop)                                                         
   Qsh Cmd('db2 "Drop Alias TMPFTPLOG"')                                         
   RmvEnvVar EnvVar(QIBM_QSH_CMD_OUTPUT)                                        
Endsubr

Member FTPLOG in file FTPLOGFILE is where the FTP logging data will be captured. This member is then copied to the permanent member as decreed by the PHB.

The example should speak for itself but what I am doing is creating an alias to the FTPLOG member, writing a single line to that member with a timestamp retirieved from the DateTime System value, and then dropping the alias.

If you want to get started with QShell, the best approach is to type Strqsh from a command line and then type help.