Most of us have been there … something somewhere goes wrong … things are checked, changes are undone, tests are re-run and in the end you have no idea why a failure happens.
Who You Gonna Call?
Yep, it is time to open a Case at IBM technical support … So you open a Case and you type in as much detail as possible about when and what happened but it is *never* enough! In the world of Db2, the first question that *always* comes back is „Please supply us with further information“, like:
- SYSLOG
- Master Log
- MEPL
- Detailed EREP
- Complete SVC dump
WTF? („What’s That For“ before anyone complains)
SYSLOG
The syslog is the console of a z/OS system and any and all interesting, and sometimes not so interesting, messages from *all* running „things“ are in here – it is normally enormous! The problem begins when IBM Technical Support asks „please provide us with the SYSLOG from 06:00 to 06:30 on the day of the event“.
SDSF
SDSF is your friend here and I really mean it! All you do is go to SDSF and then enter primary command LOG. From this panel you enter three primary commands, one after another, and you are done!
- PT ODSN ‚your.dataset.name‘ * NEW
- PT 06.00.00 22/06/2022 06.30.00 22/06/2022
- PT CLOSE
That is it! Your dataset will then just have the data from between those times. This is *extremely* handy! Note that the date format is locale-dependent and, as I am in Europe, we have DD/MM/YYYY. I am sure you know your own date format!
Master LOG
This is the first SDSF dataset in your ssidMSTR STC. So, once more in SDSF, using *MSTR as a prefix and then putting line command ? next to the sub-system in question shows you three DDNAMEs. The first one, JESMSGLG, is the one they normally need. Here you use line command XDC to get an SDSF Open Print Data Set window:
xxxxMSTR STC09394 SDSF Open Print Data Set
COMMAND INPUT ===> SCROLL ===> CSR
Data set name ===> 'xxxxxxx.SYSLOG.PRINT'
Member to use ===>
Disposition ===> NEW (OLD, NEW, SHR, MOD)
Management class ===> (Blank for default management class)
Storage class ===> (Blank for default storage class)
Volume serial ===> (Blank for authorized default volume)
Device type ===> (Generic unit or device address)
Data class ===> (Blank for default data class)
Space units ===> CYLS (BLKS, TRKS, CYLS, BY, KB, or MB)
Primary quantity ===> 19 (In above units)
Secondary quantity ===> 19 (In above units)
Directory blocks ===> (Zero for sequential data set)
Record format ===> FBA
Record length ===> 121
Block size ===>
Data set name type ===> (LIBRARY, blank, ... See Help for more)
Extended attributes ===> (NO, OPT, or blank) Here you can see I choose type FBA, LRECL 121 and a disposition of NEW for a new dataset. Hit ENTER and SDSF tells you how many lines it just wrote to that file:
PRINT CLOSED 23025 LINE
View the file and max down to the bottom:
023019 0------ JES2 JOB STATISTICS ------ 023020 - 17 MAY 2022 JOB EXECUTION DATE 023021 - 2 CARDS READ 023022 - 28,616 SYSOUT PRINT RECORDS 023023 - 0 SYSOUT PUNCH RECORDS 023024 - 3,099 SYSOUT SPOOL KBYTES 023025 - 50,488.70 MINUTES EXECUTION TIME
So we know we are in the correct file! Here you can do some updating of „sensitive“ data like IP address, User Name etc. Remember to just change the data, not blindly delete it! Naturally, you can delete stuff *after* the event of interest and probably a ton of stuff from *before* but be careful what you delete!
MEPL
Say what? MEPL is the Module Entry Point List and IBM need it to see which PTFs and APARs have been applied in the application address space and the Db2 system. To get a MEPL I use a normal Utility job jcl with DIAGNOSE and a DISPLAY MEPL like this:
//MEPL EXEC PGM=DSNUTILB,REGION=32M,
// PARM=(ssss,'DIAGNOSEMEPL')
//STEPLIB DD DISP=SHR,DSN=DSNsss.SDSNEXIT.ssss
// DD DISP=SHR,DSN=DSNsss.SDSNLOAD
//CEEDUMP DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DIAGNOSE
DISPLAY MEPL
DIAGNOSE END
/* This will output the MEPL to SYSPRINT which starts like this:
DSNU000I 173 12:57:08.82 DSNUGUTC - OUTPUT START FOR UTILITY, UTILID = DIAGNOSEMEPL
DSNU1044I 173 12:57:08.83 DSNUGTIS - PROCESSING SYSIN AS EBCDIC
DSNU050I 173 12:57:08.83 DSNUGUTC - DIAGNOSE DISPLAY MEPL
DSNU861I 173 12:57:08.84 DSNUDIAG - DISPLAY MEPL FOR SUBSYSTEM xxxx
0000 20B92820 C2C5D7D3 0140D4C5 D7D360D3 C9D2C540 C6D6D940 C4E2D5E4 E3C9D3C2 *....BEPL. MEPL-LIKE FOR DSNUTILB*
0020 28100000 C4E2D5C1 C1404040 F0F761F1 F461F1F6 E4C9F3F9 F3F9F340 00000000 *....DSNAA 07/14/16UI39393 ....*
0040 28100100 C4E2D5C1 D7D9C840 F1F261F2 F361F1F5 F1F34BF4 F6404040 00000000 *....DSNAPRH 12/23/1513.46 ....*
0060 28100200 C4E2D5C6 D4D5C6D4 F1F061F1 F761F1F8 E4C9F5F8 F8F4F040 00000000 *....DSNFMNFM10/17/18UI58840 ....*
0080 28100240 C4E2D5C6 D7D4E2C7 F1F061F1 F761F1F8 E4C9F5F8 F8F4F040 00000000 *... DSNFPMSG10/17/18UI58840 ....*
.
.
.It is quite long! Here in my test system nearly 5000 lines are written to SYSPRINT. Then, like with the ssidMSTR, I use ? against the job and then XDC against the SYSPRINT DD card this time to create another file with type FBA and LRECL 133 to get your.mepl.list.
Detailed EREP
Now it gets interesting… The EREP (Environmental Record Editing and Printing Program) is the API to the system LOGREC dataset where all „events of interest“ on a z/OS LPAR are recorded. It contains far less than the console log but is a treasure trove of data for the IBM Technical Support.
Here’s my job to simply do a Detailed EREP as per IBM standards:
//*------------------------------------------------------------------*/ //* EREP: DETAILED REP PRINT */ //*------------------------------------------------------------------*/ //EREP EXEC PGM=IFCEREP1,PARM='CARD' //SERLOG DD DISP=SHR,DSN=xxxxxxxx.LOGREC //DIRECTWK DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(50,50)) //EREPPT DD SYSOUT=*,DCB=BLKSIZE=133 //TOURIST DD SYSOUT=*,DCB=BLKSIZE=133 //SYSIN DD * ACC=N HIST=N ZERO=N PRINT=PS TYPE=S /* //* IF REQUIRED YOU CAN ADD DATE, TIME RANGES TO FILTER DOWN //* WITHIN THE SYSIN LIKE: //* DATE=(YYDDD,YYDDD) //* TIME=(HHMM-HHMM)
Do not forget to give your LOGREC DSN for the SERLOG DD. Most of the time I get just a few rows of output and then create another file using XDC from the EREPPT DD name but this time with type FB and LRECL 133 to get the.erep.list
Complete SVC Dump
If your Db2 system receives a dump, for whatever reason, it normally writes out an SVC dump to a special dataset that can be used to analyze what went wrong. It is very important that the SVC dump is complete and *not* partial …
Default Size
The default size is only 500MB which is way too small for a halfway decent production Db2 sub-system these days. It must normally be increased to at least 16000MB. To change this you issue a console command like:
CHNGDUMP SET,SDUMP,MAXSPACE=16000M
But make sure you have enough local page datasets space to handle your normal load PLUS the size of this dump dataset…auxilliary swapping (paging) while dumping is a painfully slow experience you do not want to suffer!
If successful, the SVC dump will be COMPLETE and then you are nearly done …
File Transfer
Most of the files I have described so far are quite small but the SVC dump is a monster. You must TERSE it using JCL like:
//AMATERSE EXEC PGM=AMATERSE,PARM='SPACK' //SYSPRINT DD SYSOUT=* //SYSUT1 DD DISP=SHR, // DSN=xxxxxxxx.xxxxxxxx //SYSUT2 DD DISP=(,CATLG),UNIT=SYSDA, // DSN=xxxxxxxx.xxxxxxxx.TRS, // SPACE=(CYL,(99,99),RLSE)
I use the SPACK parameter which is, according to the documentation, much better at compression than the PACK parameter. Fun factoid of the day: SPACK is the „complex“ format whereas PACK is the „simple“ format – Gotta love IBM for that! IBM do prefer the TERSE style of compression, and please do *not* change the file ending! Then doing a ZIP has no real bonus and just confuses the automatic systems at IBM. Leave „.TRS“ at the end and they know it has been TERSED.
Then download the xxxxx.xxxxx.TRS file as BINARY to the PC and all the other files as TEXT to the PC. Then simply upload by drag-and-drop to your IBM Case and you are ready for the next question!
Have you Switched it Off and On again?
I wish I never hear this about a mainframe Db2 problem!
I hope this was of some interest, and if you have any other Tips & Tricks about getting „standard“ data to IBM, I would love to hear from you!
TTFN
Roy Boxwell