*COMDECK COMCTOK
 TOK      CTEXT  COMCTOK 2.0 - TOKEN GENERATOR. 
 TOK      SPACE  4,10 
          IF     -DEF,QUAL$,1 
          QUAL   COMCTOK
          BASE   D
 TOK      SPACE  4,10 
***       TOK - TOKEN GENERATOR.
*         P CLARE  77/01/20 
*         P CLARE  78/04/27  VERSION 2.0
 TOK      SPACE  4,10 
***       TOK - TOKEN GENERATOR.
* 
* 
*         *COMCTOK* TRANSLATES AN INPUT SOURCE LINE FROM ITS PACKED 
*         (10 CHARACTER PER WORD) FORM TO ITS ENTOKENED FORM.  THIS 
*         *ENTOKENED FORM* IS DESCRIBED BY A HUMAN USING THE *TOGEL*
*         MACROS (DEFINED IN *COMATOK*).  THE PROGRAMMER ENCODES
*         THE *TOGEL* MACROS IN SUCH A FASHION AS TO DESCRIBE THE 
*         PROCESS BY WHICH *TOK* CAN ENTOKEN ANY INPUT SOURCE LINE
*         THAT THE PROGRAMMER IS INTERESTED IN. 
* 
*         THE *TOGEL* MACROS GENERATE A *TOGEL* OBJECT MODULE,
*         CALLED *TOM*, WHICH THE PROGRAMMER THEN FEEDS TO *TOK*. 
*         *TOK* WILL EXECUTE/INTERPRET A *TOM*, PERFORMING THE
*         SPECIFIED ENTOKENING TASKS UPON THE INPUT SOURCE LINE.
* 
*         *TOK* CAN INTERPRET THE FOLLOWING *TOGEL* BINARY
*         INSTRUCTIONS -- 
* 
*                GROUP (A..Z),TOT,SQZ 
*                GROUP (A..Z),TOT,NSQZ
*                IFT (A..Z) 
*                  THEN 
*                  ELST 
*                  ENDT 
*                GOTO XX
*                CALT XX
*                CASEOF (A..Z)
*                  TOKEN
*                ELSC 
*                  TOKEN
*                ENDC 
*                GENTOK TOT 
*                XEQ XXX
*                PROC XXX 
* 
*         THE TOKEN GENERATION PROCESS, THEN, IS EXTREMELY TABLE DRIVEN.
*         *TOK* HAS VERY LITTLE INTERNAL LOGIC OF ITS OWN.  IT DOES 
*         EXACTLY WHAT THE *TOM* THAT IS DRIVING IT TELLS IT TO DO. 
* 
*         EACH PROCESSING NODE WITHIN *TOK* EXECUTES/INTERPRETS A 
*         SINGLE *TOGEL* BINARY INSTRUCTION FROM THE *TOM*.  THESE
*         *TOGEL* BINARY INSTRUCTION INTERPRETERS ARE CALLED
*         *TOK* FUNCTIONAL UNITS, *TOFU*S FOR SHORT.
* 
*         THE ENTIRE SUBJECT OF TOKEN GENERATION AND HOW TO UNDERSTAND
*         AND USE *TOGEL* AND *TOK* IS DEALT WITH MUCH MORE THOROUGHLY
*         IN THE FTN 5 IMS IN THE CHAPTER FOR THE DECK *LEX*. 
*         I STRONGLY ADVISE READING IT BEFORE TRYING TO MESS IN THIS
*         CODE.  IT WILL SAVE YOU A LOT OF TIME...
* 
************************************************************************
* 
*         USING *COMCTOK* 
* 
*         USING *COMCTOK* INVOLVES DOING THE FOLLOWING -- 
* 
*           1. CONFIGURING *COMSTOK* (COMMON TOKEN GENERATOR INTERFACE
*              TEXT), *COMATOK* (COMMON TOKEN GENERATOR TOGEL MACROS),
*              AND *COMCTOK* (COMMON TOKEN GENERATOR).
* 
*              THIS ALSO INVOLVES THE SETTING UP OF SOME ASSEMBLY-TIME
*              DEPENDENCIES.  SEE BELOW.
* 
*           2. LEARNING ENOUGH ABOUT *TOGEL* (TOKEN GENERATION LANGUAGE)
*              TO WRITE A *TOGEL* PROGRAM THAT WILL DIRECT *TOK* AS TO
*              HOW YOU WISH HIM TO ENTOKEN.  I SUGGEST A READING OF THE 
*              FTN 5 IMS FOR THE DECK *LEX*, AS THERE IS A THOROUGH 
*              DESCRIPTION OF *TOGEL* PROGRAMMING THERE.
* 
*           3. WRITING YOUR *TOGEL* PROGRAM USING THE MACROS CONTAINED
*              IN *COMATOK* (COMMON TOKEN GENERATOR *TOGEL* MACROS).
* 
*           4. SETTING UP *TOKCOM* (THE USER/TOK COMMUNICATIONS AREA).
*              *TOKCOM* IS CONCEPTUALLY SOMETHING LIKE A FILE *FET* 
*              IN THAT IT IS A PARAMETER LIST THAT IS USED BOTH FOR 
*              INPUT-TO-TOK AND OUTPUT-FROM-TOK INFORMATION.
* 
*              IT SHOULD CONTAIN, AMONG OTHER THINGS, THE ADDR THAT YOU 
*              WANT *TOK* TO ENTOKEN FROM, AND THE ADDR THAT YOU WISH 
*              *TOK* TO ENTOKEN TO.  SEE NEXT SECTION FOR MORE
*              INFORMATION ON *TOKCOM*. 
* 
*           5. CALLING *TOK* (COMMON TOKEN GENERATOR) VIA AN *RJ* WITH
*              (B2) SET TO THE FWA OF YOUR *TOKCOM*.
* 
*              NOTE THAT THIS CALLING SEQUENCE ALLOWS *TOK* TO BE USED
*              WITH MULTIPLE *TOKCOM*S. 
* 
************************************************************************
* 
*         CONFIGURING *COMCTOK* 
* 
*         *COMCTOK* REQUIRES THE FOLLOWING COMDECKS --
* 
*                COMACPU - GENERAL CPU MACROS (IN *CPUTEXT*). 
* 
*                COMADEF - STRUCTURED FIELD DECLARATION MACROS. 
* 
*                COMAREG - REPLACEMENT FOR R= PSEUDO INSTRUCTION
*                          (IN *CPUTEXT*).
* 
*                COMATOK - COMMON TOKEN GENERATOR TOGEL MACROS. 
* 
*                COMCBUB - BURST/BUILD CHARACTERS WITH BLANK SQUEEZE. 
* 
*                COMCBUN - BURST/BUILD CHARACTERS WITH NO BLANK SQUEEZE.
* 
*                COMSTOK - COMMON TOKEN GENERATOR INTERFACE TEXT. 
* 
*         IN ADDITION, IF YOU DECIDE TO INSTALL *COMCTOK* IN *TEST* MODE
*         (SEE BELOW), THEN *COMDTOK* (COMMON TOKEN GENERATOR IDP 
*         UTILITY) MUST BE PROVIDED (ALONG WITH A PROPERLY CONFIGURED 
*         *COMCIDP*, OF COURSE).
* 
*         IN THE FOLLOWING DISCUSSION, *ASSEMBLY-TIME DEPENDENCIES* 
*         ARE DEFINED TO BE THOSE INTERFACES THAT THE *COMCTOK* 
*         INSTALLER **MUST** PROVIDE, WHILE *ASSEMBLY-TIME OPTIONS* ARE 
*         THOSE INTERFACES THAT ARE OPTIONAL. 
* 
*-----------------------------------------------------------------------
* 
*         ASSEMBLY-TIME DEPENDENCIES -- 
* 
*-----------------------------------------------------------------------
* 
*         BB./BN. STRUCTURE DEFINITIONS.
* 
*         *BB.* AND *BN.* ARE THE STRUCTURE DEFINITIONS FOR *COMCBUB* 
*         AND *COMCBUN*, RESPECTIVELY.  THESE STRUCTURES DESCRIBE/DEFINE
*         WHAT A *GROUP* GENERATED VIA *GROUP,,SQZ/COMCBUB* AND/OR
*         *GROUP,,NSQZ/COMCBUN* LOOKS LIKE. 
* 
*         FIELDS WITHIN *BB./BN.* STRUCTURES ARE AS FOLLOWS --
* 
*                TOC = TOKEN CHARACTERS.  I.E. *COMCBUB/COMCBUN* WILL 
*                      BURST/BUILD CHARACTERS INTO THIS FIELD.
* 
*                TOT = TOKEN TYPE.  I.E. *COMCTOK* WILL FILL THIS 
*                      FIELD WITH THE SPECIFIED TOKEN TYPE. 
* 
*                      NOTE THAT *TOT* FIELD CAN BE OF NULL LENGTH
*                      IF DESIRED.  I.E. -- 
* 
*                               DESCRIBE BN.
*                          TOC  DEFINE   60 
*                          TOT  DEFINE   0
* 
*                      THIS CAN BE USEFUL FOR PACKING UP CHARACTER
*                      STRINGS THAT DO NOT HAVE A TOKEN TYPE ASSOCIATED 
*                      WITH THEM. 
* 
*-----------------------------------------------------------------------
* 
*         "O."XXX - TOKEN (OPERATOR/OPERAND) TYPES. 
* 
*         THE *COMCTOK* INSTALLER MUST DEFINE ALL POSSIBLE TOKEN TYPES
*         USED IN THE TOGEL PROGRAM, WITH THE EXCEPTION OF "O."BOS
*         (BEGINNING OF STMT) AND "O."EOS (END OF STMT) TOKENS. 
*         SEE "O."BOS AND "O."EOS IN *ASSEMBLY-TIME OPTIONS* BELOW. 
* 
*         TOKEN TYPES ARE COMPASS SYMBOLS THAT HAVE THE FORM -- 
* 
*                "O."XXX
* 
*         WHERE  "O." = TOKEN TYPE SYMBOL PREFIX MICRO.  SEE "O." 
*                       BELOW IN *ASSEMBLY-TIME OPTIONS*. 
* 
*                XXX  = A UNIQUE TOKEN TYPE, AS USED IN TOGEL MACRO 
*                       REFERENCES. 
* 
*                       FOR EXAMPLE --
* 
*                          GROUP (0..9),XXX,SQZ 
* 
*                       WILL GROUP THE CHARACTERS 0 THRU 9 INCLUSIVE
*                       INTO THE TOKEN *"O."XXX*. 
* 
* 
*-----------------------------------------------------------------------
* 
*         ASSEMBLY-TIME OPTIONS --
* 
*         NOTE THAT FOR THE INSTALLER'S OPTIONS TO OVERRIDE THE DEFAULTS
*         SET UP IN *COMSTOK* (COMMON TOKEN GENERATOR INTERFACE TEXT),
*         THAT THE OVERRIDING OPTIONS MUST **PRECEDE** *COMSTOK*. 
* 
*-----------------------------------------------------------------------
* 
*         BICH - BIAS CHARACTER.
* 
*         *BICH* SPECIFIES THE CHARACTER THAT IS TO HAVE BINARY VALUE 
*         00B (I.E. BIT 59 IN CHARACTER SHIFT MASK) IN *COMCTOK/COMCBUB/
*         COMCBUN*S ROTATED CHARACTER SET.  SEE *COMCBUB* PREAMBLE FOR
*         A DESCRIPTION OF *BICH*.
* 
*         DEFAULT:  BICH  =  1RM
* 
*-----------------------------------------------------------------------
* 
*         "BLOWUP" - CATASTROPHIC ERROR ADDRESS MICRO.
* 
*         *COMCTOK* WILL JUMP TO "BLOWUP" IF IT DETECTS AN UNRECOVERABLE
*         ERROR.
* 
*         DEFAULT:  BLOWUP  MICRO  1,,/*O+4S15/ 
* 
*-----------------------------------------------------------------------
* 
*         CHAR - CHARACTER WIDTH. 
* 
*         *CHAR* SPECIFIES THE WIDTH (IN BITS) OF A SINGLE CHARACTER. 
*         THIS OPTION IS INTENDED FOR DPC/ASCII CHARACTER SETS, BUT 
*         IS USELESS BECAUSE CURRENTLY *COMCTOK* DOES NOT SUPPORT 
*         ASCII.
* 
*         DEFAULT:  CHAR  =  6  (I.E. DPC)
* 
*-----------------------------------------------------------------------
* 
*         "O." - TOKEN (OPERATOR/OPERAND) TYPE PREFIX MICRO.
* 
*         *COMCTOK* CONSTRUCTS ALL TOKEN TYPE SYMBOLS SPECIFIED ON
*         *TOGEL* INSTRUCTION/MACRO REFERENCES BY PREFIXING THE 
*         SPECIFIED CHARACTERS WITH "O.". 
* 
*         FOR EXAMPLE --
* 
*                GROUP (A..Z),VAR,SQZ 
* 
*         GENERATES A REFERENCE TO THE TOKEN TYPE *VAR*, WHICH
*         IS REALLY A REFERENCE TO THE COMPASS SYMBOL *"O."VAR*.
* 
*         DEFAULT:  O.  MICRO 1,,/O./ 
* 
*-----------------------------------------------------------------------
* 
*         "O."BOS - BEGINNING OF STATEMENT TOKEN TYPE.
* 
*         *COMCTOK* GENERATES AN *"O."BOS* TOKEN AS THE 1ST TOKEN IN
*         THE STATEMENT, IF NOT IN *RESTART* MODE.  SEE *TC.RSF/TC.REST*
*         IN *TOKCOM* BELOW.
* 
*         DEFAULT:  "O."BOS  =  0 
* 
*-----------------------------------------------------------------------
* 
*         "O."EOS - END OF STATEMENT TOKEN TYPE.
* 
*         *COMCTOK* GENERATES AN *"O."EOS* TOKEN, BEFORE EXITING, TO
*         MARK THE END OF THE STATEMENT IN THE TOKEN BUFFER.
* 
*         DEFAULT:  "O."EOS  =  0 
* 
*         NOTE THAT THE DEFAULTS FOR "O."BOS AND "O."EOS MEAN THAT A
*         STATEMENT IN THE TOKEN BUFFER WILL BE DELIMITED BY 2 FULL ZERO
*         WORDS.
* 
*-----------------------------------------------------------------------
* 
*         "O=" - *TOKEN* *TOGEL* INSTRUCTION LABEL PREFIX MICRO.
* 
*         IF THE *CASEOF* TOGEL INSTRUCTION MACRO IS USED, THEN TOKENS
*         THAT ARE TO BE GENERATED VIA THIS MECHANISM WILL HAVE A 
*         COMPASS LABEL THAT IS CONSTRUCTED BY PREFIXING "O=" TO THE
*         TOKEN TYPE. 
* 
*         FOR EXAMPLE --
* 
*                CASEOF (+..*)
*                  TOKEN PLUS  .... HAS LABEL *"O="PLUS*
*                  TOKEN MIN   .... HAS LABEL *"O="MIN* 
*                  TOKEN STAR  .... HAS LABEL *"O="STAR*
*                ENDC 
* 
*         DEFAULT:  O=  MICRO  1,,/O=/
* 
*         THIS OPTION IS INTENDED PRIMARILY TO AVOID SYMBOL CONFLICTS 
*         THAT MIGHT OCCUR WITH THE HOST. 
* 
*-----------------------------------------------------------------------
* 
*         TEST - *TEST* MODE. 
* 
*         THE *COMCTOK* INSTALLER CAN SELECT *COMCTOK* TO BE IN *TEST*
*         MODE, WHICH FORCES *COMCTOK* TO HELP US DEBUG OUR INTERFACE 
*         TO *COMCTOK* AND ALSO HELP US TO DEBUG OUR TOGEL PROGRAMS.
* 
*         IF THE SYMBOL *TEST* IS SET .NZ., THEN *COMCTOK* WILL FORCE 
*         ALL REGISTERS BACK OUT TO *TOKCOM* (WHERE THEY ARE KEPT 
*         WHEN NECESSARY) BEFORE EACH AND EVERY TOGEL BINARY INSTRUCTION
*         IS EXECUTED/INTERPRETED.  THIS HELPS PROVIDE AN AUDIT TRAIL 
*         OF WHAT IS GOING ON.
* 
*         IF THE SYMBOL *TEST* IS SET .NZ., THEN THE INSTALLER ALSO 
*         OPTIONALLY HAS THE ABILITY TO INTERFACE *COMCTOK* WITH
*         *COMCIDP* (INTERACTIVE DEBUG PACKAGE) VIA *COMDTOK* (IDP
*         UTILITY FOR *COMCTOK*). 
* 
*         *COMDTOK* CONTAINS SOME USEFUL DEBUGGING TOOLS THAT INTERFACE 
*         DIRECTLY (AND EASILY) WITH IDP.  *COMDTOK* PROVIDES THE 
*         FOLLOWING COMMANDS, AVAILABLE THRU IDP -- 
* 
*                BRTOK XXX    BREAKPOINT WHEN THE TOGEL INSTRUCTION 
*                             *XXX* IS NEXT TO BE EXECUTED.  IF *XXX* 
*                             IS OMITTED, *COMDTOK* WILL BREAKPOINT 
*                             ON ANY AND ALL TOGEL INSTRUCTIONS.
* 
*                TOGEL FWA,LWA,LEN  DEASSEMBLES TOGEL BINARY
*                             INSTRUCTIONS BACK INTO A READABLE FORM. 
* 
*                TOKEN FWA,LWA,LEN  DISPLAYS THE TOKEN BUFFER.
* 
*                UBTOK        UNBREAK TOGEL BREAKPOINT SET VIA *BRTOK*. 
* 
************************************************************************
* 
*         *TOKCOM*
* 
*         *TOKCOM* IS THE USER/TOK COMMUNICATIONS AREA THAT IS USED 
*         THROUGHOUT TOKEN GENERATION TO GET AND SAVE CRITICAL
*         ENTOKENING INFORMATION/REGISTERS. 
* 
*         LOCATIONS IN *TOKCOM* ARE DEFINED VIA ORDINALS THAT TAKE
*         THE FORM -- 
* 
*                TC.XXX 
* 
*         WHERE *TC.* IS A COMMON PREFIX, AND *XXX* IS A UNIQUE ORDINAL 
*         NAME.  THESE SYMBOLS ARE DEFINED IN *COMSTOK* (COMMON TOKEN 
*         GENERATOR INTERFACE TEXT), AND IT IS **STRONGLY** ADVISED 
*         THAT THE USER/INSTALLER MAKE HER *TOKCOM* REFERENCES VIA THESE
*         SYMBOLS.  THIS WILL PREVENT ANY STUPID BUGS ASSOCIATED WITH 
*         *TOKCOM* CHANGES OR ADDITIONS FROM BURNING YOU.  ALSO, THE
*         LENGTH OF *TOKCOM* (IN WORDS) IS DEFINED VIA THE SYMBOL 
*         *L.TCOM* (ALSO DEFINED IN *COMSTOK*).  *TC.XXX* SYMBOLS ARE 
*         DESCRIBED BELOW.
* 
*         *COMCTOK* EXPECTS THAT THE CALLER SET UP *TOKCOM* IN ONE
*         OF TWO WAYS: THE 1ST IS AN *EASY* FORM; AND THE 2ND, A MORE 
*         DIFFICULT (ALBEIT, MORE FLEXIBLE) FORM.  THE ORIGINAL VERSION 
*         OF *COMCTOK* DID NOT HAVE THE *EASIER* FORM: IT AROSE BECAUSE 
*         THE *HARDER* FORM TURNED OUT TO BE TOO HARD, AND OVERLY 
*         FLEXIBLE FOR MOST *COMCTOK* APPLICATIONS. 
* 
*         IN THE FOLLOWING DISCUSSION, USE OF SYMBOLS OF THE FORM 
*         *TC.XXX* IMPLIES THE CONTENTS OF A CELL THAT IS AT THE
*         LOCATION -- 
* 
*                FWA+TC.XXX 
* 
*         WHERE *FWA* IS THE FWA OF THE CALLER'S *TOKCOM*.
* 
* 
*         THE *EASY* FORM --
*         --------------- 
* 
*           TC.FWA  = ADDR OF THE WORD CONTAINING THE 1ST CHARACTER TO
*                     ENTOKEN (SOURCE IS IN PACKED, 10 CHARS PER WORD,
*                     FORMAT).
* 
*                     (TC.FWA) IS NOT ALTERED BY *COMCTOK*. 
* 
*           TC.FCP  = FIRST CHARACTER POSITION WITHIN (TC.FWA) TO BEGIN 
*                     ENTOKENING.  CHARACTERS ARE NUMBERED FROM LEFT-TO-
*                     RIGHT, 1 TO 10, WITHIN A WORD AS FOLLOWS -- 
* 
*                         +--+--+--+--+--+--+--+--+--+--+ 
*                         I 1I 2I 3I 4I 5I 6I 7I 8I 9I10I 
*                         +--+--+--+--+--+--+--+--+--+--+ 
* 
*                     FOR EXAMPLE, IN 10HABCDEFGHIJ, *A* IS IN CHARACTER
*                     POSITION 1, *B* IS IN CHARACTER POSITION 2, ETC.
* 
*                     (TC.FCP) IS NOT ALTERED BY *COMCTOK*. 
* 
*           TC.LWA  = ADDR OF THE WORD CONTAINING THE LAST CHARACTER TO 
*                     ENTOKEN.
* 
*                     (TC.LWA) IS NOT ALTERED BY *COMCTOK*. 
* 
*           TC.LCP  = CHARACTER POSITION WITHIN (TC.LWA) OF LAST CHAR 
*                     TO ENTOKEN.  CHARACTERS ARE NUMBERED AS FOR 
*                     *TC.FCP* ABOVE. 
* 
*                     (TC.LCP) IS NOT ALTERED BY *COMCTOK*. 
* 
*           TC.P    = ADDR OF NEXT *TOGEL* INSTRUCTION TO EXECUTE/
*                     INTERPRET.
* 
*           TC.TBA  = ADDR TO STORE NEXT TOKEN THAT IS GENERATED. 
* 
*                     NOTE THAT CALLER **MUST** HAVE ALLOCATED ENOUGH 
*                     SPACE TO ACCOMODATE THE **MAXIMUM** NR OF TOKENS
*                     THAT *COMCTOK* CAN GENERATE.  THIS IS TO SAY THAT 
*                     *TOK* DOES NOT CHECK THE *TOKEN BUFFER* FOR ANY 
*                     SORT OF OVERFLOW CONDITION...HE JUST KEEPS
*                     GENERATING TOKENS FOR THE LENGTH SPECIFIED
*                     BY THE CALLER VIA *TC.LWA/TC.LCP* OR *TC.SOL*.
* 
*                     UPDATED UPON HITTING END-OF-LINE (IN *TEST* 
*                     MODE, UPDATED BEFORE EVERY *TOGEL* INSTRUCTION).
* 
*           TC.RSF  = .NZ. IF THIS IS A *COMCTOK* RESTART, ELSE .ZR.
* 
*                     THIS OPTION IS FOR CALLERS WHO HAVE TO DEAL WITH
*                     CONTINUATION LINES, AND WOULD LIKE A WAY TO 
*                     RESTART *COMCTOK* WHERE IT LEFT OFF ON THE
*                     PRECEDING LINE. 
* 
*                     (TC.RSF) IS NOT ALTERED BY *COMCTOK*. 
* 
*           TC.REST = ADDR OF A USER PROCESSOR FOR *RESTART*.  CAN BE 
*                     SET VIA *ON RESTART* TOGEL MACRO. 
* 
* 
*         THE *HARD* FORM --
*         --------------- 
* 
*           TC.FWA  = .ZR. TO INDICATE *HARD* FORM. 
* 
*           TC.FCP  = NOT USED. 
* 
*           TC.LWA  = NOT USED. 
* 
*           TC.LCP  = NOT USED. 
* 
*           TC.P    = SAME AS FOR *EASY* FORM.
* 
*           TC.SOA  = ADDR OF CURRENT SOURCE INPUT (10 CHARS PER WORD)
*                     WORD.  FOR A 1ST ENTRY, THIS IS THE SAME AS FOR 
*                     *TC.FWA*. 
* 
*           TC.SOX  = CURRENT SOURCE INPUT WORD, WITH NEXT CHARACTER TO 
*                     EXTRACT IN LEFT-MOST *CHAR* BITS (WHERE *CHAR* IS 
*                     THE BIT WIDTH OF A SINGLE CHARACTER, I.E. 6 FOR 
*                     DISPLAY CODE).  THAT IS, FIRST CHAR TO EXTRACT IS 
*                     IN BITS 59 THRU 54 (FOR DPC). 
* 
*                     NOTE THAT *TC.SOA* AND *TC.SOX* NEED NOT *MATCH*
*                     AS MIGHT BE THE CASE IF SOME USER PRE-TOK 
*                     PROCESSOR HAD, FOR EXAMPLE, STRIPPED OFF A LINE 
*                     LABEL/NUMBER. 
* 
*           TC.SOL  = TOTAL NUMBER OF **BITS** IN THIS SOURCE LINE TO 
*                     ENTOKEN.
* 
*           TC.SOB  = NR OF BITS **REMAINING** IN (TC.SOX).  I.E. FOR 
*                     A FULL WORD, (TC.SOB) = 60D.
* 
*           TC.SOC  = CURRENT SOURCE CHARACTER, ELSE .MI. IF NEED TO
*                     EXTRACT A CHARACTER FROM (TC.SOX) FIRST.
* 
*           TC.TBA  = SAME AS FOR *EASY* FORM.
* 
*           TC.TBB  = NR OF BITS REMAINING IN TOKEN PACKING REGISTER. 
* 
* 
*         IF YOU DECIDE THAT YOU WANT OR NEED TO USE THE *HARD* FORM, I 
*         SUGGEST A READING OF *COMCBUB* (BURST/BUILD CHARACTERS WITH 
*         BLANK SQUEEZE) AND *COMCBUN* (BURST/BUILD CHARACTERS WITH NO
*         BLANK SQUEEZE).  MOST OF THESE *TC.* CELLS ARE REALLY JUST
*         MATERIALIZATIONS OF *BUB/BUN* REGISTERS...
* 
************************************************************************
 TOFU     SPACE  4,10 
**        TOFU - *TOK* FUNCTIONAL UNIT. 
* 
* 
*         THIS MACRO DEFINES THE ENTRY POINT TO A *TOK* FUNCTIONAL
*         UNIT (TOFU).  EACH *TOFU* WITHIN *TOK* EXECUTES/INTERPRETS
*         A SINGLE BINARY *TOGEL* INSTRUCTION.
* 
*         NOTE - SEE FTN 5 IMS/DECK *LEX* FOR MORE INFORMATION. 
* 
* NAM     TOFU   RAD
* 
*         ENTRY  NAM = NAME OF THIS *TOFU*
*                RAD = RESTART ADDR.  ADDR TO TRANSFER CONTROL
*                      OF CPU TO WHEN RESTARTING TOKEN GENERATOR. 
* 
*         EXIT   NONE 
* 
*         USES   NONE 
* 
*         CALLS  NONE 
  
  
          PURGMAC TOFU
  
          MACRO  TOFU,NAM,RAD 
 +        EQ     RAD TOK=MN 
 NAM      BSS    0
* 
          IF     -DEF,QUAL$,3 
          QUAL   *
 NAM      BSS    0
          QUAL   COMCTOK
 TOFU     ENDM
 FW=TCOM  SPACE  4,10 
**        FW=TCOM - FWA OF *TOKCOM*.
* 
* 
*         *FW=TCOM* CONTAINS THE FWA OF THE CURRENT *USER/TOK*
*         COMMUNICATIONS AREA, *TOKCOM*.
  
  
 FW=TCOM  BSSZ   1
 TF=      SPACE  4,10 
**        TF=XXX - *TOK* MASTER CONTROL FLAGS.
* 
* 
*         SYMBOLS OF THE FORM *TF=XXX*, WHERE *TF=* IS A COMMON PREFIX
*         AND *XXX* IS A FLAG NAME, CONTAIN MISCELLANEOUS CONTROL FLAG
*         INFORMATION THAT IS USED DURING TOKEN GENERATION. 
* 
*         FLAGS, *XXX*, ARE --
* 
*         SQZ  = .NZ. IF LAST CHARACTER ACCESS WAS PERFORMED WITH 
*                     BLANK SQUEEZE (*COMCBUB* USED), ELSE
*              = .ZR. IF LAST ACCESS WAS NO BLANK SQUEEZE (COMCBUN).
* 
*                USED SO THAT *TOK* CAN KNOW WHETHER TO ADVANCE 
*                THE SOURCE LINE CHARACTER POINTER WITH BLANK SQUEEZE 
*                OR NOT.
  
  
 TF=SQZ   BSSZ   1
  
          IFNE   TEST,0,1 
 TF=IDP   BSSZ   1
 TOKCOM   SPACE  4,10 
**        TOKCOM - *TOK* COMMUNICATIONS AREA. 
* 
*         *TOKCOM* IS THE *USER/TOK* COMMUNICATIONS AREA.  INFORMATION
*         IS PASSED BACK AND FORTH BETWEEN *TOK* AND ITS CALLER 
*         VIA THE LOCATIONS IN *TOKCOM*.
* 
*         SEE *ENTRY/EXIT* CONDITIONS IN *COMCTOK* PREAMBLE.
* 
*         NOTE -- 
* 
*         CRITICAL ENTOKENING REGISTERS ARE SAVED HERE WHEN NECESSARY.
*         ONE CAN NOT, HOWEVER, EXPECT SIGHT-UNSEEN THAT THESE SAVED
*         REGISTERS ARE ALWAYS UP-TO-DATE.  *TOK* ONLY USES THIS
*         SAVED AREA WHEN ABSOLUTELY NECESSARY SO THAT ENTOKENING 
*         CAN PROCEED AS RAPIDLY AS POSSIBLE. 
 TOK      SPACE  4,10 
**        TOK - TOKEN GENERATOR.
* 
* 
*         THIS IS THE TOKEN GENERATOR.  IT CONSISTS OF A MAIN DRIVER, 
*         *TOK=MN*, SURROUNDED BY *TOK* FUNCTIONAL UNITS, *TOFU*S.
* 
*         *TOK=MN* EXTRACTS THE NEXT *TOGEL* BINARY INSTRUCTION TO
*         EXECUTE/INTERPRET FROM THE CURRENT *TOM* AND FARMS
*         IT OUT TO THE APPROPRIATE *TOFU*, WHERE THIS *TOFU* 
*         WILL DO ITS BIT TOWARD ENTOKENING THE INPUT SOURCE LINE.
* 
*         THIS PROCESS REPEATS ITSELF UNTIL A *TOFU* SENSES AN
*         END-OF-LINE (EOL) CONDITION ON THE INPUT SOURCE LINE. 
*         WHEN THIS OCCURS, THE EOL SENSING *TOFU* TRANSFERS
*         CONTROL TO *TOK=EOL* WHERE SOME FINAL PROCESSING OCCURS 
*         BEFORE *TOK* EXITS BACK TO ITS CALLER.
* 
* 
*         ENTRY  (B2) = FWA OF *TOKCOM*, THE *USER/TOK* COMMUNICATIONS
*                       AREA. 
* 
* 
*         EXIT   NONE 
* 
*         USES   ALL         (BELIEVE IT) 
* 
*         CALLS  BUB,BUN,CCD,RER,SER,TID(IN *TEST* MODE)
  
  
 TOK      SUBR               ** ENTRY/EXIT ** 
  
*         INITIALIZATION. 
  
          IF     -DEF,B1=1,1
          SB1    1
  
          SX6    B2+
          SA6    FW=TCOM     SAVE (B2) = FWA OF *TOKCOM*
  
          SA1    B2+TC.FWA
          ZR     X1,TOK2     IF CALLER HAS SET UP *TOKCOM* HIMSELF
  
*         HERE IF CALLER IS ASKING US TO SET UP SOURCE INPUT POINTERS.
* 
*         THIS MEANS THAT THE CALLER IS USING THE *FWA/FCP* AND 
*         *LWA/LCP* CALLING SEQUENCE.  WE NEED TO SET UP -- 
* 
*           1. *S0A* = SOURCE ADDR, I.E. NEXT ADDR TO GET A SOURCE WORD.
* 
*           2. *SOX* = CURRENT SOURCE WORD, I.E. THE CONTENTS OF THE
*                      LOCATION POINTED TO BY *SOA* SHIFTED 
*                      APPROPRIATELY SO THAT 1ST CHAR TO EXTRACT IS IN
*                      LEFT-MOST CHAR POSITION (BITS 59-54).
* 
*           3. *SOB* = SOURCE BIT, I.E. BIT POSITION WITHIN *SOX* OF
*                      NEXT CHAR TO EXTRACT.
* 
*           4. *SOL* = NR OF BITS IN SOURCE LINE TO ENTOKEN.
* 
*           5. *SOC* = CURRENT SOURCE CHAR FROM (SOX), I.E. SET 
*                      TO -1 TO INDICATE *NO AVAILABLE CHAR*. 
  
          =A2    A1+TC.FCP-TC.FWA  (X2) = 1ST CHAR POSITION 
          SB3    X2 
          SB4    60D/CHAR    (B4) = NR OF CHARS IN A WORD 
  
          LE     B3,B0,"BLOWUP"    IF BAD CHAR POSITION 
          GT     B3,B4,"BLOWUP"    IF BAD CHAR POSITION 
  
          SX3    CHAR        (X3) = BIT WIDTH OF 1 CHARACTER
          SX4    60D+CHAR 
          IX6    X3*X2       (X6) = 6 * NR OF CHARS (FOR DPC) 
          IX2    X4-X6       (X2) = 66 - (6 * NR OF CHARS), FOR DPC 
          BX6    X1          (X6) = *SOA* 
          LX7    X2          (X7) = *SOB* 
          SA6    B2+TC.SOA
          SA7    B2+TC.SOB
  
          SX6    60D
          SA5    X1          (X5) = 1ST WORD TO ENTOKEN 
          IX7    X6-X2
          SB5    X7          (B5) = 60D - SOB 
          LX6    X5,B5       (X6) = 1ST WORD TO ENTOKEN W/ 1ST CHAR TO
*                                   ENTOKEN IN LEFT-MOST CHAR POSITION
          SA6    B2+TC.SOX
  
*         COMPUTE *SOL*, AND SET *SOC*. 
  
          SA3    B2+TC.LWA         (X3) = LWA 
          =A4    A3+TC.LCP-TC.LWA  (X4) = LCP, LAST CHAR POSITION 
          SB3    X4 
  
          LE     B3,B0,"BLOWUP"    IF BAD CHAR POSITION 
          GT     B3,B4,"BLOWUP"    IF BAD CHAR POSITION 
  
          SB3    B2          SAVE (B2)
          RJ     CCD         COMPUTE CHARACTER DIFFERENCE 
          MI     X7,"BLOWUP" IF LAST CHAR IS TO THE LEFT OF 1ST CHAR
          MX6    -1 
          SB2    B3          RESTORE (B2) 
          SA6    B2+TC.SOC   SET TO *NO AVAILABLE CHAR IN (X4)* 
          SA7    B2+TC.SOL
  
*         RESTORE ENTOKENING REGISTERS. 
  
 TOK2     BSS    0
          RJ     RER         RESTORE ENTOKENING REGISTERS 
          SA2    B2+TC.TBA
          SA3    B2+TC.RSF
          =X6    "O."BOS
          NZ     X3,TOK=RS   IF THIS IS A RESTART 
          SA6    X2          INITIALIZE TOKEN BUFFER W/ *BOS* TOKEN 
 TOK=MN   SPACE  4,10 
**        MAIN LOOP NODE. 
* 
*         HERE TO EXTRACT NEXT *TOGEL* BINARY INSTRUCTION FROM *TOM*
*         AND TRANSFER CONTROL TO APPROPRIATE *TOFU*. 
* 
*         FIRST, HOWEVER, NEED TO ASSURE THAT (X4) CONTAINS 
*         A USEABLE CHARACTER.
* 
*         IF IN *TEST* MODE --
* 
*           1. SAVE CRITICAL ENTOKENING REGISTERS IN THEIR APPROPRIATE
*              *TOKCOM* LOCATION TO PROVIDE AN AUDIT TRAIL. 
* 
*           2. CALL *TID* (TOK INTERACTIVE DEBUG INTERFACE, 
*              IN COMDECK *COMDTOK*) IF *TF=IDP* HAS BEEN SET.
  
  
 TOK=MN   BSS    0           ** MAIN LOOP NODE ** 
          MX0    -CHAR
          PL     X4,TOK4     IF (X4) ALREADY CONTAINS A USEABLE CHAR
          BX1    X1-X1       (X1) = CHAR SHIFT MASK (NO CHARS IN GROUP) 
          SA2    TF=SQZ 
          MX6    0           CLEAR PACKING REGISTER 
          NZ     X2,TOK3     IF BU/BU WITH BLANK SQUEEZE
          SB3    BN.TOCP+BN.TOCL-CHAR 
          RJ     =XBUN       BURST/BUILD WITH NO BLANK SQUEEZE
          GT     B7,B0,TOK4  IF NO EOL ENCOUNTERED
          EQ     TOK=EOL
  
 TOK3     SB3    BB.TOCP+BB.TOCL-CHAR 
          RJ     =XBUB       BURST/BUILD WITH BLANK SQUEEZE 
          GT     B7,B0,TOK4  IF NO EOL ENCOUNTERED
          EQ     TOK=EOL
  
 TOK4     SA2    A0          (X2) = NEXT TOGEL BINARY INST TO XEQ 
          SA0    A0+B1
  
 .T       IFNE   TEST,0 
          RJ     SER         SAVE ENTOKENING REGISTERS
          SA1    TF=IDP 
          ZR     X1,TOK5     IF NO IDP INTERFACE SELECTED 
  
          SB2    =YTOK=IDP
          MI     B2,TOK5     IF USER DID NOT PROVIDE *TOK=IDP*
          RJ     =YTOK=IDP   *TOK* INTERACTIVE DEBUG INTERFACE
  
 TOK5     BSS    0
 .T       ENDIF 
  
          LX2    0-TG.TOFUP 
          ERRNZ  TG.TOFUL-18
          SB2    X2          (B2) = TOFU ADDR FOR THIS INSTRUCTION
          JP     FW.TOK+B2
 TOK=RS   SPACE  4,10 
**        TOK=RS - *TOK* RESTART NODE.
* 
*         HERE IF CALLER SPECIFIED *RESTART*, I.E. (TC=RSF) .NZ., 
*         AS WOULD COMMONLY OCCUR WHEN A HOST WAS USING *TOK* TO
*         ENTOKEN MULTIPLE SOURCE LINES AS A SINGLE SOURCE STMT.
* 
*         IN THIS CASE, TOKEN GENERATION IS INTERRUPTED AT THE
*         END OF EACH SOURCE LINE AND NEEDS TO BE RESUMED/RESTARTED 
*         FOR EACH NEW LINE THAT IS FED TO *TOK*. 
* 
*         CONSIDER -- 
* 
*           COL 1    6
*                     CRAWLED = OUT OF * A
*                    + WELL 
* 
*         LINE 1 OF THIS STMT ENDS WITH THE TOKEN -A-.  WHEN LINE 2 
*         IS ACQUIRED, *TOK* HAS TO BE RESTARTED SO THAT THE
*         TOKEN -AWELL- CAN BE GENERATED. 
  
  
 TOK=RS   BSS    0
  
*         SET UP (A6).  NEXT ADDR TO STORE TOKEN IS AT (A6)+1.
  
 +        SA1    X2-1 
          BX6    X1 
          SA6    A1          RESTORE (A6) 
  
*         RESTORE *TOGEL* BINARY INSTRUCTION WE WERE
*         EXECUTING WHEN INTERRUPTED. 
  
          SA1    B2+TC.REST 
          SA2    A0-B1
          LX2    0-TG.TOFUP 
          MX0    -CHAR
          SB4    X2-1        (B4) = TOFU RESTART ADDR 
          ZR     X1,TOK5A    IF NO *ON RESTART* PROCESSING
          SB2    FW.TOK+B4   (B2) = ADDR OF *TOK* RESTART NODE
          SB4    FW.TOK+X1   (B4) = ADDR OF *RESTART* PROCESSOR 
  
 TOK5A    JP     B4+         BANG...
 TOK=CAL  SPACE  4,10 
**        HERE IF *CALT XX*.
  
  
 TOK=CAL  TOFU
          LX2    0-TG.CADP+TG.TOFUP-0 
          ERRNZ  TG.CADL-18 
          SB2    X2          (B2) = *CALT* ADDR 
          JP     FW.TOK+B2
 TOK=COF  SPACE  4,10 
**        HERE IF *CASEOF (...)*. 
* 
*         READY TO PROCESS AND ENTOKEN THE CHARACTER IN (X4) WHICH
*         TERMINATED THE LAST BURST/BUILD.
* 
*         FIRST, THE CHARACTER IS USED AS AN ORDINAL INTO THE CHARACTER 
*         MAP *TOKEN* TABLE WHICH CONTAINS A TOKEN SKELETON FOR THIS
*         CHARACTER.  IN THE SIMPLEST CASE, THIS TOKEN SKELETON MERELY
*         GETS COPIED TO THE TOKEN BUFFER, AND CONTROL PASSES BACK TO 
*         *TOK=MN*. 
* 
*         THIS TOKEN GENERATOR ALSO HAS THE ABILITY TO MAP MULTIPLE 
*         TOKENS BACK TO A NEW SINGLE TOKEN.  THIS PROCESS CAN GREATLY
*         SIMPLIFY THE PARSING OF MULTIPLE CHARACTER OPERATORS. 
* 
*         FOR EXAMPLE, THE FORTRAN EXPONENTIATION OPERATOR ** CAN BE
*         LOOKED AT AS A * TOKEN FOLLOWED BY A * TOKEN.  HOWEVER, 
*         THE PARSER CAN BE MORE STRAIGHT-FORWARD IF A ** TOKEN IS
*         GENERATED INSTEAD.
* 
*         THE TECHNIQUE USED HERE IS TO LOOK BACK AT THE TOKEN PRECEDING
*         THE ONE BEING GENERATED.  FOR THE ** CASE, WHEN A * TOKEN IS
*         GENERATED, THE TOKEN PRECEDING THE * IS EXAMINED;  IF FOUND 
*         TO BE A * TOKEN, A ** TOKEN REPLACES THEM.
* 
*         THIS ENTIRE PROCESS IS TABLE DRIVEN, WHICH ALLOWS THE 
*         *COMCTOK* USER TO DEFINE HER OWN OPERATORS WITHOUT A LOT
*         OF GRIEF  (SEE *COMATOK*).
* 
*         THE SYNTAX TABLE IS A BINARY TREE WHERE *NO MATCH* MOVEMENT 
*         IS LEFT-TO-RIGHT, AND *MATCH* MOVEMENT IS TOP-TO-BOTTOM.
* 
*         FOR EXAMPLE, ARBITRARILY CONSIDER THE 2 OPERATORS 
*                /*$  AND  #:$
* 
*                TOKEN
*                     +---+---+---+ 
*                     + T + 0 + $ + 
*                     +---+---+---+ 
*                       . 
*                TLINK  .   *$                 :$ 
*                     +---+---+---+      +---+---+---+
*                     + T + F + * +  ..> + T + 0 + : +
*                     +---+---+---+  .   +---+---+---+
*                       .   .        .     .
*                       .   ..........     .
*                       .                  .
*                       .  /*$             .  #:$ 
*                     +---+---+---+      +---+---+---+
*                     + 0 + 0 + / +      + 0 + 0 + # +
*                     +---+---+---+      +---+---+---+
* 
*                WHERE EACH RECTANGLE CONTAINS 3 SQUARES -- 
* 
*                  SQUARE 1  CONTAINS THE *MATCH* LINK.  I.E. IF THE
*                            TOKEN WE ARE LOOKING AT IS PRECEDED BY 
*                            THE TOKEN IN SQUARE 3, GO TO WHERE SQUARE 1
*                            POINTS.  (0 DENOTES END-OF-CHAIN)
*                  SQUARE 2  CONTAINS THE *NO MATCH* LINK.  I.E. IF THE 
*                            TOKEN WE ARE LOOKING AT IS NOT PRECEDED BY 
*                            THE TOKEN IN SQUARE 3, GO TO WHERE SQUARE 2
*                            POINTS.
*                  SQUARE 3  CONTAINS THE TOKEN VALUE TO TRY AND MATCH. 
  
  
 TOK=COF  TOFU   TOK=MN 
  
*         CHECK TO SEE IF CHARACTER TO GENERATE TOKEN FOR IS
*         IN *CASEOF* RANGE.  IF SO, USE CHAR AS AN ORDINAL INTO
*         THE *TOKEN* TABLE TO PICK UP TOKEN SKELETON FOR THIS CHAR.
*         IF NOT, PICK UP /ELSC/ *TOKEN* ENTRY. 
* 
*         IF THE CHARACTER IS WITHIN *CASEOF* RANGE, THEN CHAR IS 
*         CONVERTED TO A *ROTATED CHARACTER SET*, CALLED *RCS*, 
*         WHERE THE 1ST CHARACTER IN *RCS* IS THE 1ST CHARACTER 
*         IN *CASEOF* RANGE, *COA*, (BINARY VALUE = 00B).  THE
*         FORMULA FOR CONVERTING CHAR TO ENTOKEN, *CTE*, TO *RCS* 
*         IS -- 
* 
*                RCS = CTE+10000B-COA 
* 
*         IF THE HOST CHARACTER SET CONSISTS OF *CHAR* BIT CHARACTERS,
*         THEN THE ROTATED CHARACTER SET VALUE IS THE LOWER *CHAR*
*         BITS OF *RCS* ABOVE.
* 
*         FOR EXAMPLE, CONSIDER --
* 
*                CASEOF (+..:)
* 
*         THE *RCS* VALUES FOR THIS *CASEOF* RANGE (USING DISPLAY 
*         CODE, 6 BIT, CHARACTERS) ARE -- 
* 
*                +      RCS = 45B+10000B-45B
*                           = 10000B
* 
*                       EXTRACT LOWER 6 BITS --  RCS = 00B
* 
*                -      RCS = 46B+10000B-45B
*                           = 10001B
* 
*                       EXTRACT LOWER 6 BITS --  RCS = 01B
* 
*                SKIP TO
* 
*                ;      RCS = 77B+10000B-45B
*                           = 10032B
* 
*                       EXTRACT LOWER 6 BITS --  RCS = 32B
* 
*                :      RCS = 00B+10000B-45B
*                           = 7733B 
* 
*                       EXTRACT LOWER 6 BITS --  RCS = 33B
* 
*         IT IS EASY TO SEE NOW THAT THE *RCS* VALUES ARE ALSO
*         *TOKEN* ORDINALS. 
  
  
          LX2    0-TG.COAP+TG.TOFUP-0 
          BX6    -X0*X2      EXTRACT *COA*
          LX2    0-TG.COZP+TG.COAP-0
          BX7    -X0*X2      EXTRACT *COZ*
          SX1    X4+10000B
          LX2    0-TG.TADP+TG.COZP-0
          IX3    X1-X6
          ERRNZ  TG.TADL-18 
          =B3    FW.TOK+X2   (B3) = FWA OF *TOKEN* CHAR MAP 
*                                   FOR THIS *CASEOF* 
          BX1    -X0*X3      (X1) = *RCS* REPRESENTATION OF CHAR
*                                   TO GENERATE TOKEN FOR 
          IX6    X7-X1
          SB2    X1          (B2) = ORD OF *TOKEN* SKEL FOR THIS CHAR 
          SB5    A6          INITIALIZE (B5) = ADDR OF PRECEDING TOKEN
          PL     X6,TOK6     IF CHAR IS WITHIN *CASEOF* RANGE 
          SB2    X7+B1       (B2) = ORD OF /ELSC/ *TOKEN* SKEL
  
 TOK6     SA1    B3+B2       (X1) = *TOKEN* SKEL FOR THIS CHAR
  
*         INITIALIZE FOR *TOKEN/TLINK* SYNTAX CHECKING. 
  
          ERRNZ  TK.TOTP-0
          ERRNZ  TK.TOTL-18 
          ERRNZ  TK.PADL-18 
          ERRNZ  TK.LNKL-18 
          SB5    A6          INITIALIZE (B5) = ADDR OF PRECEDING TOKEN
          SX6    X1 
          LX1    0-TK.PADP
          SX0    X1          (X0) = ADDR OF USER PROCESSING ADDR FOR
*                                   THIS TOKEN, ELSE .ZR. IF NONE 
          LX1    0-TK.LNKP+TK.PADP-0
          NO
          =B4    FW.TOK+X1   (B4) = ADDR OF 1ST *TLINK* ENTRY, ELSE 
*                                   .ZR. IF NONE
 TOK=COF  SPACE  4,10 
**        THIS IS THE MAIN LOOP FOR MOVEMENT THRU *TLINK*, THE
*         MULTIPLE CHARACTER OPERATOR SYNTAX TREE --
* 
*           1. IF HAVE JUST FOUND A *TRUE* TOKEN PAIR, THEN WE WOULD
*              HAVE MOVED *DOWN* THE TREE  (SEE DIAGRAM). 
* 
*           2. IF HAVE JUST ENCOUNTERED A *FALSE* TOKEN PAIR, THEN
*              WE WOULD HAVE MOVED *RIGHT* IN THE TREE. 
* 
*         ENTRY  (B4) = ADDR OF NEXT *TLINK* ENTRY  (EITHER *DOWN* OR 
*                       *RIGHT* OF CURRENT ENTRY), ELSE 
*                     = .ZR. IF CURRENT ENTRY WAS END-OF-CHAIN
*                (B5) = ADDR OF NEXT TOKEN TO CHECK 
* 
*         LOCKED X - 0,4,5,6
*                A - 0,4,5,6
*                B - 6,7
  
          ERRNZ  TL.TRUL-18 
          ERRNZ  TL.FALL-18 
          ERRNZ  TL.PREL-18 
  
 TOK7     ZR     B4,TOK8A    IF END-OF-CHAIN ENCOUNTERED
          SA1    B4          (A1,X1) = A+C OF 1ST WORD OF TLINK ENTRY 
          SA2    B5          (A2,X2) = A+C OF PRECEDING TOKEN 
          LX1    0-TL.PREP
          SB2    X1          (B2) = O.PRE 
          SB3    X2          (B3) = O.XXX 
          LX1    0-TL.FALP
          =B4    FW.TOK+X1   (B4) = ADDR OF NEXT TLINK ENTRY IN 
*                                   *FALSE* CHAIN 
          NE     B2,B3,TOK7  IF *O.XXX* NOT PRECEDED BY *O.PRE* 
          LX1    0-TL.TRUP+TL.FALP-0
          =B3    FW.TOK+X1   (B3) = ADDR OF NEXT TLINK ENTRY IN 
*                                   *TRUE* CHAIN
  
*         HERE IF <O.PRE><O.XXX> OCCURRED.
  
          LX1    59-TL.MAFP+TL.TRUP-0 
          PL     X1,TOK8     IF EXACT CHARACTER MATCH NOT NEEDED
  
*         HERE IF AN EXACT CHARACTER MATCH REQUIRED.
  
          SA1    A1+B1       (X1) = CHARS TO CHECK FOR, -L- FMT 
          MX7    TL.MACL
          ERRNZ  TL.MACP+TL.MACL-60D
          BX2    X7*X2       (X2) = CHARS TO CHECK, -L- FMT 
          NO
          BX7    X1-X2
          NZ     X7,TOK7     IF NO MATCH
  
 TOK8     SB4    B3 
          SB5    B5-B1
          NZ     B3,TOK7     IF NEED TO TRY NEXT *TRUE* TLINK ENTRY 
          SA1    A1+B1       (X1) = REPLACEMENT TOKEN 
          SX6    X1 
          LX1    0-TK.PADP
          SX0    X1 
          LX1    0-TK.LNKP+TK.PADP-0
          =B4    FW.TOK+X1
          EQ     TOK9 
  
*         HERE WHEN END-OF-CHAIN ENCOUNTERED -- TIME TO INVOKE
*         ANY SPECIAL PROCESSING THAT MIGHT BE REQUIRED FOR TOKEN 
*         THAT IS TO BE GENERATED.
  
 TOK8A    SB5    A6+         RESET (B5) = ADDR WE ORIGINALLY BEGAN AT 
  
 TOK9     SB3    FW.TOK+X0   (B3) = ADDR OF SPECIAL PROCESSOR, ELSE 
*                                   .ZR. IF NONE
          ZR     B3,TOK10    IF NO SPECIAL PROCESSOR
          MI     B3,TOK9A    IF MERELY XFERRING WITHIN *TOM*
          JP     B3+
  
 TOK9A    SA0    -B3         RESET -P- REGISTER 
  
*         RETURN FROM SPECIAL PROCESSING -- ENTER NEW TOKEN 
*         INTO TOKEN BUFFER.
  
 TOK=COS  BSS    0           ** RETURN WITH TOKEN STORE **
 TOK10    SA6    B5+1 
  
 TOK=CON  BSS    0           ** RETURN WITH NO TOKEN STORE ** 
 TOK11    NZ     B4,TOK7     IF NEED TO CHECK LINK FOR NEW TOKEN
          SX4    -1          SET TO *NO USEABLE CHAR IN (X4)* 
          EQ     TOK=MN 
 TOK=ELS  SPACE  4,10 
**        HERE IF *ELST*. 
* 
*         NOTE: *TEST* MODE ONLY. 
  
 .T       IFNE   TEST,0 
 TOK=ELS  TOFU
          EQ     TOK=MN 
 .T       ENDIF 
 TOK=ENT  SPACE  4,10 
**        HERE IF *ENDT*. 
* 
*         NOTE: *TEST* MODE ONLY. 
  
 .T       IFNE   TEST,0 
 TOK=ENT  TOFU
          EQ     TOK=MN 
 .T       ENDIF 
 TOK=EOL  SPACE  4,10 
**        HERE WHEN EOL ENCOUNTERED.
* 
*           1. SAVE CRITICAL ENTOKENING REGISTERS SO THAT WE CAN
*              RECOVER IF NEXT LINE IS CONTINUATION OF THIS LINE. 
* 
*           2. IF *ON EOL,XXX* INDICATED THAT CALLER WISHES 
*              TO PERFORM SOME END-OF-LINE PROCESSING, THEN GO DO IT. 
  
 TOK=EOL  BSS    0           ** ENTRY **
          SX6    "O."EOS
          MX4    -1          SET TO *NO USEABLE CHAR IN (X4)* 
          SA6    A6+B1       MARK EOS 
          RJ     SER         SAVE ENTOKENING REGISTERS
          SA1    B2+TC.EOL
          ZR     X1,EXIT.    IF NO SPECIAL PROCESSING 
          SB2    X1 
          JP     B2          DO WHAT YOU WILL...
 TOK=GEN  SPACE  4,10 
**        HERE IF *GENTOK TOT*. 
  
  
 TOK=GEN  TOFU
          LX2    0-TG.GTTP+TG.TOFUP-0 
          ERRNZ  TG.GTTL-18 
          SX6    X2          (X6) = TOKEN TYPE TO GENERATE
          MX4    -1          SET TO *NO USEABLE CHAR IN (X4)* 
          SA6    A6+B1
          EQ     TOK=MN 
 TOK=GN   SPACE  4,10 
**        HERE IF *GROUP,...,NSQZ*. 
  
 TOK=GN   TOFU   TOK13
          LX2    0-TG.TOTP+TG.TOFUP-0 
          ERRNZ  TG.TOTL-18 
          SX6    X2          (X6) = 42/0,18/TOKEN TYPE FOR THIS *GROUP* 
          SB3    BN.TOCP+BN.TOCL-CHAR 
  
 TOK12    LX2    0-TG.MXAP+TG.TOTP-0
          SA1    FW.TOK+X2   (X1) = CHAR SHIFT MASK FOR THIS *GROUP*
          BX7    X7-X7
          SA7    TF=SQZ      SET TO *BU/BU WITH NO BLANK SQZ* 
          RJ     BUN         BURST/BUILD CHARACTERS WITH NO BLANK SQZ 
          GT     B7,B0,TOK=MN IF DID NOT ENCOUNTER END-OF-LINE
          EQ     TOK=EOL
  
**        HERE FOR *GROUP,...,NSQZ* RESTART.
  
 TOK13    SB2    BN.TOCP+BN.TOCL-CHAR 
          EQ     B3,B2,TOK=GN IF LAST TOKEN GENERATED WAS FULL
 +        SA1    A6-B1
          BX7    X6          SAVE (X6)
          LX6    X1 
          SA6    A1          (A6+1) = ADDR TO (RE)STORE CURRENT 
*                                     UNFINISHED TOKEN
          LX2    0-TG.TOTP+TG.TOFUP-0 
          BX6    X7          RESTORE (X6) = PARTIALLY FILLED TOKEN
          EQ     TOK12
 TOK=GO   SPACE  4,10 
**        HERE IF *GOTO XX*.
  
 TOK=GO   TOFU
          LX2    0-TG.GOTP+TG.TOFUP-0 
          ERRNZ  TG.GOTL-18 
          SA0    FW.TOK+X2
          EQ     TOK=MN 
 TOK=GS   SPACE  4,10 
**        HERE IF *GROUP,...,SQZ*.
  
 TOK=GS   TOFU   TOK15
          LX2    0-TG.TOTP+TG.TOFUP-0 
          ERRNZ  TG.TOTL-18 
          SX6    X2          (X6) = 42/0,18/TOKEN TYPE FOR THIS *GROUP* 
          SB3    BB.TOCP+BB.TOCL-CHAR 
  
 TOK14    LX2    0-TG.MXAP+TG.TOTP-0
          ERRNZ  TG.MXAL-18 
          SA1    FW.TOK+X2   (X1) = CHAR SHIFT MASK FOR THIS *GROUP*
          SX7    B1 
          SA7    TF=SQZ      SET TO *BU/BU WITH BLANK SQZ*
          RJ     BUB         BURST/BUILD WITH BLANK SQUEEZE 
          GT     B7,B0,TOK=MN  IF DID NOT ENCOUNTER END-OF-LINE 
          EQ     TOK=EOL
  
*         HERE FOR *GROUP,...,SQZ* RESTART. 
  
 TOK15    SB2    BB.TOCP+BB.TOCL-CHAR 
          EQ     B3,B2,TOK=GS IF LAST TOKEN GENERATED WAS FULL
 +        SA1    A6-B1
          BX7    X6          SAVE (X6)
          LX6    X1 
          SA6    A1          (A6+1) = ADDR TO (RE)STORE CURRENT 
*                                     UNFINISHED TOKEN
          LX2    0-TG.TOTP+TG.TOFUP-0 
          BX6    X7          RESTORE (X6) = PARTIALLY FILLED TOKEN
          EQ     TOK14
 TOK=IFT  SPACE  4,10 
**        HERE IF *IFT (...)*.
  
 TOK=IFT  TOFU
          LX2    0-TG.MXAP+TG.TOFUP-0 
          ERRNZ  TG.MXAL-18 
          =A1    FW.TOK+X2   (X1) = CHAR SHIFT MASK FOR THIS *IFT*
          SB2    X4+10000B-BICH 
          LX2    0-TG.FALP+TG.MXAP-0
          LX7    X1,B2
          MI     X7,TOK=MN   IF CHARACTER IS IN THIS GROUP (TRUE)...
          ERRNZ  TG.FALL-18 
          SA0    FW.TOK+X2   (A0) = ADDR OF *FALSE* BINARY TOGEL INST 
          EQ     TOK=MN 
 TOK=ON   SPACE  4,10 
**        HERE IF *ON I,V*. 
  
  
 TOK=ON   TOFU
          SA1    FW=TCOM     (X1) = FWA OF *TOKCOM* 
          LX2    0-TG.INVP+TG.TOFUP-0 
          ERRNZ  TG.INVL-18 
          SX7    X2          (X7) = INTERUPT VALUE
          SB2    X1 
          LX2    0-TG.INCP+TG.INVP-0
          ERRNZ  TG.INCL-18 
          SA7    B2+X2       INTERUPT CELL = INTERUPT VALUE 
          EQ     TOK=MN 
 TOK=XEQ  SPACE  4,10 
**        HERE IF *XEQ P*.
  
 TOK=XEQ  TOFU
          LX2    0-TG.XADP+TG.TOFUP-0 
          ERRNZ  TG.XADL-18 
          SA1    XEQA        (X1) = *GOTO* SKELETON 
          SX7    A0          (X7) = RETURN ADDR 
          =A0    FW.TOK+X2+1 (A0) = ADDR OF 1ST INST TO XEQ IN PROC 
          LX7    TG.GOTP-0
          BX7    X1+X7       (X7) = FABRICATED /GOTO *+1/ 
          SA7    A0-B1       PLUG /GOTO *+1/  (JUST LIKE SEYMOUR...)
          EQ     TOK=MN 
  
  
 XEQA     GOTO   0           SKELETON FOR /GOTO *+1/ PLUG 
 CCD      SPACE  4,10 
**        CCD - COMPUTE CHARACTER DIFFERENCE. 
* 
* 
*         GIVEN 2 CHARACTER POSITIONS WITHIN A SOURCE LINE, *CCD* 
*         WILL RETURN THE NR OF CHARACTERS THAT LIE BETWEEN THEM. 
* 
*         EACH CHARACTER POSITION IS REPRESENTED IN *TOK*S WORD+BIT 
*         NOTATION. 
* 
*         E.G. GIVEN THE FOLLOWING -- 
* 
*         WORD  0         1          2           3
*               ......ALPHA.=.BETA.+.1000........0000000000 
*                           .        .
*                           .        ......... W2 = 2 
*                           .                  C2 = 60D 
*                           . 
*                           .................. W1 = 1 
*                                              C1 = 48D 
* 
*         WHERE  W1 = WORD POSITION OF 1ST CHAR.
*                C1 = CHAR/BIT POSITION WITHIN *W1* OF 1ST CHAR.
*                W2 = WORD POSTIONN OF 2ND CHAR.
*                C2 = CHAR/BIT POSTIION WITHIN *W2* OF 2ND CHAR.
* 
*         NOTE THAT *C1* AND *C2* ARE *TOK* BIT POSITIONS.  THAT IS,
*         THEY SPECIFY THE NR OF BITS **REMAINING** IN THEIR RESPECTIVE 
*         WORD.  THEREFORE, *C.N* = 60D MEANS LEFT-MOST CHAR IN *W.N*,
*         *C.N* = 54D MEANS 2ND TO THE LEFT-MOST CHAR IN *W.N*, ETC.
* 
*         SPECIAL NOTE ABOUT *C.N* = 0.  IF *C.N* = 0 AND *W.N* = X,
*         THEN *CCD* TREATS THIS CASE AS EQUIVALENT TO *C.N* = 60D AND
*         *W.N* = X+1.
* 
*         THE FORMULA FOR COMPUTING THE DIFFERENCE BETWEEN (W1,C1) AND
*         (W2,C2) IS -- 
* 
*                DIFF = ((W2-W1)*(60/CHAR))+(((60-C2)-(60-C1))/CHAR)
* 
*         THE CODE TO DO THIS WAS STOLEN FROM THE *FTN* OBJECT CODE 
*         FOR THE ABOVE *FORTRAN* STATEMENT.
* 
*         ENTRY  (X1) = W1
*                (X2) = C1
*                (X3) = W2
*                (X4) = C2
* 
*         EXIT   (X7) = NR OF CHARS BETWEEN (W1,C1) AND (W2,C2).
*                       NOTE THAT IF (W2,C2) IS TO THE RIGHT OF (W1,C1),
*                       THEN (X7) IS .PL., WHILE IF (W2,C2) IS TO THE 
*                       LEFT OF (W1,C1), THEN (X7) IS .MI.
* 
*         USES   X - 1,2,3,4,6,7
*                A - NONE 
*                B - 2
* 
*         CALLS  NONE 
  
  
 CCD      SUBR               ** ENTRY/EXIT ** 
          SX7    60D         (X7) = NR OF BITS IN A WORD
          IX6    X3-X1       (X6) = W2-W1 
          IX2    X7-X2       (X2) = 60-C1 
          IX4    X7-X4       (X4) = 60-C2 
          SX7    CHAR 
          IX2    X4-X2       (X2) = (60-C2)-(60-C1) 
          PX7    X7 
          PX2    X2 
          SX4    60D/CHAR    (X4) = NR OF CHARS IN A WORD 
          NX7    X7 
          NX2    X2 
          IX4    X6*X4       (X4) = (W2-W1)*(60/CHAR) 
          FX2    X2/X7       (X2) = ((60-C2)-(60-C1))/CHAR
          UX3    X2,B2
          LX2    X3,B2
          IX7    X4+X2
          EQ     EXIT.
 RER      SPACE  4,10 
**        RER - RESTORE ENTOKENING REGISTERS. 
* 
* 
*         ENTRY  ENTOKENING REGISTERS IN *TOKCOM*.
* 
*         EXIT   (B2) = FWA OF *TOKCOM*.
*                ENTOKENING REGISTERS SET UP, **EXCEPT** (A6,X6). 
*                (A6,X6) ARE **NOT** RESTORED BECAUSE I COULD FIND
*                NO GENERAL ALGORITHM/RULE FOR KNOWING WHETHER (A6) 
*                SHOULD POINT TO LAST TOKEN GENERATED OR NEXT TOKEN 
*                TO GENERATE.  ERGO (A6) IS LEFT ALONE... 
* 
*         USES   X - 1,2,3,4,5
*                A - 0,1,2,3,4,5
*                B - 2,3,6,7
* 
*         CALLS  NONE 
  
  
 RER      SUBR               ** ENTRY/EXIT ** 
          SA1    FW=TCOM
          SB2    X1          (B2) = FWA OF *TOKCOM* 
          =A1    B2+TC.P
          =A2    A1+TC.SOA-TC.P 
          =A3    A2+TC.SOX-TC.SOA 
          =A4    A3+TC.SOL-TC.SOX 
          SA0    X1 
          SA5    X2 
          BX5    X3 
          SB7    X4 
  
          =A1    A4+TC.SOB-TC.SOL 
          =A2    A1+TC.TBB-TC.SOB 
          =A4    A2+TC.SOC-TC.TBB 
          SB6    X1 
          SB3    X2 
          EQ     EXIT.
 SER      SPACE  4,10 
**        SER - SAVE ENTOKENING REGISTERS.
* 
* 
*         ENTRY  ENTOKENING REGISTERS INTACT. 
* 
*         EXIT   (B2) = FWA OF *TOKCOM*.
*                ENTOKENING REGISTERS STILL INTACT AND SAVED
*                IN *TOKCOM*. 
* 
*         USES   X - 1,7
*                A - 1,7
*                B - 2
* 
*         CALLS  NONE 
  
  
 SER      SUBR               ** ENTRY/EXIT ** 
          SA1    FW=TCOM
          SB2    X1          (B2) = FWA OF *TOKCOM* 
          SX7    A0-B0
          =A7    B2+TC.P
          SX7    A5-B0
          =A7    A7+TC.SOA-TC.P 
          BX7    X5 
          =A7    A7+TC.SOX-TC.SOA 
          SX7    B7-B0
          =A7    A7+TC.SOL-TC.SOX 
          SX7    B6-B0
          =A7    A7+TC.SOB-TC.SOL 
          BX7    X4 
          =A7    A7+TC.SOC-TC.SOB 
          SX7    A6-B0
          =A7    A7+TC.TBA-TC.SOC 
          SX7    B3-B0
          =A7    A7+TC.TBB-TC.TBA 
          EQ     EXIT.
 TOK      SPACE  4,10 
 QUAL$    IF     -DEF,QUAL$ 
          QUAL   *
 TOK      =      /COMCTOK/TOK 
 CCD      =      /COMCTOK/CCD 
 RER      =      /COMCTOK/RER 
 SER      =      /COMCTOK/SER 
 TF=SQZ   =      /COMCTOK/TF=SQZ
  
          IFNE   TEST,0,1 
 TF=IDP   =      /COMCTOK/TF=IDP
 TOK=MN   =      /COMCTOK/TOK=MN
 TOK=COS  =      /COMCTOK/TOK=COS 
 TOK=CON  =      /COMCTOK/TOK=CON 
 QUAL$    ENDIF 
 TOK      SPACE  4,10 
 COMCTOK  ENDX
