GlideinWMS The Glidein-based Workflow Management System

Description

This page documents a recipe for enabling and managing SciTokens Authorization between Glidein Entry Points and the Compute Elements they submit to.

Requirements

Requirement Description
Compatible HTCondor and GlideinWMS Versions Version numbers and installation instructions
An HTCondor Compute Element (htcondor-ce) Compatible with SciTokens Minimal version requirements at the time of this writing
hhtcondor-ce.noarch                 4.2.0-1.el7            @htcondor-development
htcondor-ce-client.noarch           4.2.0-1.el7            @htcondor-development
htcondor-ce-condor.noarch           4.2.0-1.el7            @htcondor-development
A SciToken that the htcondor-ce will accept An example method for generating a working token is given below. This is not how SciTokens would be generated and used in a production environment.

Obtain a SciToken

The following is an example of how to configure a website 'jobsub.fnal.gov' into a SciToken issuing and authenticating site. NB anyone that obtains the private keys used in this example could then use the website to issue valid SciTokens. These are NOT instructions on how to do this in a production environment.

1.  yum install python2-scitokens or pip install scitokens


2. Generate some keys

scitokens-admin-create-key --create-keys \
                           --pem-private > scitoken.private.pem

scitokens-admin-create-key --private-keyfile scitoken.private.pem\
                           --jwks-private > test.scitoken.private.jwks

scitokens-admin-create-key --private-keyfile test.scitoken.private.pem \
                           --jwks-public > test.scitoken.public.jwks

3. copy keys over to jobsub.fnal.gov, create some other files in the document root directory:
ssh fnalu.fnal.gov
cd /web/sites/j/jobsub.fnal.gov/htdocs
mkdir -p  oauth2/certs  .well-known
# create file  .well-known/openid-configuration with these contents:
cat .well-known/openid-configuration

{
 "issuer":"https://jobsub.fnal.gov",
 "jwks_uri":"https://jobsub.fnal.gov/oauth2/certs/test.scitoken.public.jwks"
}

exit #return to original machine that you generated the keys on

scp test.scitoken.public.jwks fnalu:/web/sites/j/jobsub.fnal.gov/htdocs/oauth2/certs



3. Generate a token to perform condor operations on  an htcondor-ce

#for ce fermicloud349.fnal.gov
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem \
                             --issuer https://jobsub.fnal.gov sub="condor"\
                             scope="condor"\
                             aud="fermicloud349.fnal.gov:9619"  > fermicloud349.scitoken

#for ce itb-ce2.chtc.wisc.edu
scitokens-admin-create-token --key_id 7b50 --keyfile test.scitoken.private.pem\
                             --issuer https://jobsub.fnal.gov sub="condor"\
                             scope="condor"\
                             aud="itb-ce2.chtc.wisc.edu:9619" > itb-ce2.chtc.wisc.edu.scitoken

          

Configure Factory to submit your SciToken

  1. Check that SCITOKENS authentication is enabled on the Factory. For example:
    [root@fermicloud359]# condor_config_val SEC_DEFAULT_AUTHENTICATION_METHODS
    SCITOKENS,TOKEN,FS,GSI
                  
  2. Rename the previously genereated SciToken and place it in the correct directory so that the Factory knows to submit it to the CE.

    The previously generated SciTokens included an 'aud' (audience) claim for example aud="fermicloud349.fnal.gov:9619". This host/portnumber will be found in the glidein entry point configuration file. The SciToken must be named for the corresponding entry name appended by '_scitoken'. To illustrate:

    [root@fermicloud359 config.d]# cd /etc/gwms-factory/config.d
    [root@fermicloud359 config.d]# grep fermicloud349.fnal.gov:9619 *.xml
    Dev_Sites3.xml:      <entry name="el7_osg35" auth_method="grid_proxy" enabled="True"
    gatekeeper="fermicloud349.fnal.gov fermicloud349.fnal.gov:9619" gridtype="condor"
    rsl="(queue=default)(jobtype=single)" schedd_name="fermicloud173.fnal.gov"
    trust_domain="OSG" verbosity="fast" work_dir="OSG">
    [root@fermicloud359 config.d]#
    [root@fermicloud359 config.d]#
              

    In this example, the SciToken must be named el7_osg35_scitoken so the factory knows to associate it with entry point el7_osg35 and submit it to fermicloud349.fnal.gov:9619. The SciToken goes in directory
    /var/lib/gwms-factory/client-proxies/user_frontend/glidein_gfactory_instance.
    In other words:

    [root@fermicloud359 ~]# ls /var/lib/gwms-factory/client-proxies/user_frontend/glidein_gfactory_instance/el7_osg35_scitoken
    /var/lib/gwms-factory/client-proxies/user_frontend/glidein_gfactory_instance/el7_osg35_scitoken
    [root@fermicloud359 ~]#
              

    As SciTokens expire, a mechanism must be in place to renew it periodically. A cron job running 'scitokens-admin-create-token' would suffice for development purposes, but some other mechanism would be used in a production environment.

Configure CE to accept your SciToken

  1. Check that htcondor on your CE accepts SCITOKENS as an authentication method:
    root@fermicloud349 ~]# condor_ce_config_val -dump | grep SCITOKENS
    COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
    COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
    COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
    MASTER.SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI
    SCHEDD.SEC_READ_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
    SCHEDD.SEC_WRITE_AUTHENTICATION_METHODS = FS,SCITOKENS,GSI
    SCITOKENS_SERVER_AUDIENCE = $(COLLECTOR_HOST)
    SEC_CLIENT_AUTHENTICATION_METHODS = SCITOKENS, FS, GSI, PASSWORD
    SEC_DEFAULT_AUTHENTICATION_METHODS = SCITOKENS,FS, GSI
                  
  2. Check that an entry in the CE's mapfile assigns your SciToken to a valid user. In this example SciTokens issued by 'https://jobsub.fnal.gov' are mapped to user 'osg'
    [root@fermicloud349 ~]# condor_ce_config_val -dump | grep mapfile
    CERTIFICATE_MAPFILE = /etc/condor-ce/condor_mapfile
    [root@fermicloud349 ~]# cat /etc/condor-ce/condor_mapfile
    SCITOKENS https://jobsub.fnal.gov osg
    GSI (.*) GSS_ASSIST_GRIDMAP
    GSI "(/CN=[-.A-Za-z0-9/= ]+)" \1@unmapped.htcondor.org
    CLAIMTOBE .* anonymous@claimtobe
    FS "^(root|condor)$" \1@daemon.htcondor.org
    FS "(.*)" \1
    [root@fermicloud349 ~]#
                  

Verify Token Functionality

Here we see that the freshly installed SciToken can be decoded by command line tools and that condor_ping to the CE's collector authenticates some condor operations as user 'osg' as configured earlier in fermicloud349's condor_mapfile:

[root@fermicloud359 ~]# export SCF=/var/lib/gwms-factory/client-proxies/user_frontend/glidein_gfactory_instance/el7_osg35_scitoken
[root@fermicloud359 ~]# ~dbox/bin/decode_jwt_human $SCF
{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "7b50"
}
{
  "jti": "9186e3ab-5664-448c-a57c-69745967ec91",
  "sub": "condor",
  "exp": "Wed Jun 10 06:25:18 CDT 2020",
  "iat": "Wed Apr 29 14:25:18 CDT 2020",
  "iss": "https://jobsub.fnal.gov",
  "scope": "condor",
  "nbf": "Wed Apr 29 14:25:18 CDT 2020",
  "aud": "fermicloud349.fnal.gov:9619"
}
[root@fermicloud359 ~]# _condor_SCITOKENS_FILE=$SCF _condor_SSL_SKIP_HOST_CHECK=true condor_ping -address "<$(host fermicloud349.fnal.gov | awk '{print $4}'):9619>" -table ALL
         Instruction Authentication Encryption Integrity Decision Identity
               ALLOW      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
                READ           none       none      none    ALLOW unauthenticated@unmapped
               WRITE           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
          NEGOTIATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
       ADMINISTRATOR           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
               OWNER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              CONFIG           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
              DAEMON           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_STARTD      SCITOKENS       none       MD5    ALLOW osg@users.htcondor.org
    ADVERTISE_SCHEDD           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
    ADVERTISE_MASTER           FAIL       FAIL      FAIL     FAIL FAIL  (use -verbose for more info)
[root@fermicloud359 ~]#

Verify that submitting a job to the Factories entry point uses SciTokens authentication by adding D_SECURITY to the ALL_DEBUG condor setting on the appropriate condor daemons. The condor logs will show what authentication was used, and it should be SCITOKENS between the factory collector and the CE schedd.

Useful Links