Philihp.com » SAS

Abusing Hash objects as a Stack in SAS Data Step

philihp — Fri, 07 May 2010 21:00:40 +0000

Everyone in the computer science field (should) eventually learn or realize that any recursive function can be rewritten as an iterative process with the aid of a stack. Since a SAS Data Step is iterative, it’s fairly easy to look up children of a tree node in metadata, but nearly impossible to recursively look up children of children, and so fourth, without breaking out of the Data Step loop and into macro code, because there’s no native program stack.

You can use the LINK keyword, but that just keeps the PC register in a stack to return to where you were a subroutine was called, it doesn’t recreate the data vector. Because of this, all variables in a SAS Data Step behave as global within the data step block.

You could use a _TEMPORARY_ array as a stack, but its size has to be hard-coded to a predetermined depth.

SAS 9 has a rarely used Hash object, which can be used within a Data Step. It grows as needed, so it can be used to function as a functionally infinite length array, and further abused to function as a stack within a Data Step. The following code demonstrates this:

data _null_;
  declare hash stack();
  length stackvar $100;
  length stackdepth 8.;
  stackdepth=0;
  rc=stack.defineKey('stackdepth');
  rc=stack.defineData('stackvar');
  rc=stack.defineDone();

  stackvar='a';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  stackvar='b';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  stackvar='c';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;

  stackvar='x';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  stackvar='y';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  stackvar='z';
  stackdepth+1;
  rc=stack.add();
  put 'push' stackdepth= stackvar=; 

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;

  rc=stack.find();
  rc=stack.remove();
  put 'pop' stackdepth= stackvar=;
  stackdepth+-1;
run;

It outputs this:

pushstackdepth=1 stackvar=a
pushstackdepth=2 stackvar=b
pushstackdepth=3 stackvar=c
popstackdepth=3 stackvar=c
popstackdepth=2 stackvar=b
popstackdepth=1 stackvar=a
pushstackdepth=1 stackvar=x
pushstackdepth=2 stackvar=y
pushstackdepth=3 stackvar=z
popstackdepth=3 stackvar=z
popstackdepth=2 stackvar=y
popstackdepth=1 stackvar=x

By doing this, you can traverse a tree in metadata.

Looping through SAS metadata objects

philihp — Wed, 05 May 2010 20:07:47 +0000

The following is my attempt at SAS Golf, where in a Data Step, I try to list the associations of a metadata object to the log. This is basically just taking advantage of combining SAS for loops with SAS while loops. I thought it was cool. Usually I find myself doing this with metadata_getnasn, metadata_getnprp, and metadata_getnobj

n – index for the nth association
u – uri to object
a – association

data;
  ...
  do n=1 by 1 while(n
UPDATE: The metadata_getnass function has been renamed to metadata_getnasn

Dramatically Increasing SAS DI Studio performance of SCD Type-2 Loader Transforms

philihp — Tue, 19 Jan 2010 21:55:35 +0000

In SAS DI Studio 3.4 (and I imagine in future versions), the prepackaged code for the SCD Type-2 Loader works like this: Does the dataset exist? If not, create an empty dataset with structure and indexes as defined from metadata. Then detect differences between it and the source dataset and the target dataset, expire any observations that are modified or deleted by setting their valid-to-date to now, and append any modified or new observations with a valid-from-date. The expire bit is done in-place with a data step modify statement, and the append is done with PROC APPEND. I assume this is done to reduce the amount of locking necessary on the dimension dataset. Because new observations are appended, the dataset never actually gets sorted by the business key, so this could lead to exponential growth over time on the expire bit; every time the transform wants to change a single observation’s valid-to-date, it scans the entire table. And it doesn’t help that compression is off by default.

In the instance below, we had an uncompressed dataset with a modest 220,000 rows. The dataset had a variable “description” defined as a 4000-length string which was usually but not always null. Most steps of the SCD Loader run in a few seconds, but the following is usually the bottleneck. On a run one night to populate a new variable, this data step ran in a little over 9 hours:

     data PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST;
        modify PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST
           work.etls_close
            (rename = (ETLS_KEY = SW_ENTITY_REL_X_HOST_ID
           ETLS_FROMDATE = VALID_FROM_DTTM))
            updatemode=nomissingcheck;
        by SW_ENTITY_REL_X_HOST_ID VALID_FROM_DTTM;
        VALID_TO_DTTM = ETLS_CLSDATE;
        if %sysrc(_SOK) eq _iorc_ then
           replace;
        _iorc_ = 0;
        _error_ = 0;
     run;

NOTE: There were 1 observations read from the data set PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST.
NOTE: The data set PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST has been updated.  There were 45924 observations rewritten, 0
     observations added and 0 observations deleted.
NOTE: There were 45924 observations read from the data set WORK.ETLS_CLOSE.
NOTE: DATA statement used (Total process time):
     real time           9:06:00.46
     cpu time            9:05:58.43

Turning compression on reduced the size of this dataset from 1.1 gigs to 4.5 megs, mostly from compressing the 4000-char string that was usually empty; with compression off, an X-length string always takes up X bytes because it’s faster to seek to a certain observation if all observations are the same size. Additionally, in metadata an index was defined on the two variables used above in the BY statement. I ran this PROC DATASETS statement to create the index by hand (the Loader would create them if the table didn’t exist, but it assumes indexes exist if the table exists).

proc datasets library=prestage nolist;
 modify load_sw_entity_release_x_host;
 index create ndx=(sw_entity_rel_x_host_id valid_from_dttm) / unique;
quit;

With this metadata in place, the SCD Loader uses the index in the datastep and generates the following datastep instead:

     data PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST;
        set work.etls_close(rename = (ETLS_KEY = SW_ENTITY_REL_X_HOST_ID
                                      ETLS_FROMDATE = VALID_FROM_DTTM));
        modify PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST
            key=ndx / unique;
        VALID_TO_DTTM = ETLS_CLSDATE;
        if %sysrc(_SOK) eq _iorc_ then
           replace;
        _iorc_ = 0;
        _error_ = 0;
     run;

By using the index, the runtime was reduced to under two seconds.

NOTE: There were 45924 observations read from the data set WORK.ETLS_CLOSE.
NOTE: The data set PRESTAGE.LOAD_SW_ENTITY_RELEASE_X_HOST has been updated.  There were 45924 observations rewritten, 0
     observations added and 0 observations deleted.
NOTE: DATA statement used (Total process time):
     real time           1.41 seconds
     cpu time            1.21 seconds

Easy Inserting/Appending Libraries into FMTSEARCH path lists in SAS 9.2

philihp — Tue, 27 Oct 2009 21:49:50 +0000

In SAS 9.1.3 and prior, Options list (such as FMTSEARCH for libraries with format catalogs, and SASAUTOS for paths that contain macros shared across jobs) were annoying to work with. If you have nested code that wants to add a library or a path the list, doing so like this could potentially clobber statements executed outside of the nested block:

OPTIONS FMTSEARCH=(mylib.formats);

To really be safe, you want to add your library to the list. Doing this wasn’t very intuitive. I had always done this like this:

%let FMTSEARCH = %substr(%sysfunc(getoption(fmtsearch) ) ,2);
OPTIONS FMTSEARCH = (mylib.formats &FMTSEARCH;

But in 9.2, there are two new System Options, INSERT= and APPEND=. This is now as simple as

OPTIONS INSERT=(FMTSEARCH=mylib.formats);

Counting distinct variables in SQL with SAS

philihp — Mon, 26 Oct 2009 06:44:45 +0000

One way to get the count of distinct variables, which works in most flavors of SQL, is to use a subquery. For instance, in Oracle this is:

SELECT count(SELECT DISTINCT foo FROM table) FROM dual

In SAS, using PROC SQL, you can do that too, but you can also simply do this:

SELECT count(distinct foo) FROM table

For loops in SAS

philihp — Wed, 07 Oct 2009 16:06:22 +0000

When coding for loops in SAS, one neat thing to remember is that all of the parts of it are optional.

start

http://support.sas.com/documentation/cdl/en/lrdict/62618/HTML/default/a000201276.htm

So in another language, what might be

for(int i=0;i < max;i++) {

In SAS, you can leave out the "by 1", since it's the default increment. So this would be

do i=0 to max;

What's fun is in SAS you can combine a Repeat-Until/Do-While loop with a for loop. This might be useful if for some reason you don't set the var "max" until you've iterated through the loop once, such as in this case, where "metadata_getnobj" returns a negative on fail, but otherwise returns the number of objects that match the query.

do i=1 by 1 until (i >= objs);
  rc=metadata_getnobj("omsobj:SASLibrary?@Libref='MYLIBREF'",i,uri);
  if rc>=0 then objs=rc;

  rc=metadata_getnasn(uri,"LibraryConnection",1,uri);
  if rc>=0 then leave;
end;

This code loops through all the libraries in the metadata with libref='MYLIBREF', and exits with a URI for the first one it sees with at least one LibraryConnection association. This is useful when you have a remote library pointing to a local library and they both have the same name and you want a URI to the remote one.

Another cool thing with FOR loops in SAS is iteration through lists. In Java, say you have a block like this:

for(String x : new String[] { "a","b","c" }) {
  System.out.println("x="+x);
}

In SAS, this is simply

do x="a","b","c";
  put x=;
end;

Removing formats from all variables in a SAS Dataset

philihp — Wed, 12 Aug 2009 20:42:48 +0000

In SAS, every field/variable in a table/dataset can be given a format. This format tells SAS how to display the data. The following datastep will create a table called “formatted” having 1 row containing 3 variables: x, y, and d.

data formatted;
  x=9000;
  y=42;
  now=16761;
  format x comma6.;
  format y dollar5.;
  format now date7.;
  put x=;
  put y=;
  put now=;
run;

Additionally it prints to the log the formatted values.

x=9,000
y=$42
now=21NOV05

In SAS you can also create your own formats, and you can assign these formats to whichever datasets you like. However if that format goes away for some reason and you try to look at the data again, you will get this error:

ERROR: Informat $YOURFMT not found or couldn't be loaded for variable YOURVAR.

You can get around this by turning off this option

options nofmterr;

Then your ERROR turns into NOTE, and things work normally, except when you try to view the data you see it unformatted. This could be useful, however, if you find yourself in a situation like this:

data _null_;
  a=20.0000001;
  b=20.000001;
  format a dollar4.2;
  format b dollar4.2;
  put a= b=;
  if(a=b) then put 'matches';
   else put 'no match';
run;

Runs and outputs:

a=20.0 b=20.0
no match

At any rate, there are times in SAS when you simply want to remove all the formats from a dataset. This can be done in one line in a datastep:

data unformatted;
  set formatted;
  format _all_;
run;

Converting between Date and Datetime in SAS

philihp — Mon, 01 Jun 2009 21:01:10 +0000

In SAS, fields are either character of varying length, or numeric. No exceptions. Temporal values such as Date and Date/Time are stored as either the number of days or seconds since 1960 January 1st.

In order to convert from Date/Time to Date, and from Date to Date/Time, you could divide or multiply respectively by 86400 (the number of seconds in a day).

data _null_;
  d='29FEB1984'd;
  put d date.;
  dt=d*24*60*60;
  put dt datetime.;
  d=dt/24/60/60;
  put d date.;
run;

But unless someone deals with dates every day, it might not be obvious to them by reading this code what those magic numbers are doing.

Another method would be to use put() and input() using functions to go back and fourth.

data _null_;
  d='29FEB1984'd;
  put d date.;
  dt=input(put( d  ,date7.) || ':00:00:00', datetime.);
  put dt datetime.;
  d =input(substr(put(dt,datetime.),1,7),date7.);
  put d date.;
run;

But I don’t consider this any better. It’s wordy. Functions are nested within functions. And you still have a magic number for the call to substr. Also, since it isn’t uncommon for SAS datasets to get into the billions of rows, this probably isn’t the fastest way to do it.

SAS has built-in functions called dhms() (mnemonic: date,hour,minute,second) and datepart() that should make things a lot easier.

data _null_;
  d='29FEB1984'd;
  put d date.;
  dt=dhms(d,0,0,0);
  put dt datetime.;
  d =datepart(dt);
  put d date.;
run;

Installing SAS 9.1.3 on Ubuntu Linux 8.10

philihp — Fri, 23 Jan 2009 00:42:43 +0000

It works. Just follow the instructions.

Installing SAS 9.1.3 on OpenBSD 4.3

philihp — Fri, 23 Jan 2009 00:42:00 +0000

It does not work.