SELECT * FROM apples
UNION
SELECT * FROM oranges

When you know for a fact that there will never be any common rows between the apples table and the oranges table, this query will be slightly faster with at low cardinality, and incredibly faster at high cardinality by using “UNION ALL”

SELECT * FROM apples
UNION ALL
SELECT * FROM oranges

The difference between the two queries is this: UNION ALL will simply concatenate the two queries together into the resultset. Just using UNION will concatenate, but then remove duplicates (do a distinct sort). Leaving out this second step can vastly reduce the time it takes for your query to run.

SELECT count(SELECT DISTINCT foo FROM table) FROM dual

In SAS, using PROC SQL, you can do that too, but you can also simply do this:

SELECT count(distinct foo) FROM table

NOTE: PROCEDURE SQL used (Total process time):
real time 8.83 seconds
cpu time 8.65 seconds
Here, the table imf.exchange_rate has 13416 rows, covering exchange rates at close, daily, for 39 different currencies, over nearly 1 year. Modest, but fairly small. It has no indexes, and has not been sorted (or marked as sorted). work.exchange_rate is a smaller version of it, covering only exchange rates for the last month, with 980 rows. The query is trying to return any exchange rates that we didn’t have before.

Should be simple right? There’s no reason for it to take this long. By rewriting the query to do a left join, below, SAS merges the tables behind the scenes, then finishes the query in a single scan.
proc sql;
create table new_rates as
select n.*
from work.exchange_rate n
left join imf.exchange_rate o
on (n.effective_date = o.effective_date and n.iso_char_code = o.iso_char_code)
where o.iso_char_code = '';

NOTE: PROCEDURE SQL used (Total process time):
real time 0.13 seconds
cpu time 0.13 seconds

          [node]         /           [node]    [node]      /            [node]  [node]   [node]

                     [node]

In a properly normalized OLTP database, what people usually do is create a table structure where each node only knows his parent. This satisfies the requirement of database normalization to have each fact/idea in one and only one location so that there can’t possibly be a conflict (aside: this doesn’t guarantee that there can’t be conflicts. Node A could say node B is its parent, and node B could say node A is its parent).

This is called the Adjacency Model, since it is clear when two records are adjacent siblings; they share the same parent.

The way this looks as far as table schema would be

+--------------+| NODE         | <---.+--------------+     || node_id      |     || parent_id    | ----'| node_data... |+--------------+

The other day I was asked how one could quickly query the database the question “How do I find all of the direct and indirect children of a node?”

Direct children are simple…

SELECT *  FROM node  WHERE parent_id = ?

Children of children of a node is also simple…

SELECT *  FROM node  WHERE parent_id = ?    OR parent_id in (      SELECT node_id        FROM node        WHERE parent_id = ?)

But this gets out of hand pretty quickly, and it becomes obvious we won’t be able to have a single, simple, fast query that gives ALL children of a node, no matter how far down their ancestry.

Enter the Materialized Path Model

In a Materialized Path model, each row of your table knows its entire ancestry. It’s a little like if every file on your file system knew its entire path, rather than just the folder it was in.

Probably the best way to describe it is by example. Say we had this tree:

             0           / |           1  2  3         /  /         4  5   6              /|             7 8 9

In an adjacency model, our table looks like:

+---------+-----------+| node_id | parent_id |+---------+-----------+| 0       | .         || 1       | 0         || 2       | 0         || 3       | 0         || 4       | 1         || 5       | 2         || 6       | 2         || 7       | 6         || 8       | 6         || 9       | 6         |+---------+-----------+

However in a materialized path model, we would store this as

+---------+-------+| node_id | path  |+---------+-------+| 0       |       || 1       | 1     || 2       | 2     || 3       | 3     || 4       | 1.4   || 5       | 2.5   || 6       | 2.6   || 7       | 2.6.7 || 8       | 2.6.8 || 9       | 2.6.9 |+---------+-------+

In order to ask the database “which nodes are children of x” you can query with

SELECT *  FROM node  WHERE path LIKE    (SELECT path       FROM node       WHERE node_id = x)||'%'

In some languages, the “begins with” operation is optimized. MySQL can use regular expressions. SAS has the =: operator.

This model lends itself to other nice things; finding the depth of a node can be calculated on the number of dot delimiters in the path.

The Materialized Path model works very well as opposed to the Adjacency model when the root node, and nodes with deep ancestry rarely change; which is usually the case in classification schemes (base categories tend to be the established ones) and company org hierarchies (upper level executive managers tend to be senior).

We can do a little bit better with a materialized path model, though… More in Part 2.