Humdrum Lab 3

Perhaps view Humdrum Lab 2 before this lab.

This lab demonstrate using tindex/themax/theloc to search for melodic/rhythmic patterns in **kern scores.

First download the erk songs from Humdrum Lab 2 and the Bach chorales from Humdrum Lab 1.

Split songs into major and minor groups

Similar to the last lab, split the Erk songs into those that are in major and those that are minor:

   mkdir major
   mkdir minor
   cp `egrep -l '^\*[A-G][#-]?:' *.krn` major
   cp `egrep -l '^\*[a-g][#-]?:' *.krn` major

The regular expression:

    ^\*[A-G][#-]?:

means search for a line that starts (^) with an asterisk (\*) followed by one character in the range from A to G ([A-G]) followed optionally by the character '#' or '-' ([#-]?) followed by a colon (:).

Notice the back-quotes in the last two commands. This means to run the command inside of the backquotes first, then use that content as argument input into the rest of the command.

The `-l` option for egrep is used to list the files the contain the match, not the matches themselves.

egrep is used instead of grep, because '?' is an extended regular expression metacharcter. In the basic set of regular expression meta characters, the '?' character is a normal character with no special meaning.

Side-track on regular expressions

Here are the basic regular-expression operators:

.	Any single character
*	zero or more occurrences of the previous character
[]	one of the characters in the enclosed list
^	anchor match to start of line
$	anchor match to the end of line

After the basic set was developed, people wanted more operators, so an extended set was added:

+	one or more occurrences of the previous character
?	zero or one occurrence of the previous character
()	grouping, such as (cat)+ which would match to cat, catcat, catcatcat, etc.
|	logical or operator, such as (cat|mouse) which would match to cat or mouse.
{}	generalize counter, such as {3,5} meaning between three and five occurrences of the previous character or parentheses grouping.

The confusing thing is that grep understands the basic set, and egrep understands both sets. If you use a "?" character in grep, it will be treated as a regular character, while in egrep it will be a special character. However, you can use "?" as a special character in grep by adding a backslash in front of the question mark (\?). This forces it to become a metacharacter. Oddly, doing the same thing in egrep will cause the question mark to become a regular character.

Most common scale-degree pattern in each mode

Here is a demonstration of how to extract a list of the most common scale-degree sequences in the major and minor datasets:

   cd major
   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \ 
                | sortcount | head -n 10
   470	5 4 3 2 1
   361	5 6 5 4 3
   275	6 5 4 3 2
   225	3 5 4 3 2
   209	1 2 3 4 5
   198	2 1 7 6 5
   182	5 5 4 3 2
   180	3 3 3 3 3
   179	2 3 4 3 2
   174	3 3 2 2 1

   cd ../minor
   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
               | sortcount | head -n 10
   44	5 4 3 2 1
   31	1 2 3 4 5
   30	3 2 1 7 1
   29	1 2 3 2 1
   27	5 5 5 5 5
   25	3 2 1 2 3 
   22	4 5 3 2 1
   21	5 5 4 3 2
   21	4 3 2 1 2
   19	4 3 2 1 7

The command:

    cat *.krn | deg -a 

is used (for the major files) instead of :

    deg -a *.krn

because there are too many *.krn files for the AWK interpreter to handle all of them as commad-line arguments.

Uncommon patterns

Here is how to list all of the 5-note patterns which occur only twice in all of the major works:

   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
             | sortcount | grep '^2\b' | less

Here are some of the patterns:

2       5 1 7 7 5
2       7 2 1 2 2
2       2 5 3 6 5
2       5 5 6 6 2
2       2 2 1+ 2 2
2       2 1 7 5 2
2       5 4 5 3 2-

The \b in the regular expressions "^2\b" means that there must be a word boundary at that position. This means that '^2\b' will match to 2 but not to 28, since 8 is part of the word "28".

To count the number of patterns that only occur twice:

  cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
       | sortcount | grep '^2\b' | wc -l
  1098

There are 7064 unique 5-note patterns, so 15.5% only occur twice and 2094 or 29.6% only occur once.

• Exercise: Use the frequencies of 5-note scale patterns to calculate the likely key of melodies.

Locating the patterns in the music

The scale degree pattern occurs twice in 1591 songs, but where? Does the pattern occur twice in one song, or once in two songs?

To answer these questions, use the Humdrum Extra tools related to Themefinder searching: tindex, themax, and theloc.

"tindex" (thema indexer) creates a feature database (index) of melodic lines. "themax" searches the index, and "theloc" links the search results back to the original scores.

First, run "tindex" to create the feature database for the major songs:

   tindex *.krn > index.txt

Now, you can use it to search for pattens in the songs. Here is an example of how to search by scale degrees "7 2 1 2 2":

  themax -d "7 2 1 2 2" --count index.txt
   deut1920.krn::1	1
   deut2106.krn::1	1

The pattern occurs once in two separate files. The ::1 at the end of the filename means that the match occurred in the first spine of the file (which only has one spine since the music is monophonic). The 1 at the end of the lines indicates how many time the pattern occurs in the file.

You can add the --total option to count the occurrences in all of the data. The total count is given at the end of the output:

   themax -d "7 2 1 2 2" --count --total index.txt
   deut1920.krn::1	1
   deut2106.krn::1	1
   2

To determine where the match occurs within the music, use the --loc option:

   themax -d "7 2 1 2 2" --loc index.txt
   deut1920.krn::1	28-32
   deut2106.krn::1	24-28

In the first file the pattern occurs at notes 28 to 32, counting from the start of the music.

The theloc program can mark the notes in the original score by using this sort of command pipeline:

    themax -d "7 2 1 2 2" --loc index.txt | head -n 1 | theloc --mark

Try copying this output and pasting into VHV. In MacOS, you can use pbcopy:

    themax -d "7 2 1 2 2" --loc index.txt | head -n 1 | theloc --mark | pbcopy

Displaying the location in the second file:

   themax -d "7 2 1 2 2" --loc index.txt | head -n 2 | tail -n 1  | theloc --mark | pbcopy

The command pipeline:

    head -n 2 | tail -n 1 

is one way to extract the second line of content: head -n 2 extracts the first two lines of text and tail -n 1 extracts the last line of some text.