Humdrum Lab 3

Perhaps view Humdrum Lab 2 before this lab.

This lab demonstrate using tindex/themax/theloc to search for melodic/rhythmic patterns in **kern scores.

First download the erk songs from Humdrum Lab 2 and the Bach chorales from Humdrum Lab 1.

Split songs into major and minor groups

Similar to the last lab, split the Erk songs into those that are in major and those that are minor:

   mkdir major
   mkdir minor
   cp `egrep -l '^\*[A-G][#-]?:' *.krn` major
   cp `egrep -l '^\*[a-g][#-]?:' *.krn` major

The regular expression:

    ^\*[A-G][#-]?:

means search for a line that starts (^) with an asterisk (\*) followed by one character in the range from A to G ([A-G]) followed optionally by the character '#' or '-' ([#-]?) followed by a colon (:).

Notice the back-quotes in the last two commands. This means to run the command inside of the backquotes first, then use that content as argument input into the rest of the command.

The `-l` option for egrep is used to list the files the contain the match, not the matches themselves.

egrep is used instead of grep, because '?' is an extended regular expression metacharcter. In the basic set of regular expression meta characters, the '?' character is a normal character with no special meaning.

Aside on regular expressions

Here are the basic regular-expression operators:

.	Any single character
*	zero or more occurrences of the previous character
[]	one of the characters in the enclosed list
^	anchor match to start of line
$	anchor match to the end of line

After the basic set was developed, people wanted more operators, so an extended set was added:

+	one or more occurrences of the previous character
?	zero or one occurrence of the previous character
()	grouping, such as (cat)+ which would match to cat, catcat, catcatcat, etc.
|	logical or operator, such as (cat|mouse) which would match to cat or mouse.
{}	generalize counter, such as {3,5} meaning between three and five occurrences of the previous character or parentheses grouping.

The confusing thing is that grep understands the basic set, and egrep understands both sets. If you use a "?" character in grep, it will be treated as a regular character, while in egrep it will be a special character. However, you can use "?" as a special character in grep by adding a backslash in front of the question mark (\?). This forces it to become a metacharacter. Oddly, doing the same thing in egrep will cause the question mark to become a regular character.

Aside on shell scripts

Suppose that you want to often split scores into major and minor mode groups. You can store the commands to do that in a shell script. Create a file called "splitbymode" with this contents:

  #!/bin/bash
  mkdir major
  mkdir minor
  cp `egrep -l '^\*[A-G][#-]?:' *.krn` major
  cp `egrep -l '^\*[a-g][#-]?:' *.krn` major

After saving it to the disk, change the permissions of the file to allow it to be run as a program:

   chmod 0755 splitbymode

Then to run it:

   ./splitbymode

The "./" prefix will probably be needed so that the shell knows where to look for the command. You can run the command

    echo $PATH | tr : '\n'

to see a list of the directories in which the shell will look for commands. If "." is not in the list (which means the current directory/folder), then you must add "./" to tell the terminal to look in the current directory for the command.

The first line is called a shebang. It specifies the location of the interpreter for the programming language used in the script. In this case the syntax is bash shell commands, so the bash interpreter is used.

Other languages and interpreters can be used. For example try this PERL script:

  #!/usr/bin/perl
  `mkdir -p major`;
  `mkdir -p minor`;
  system("cp `egrep -l '^\\*[A-G][#-]?:' *.krn` major");
  system("cp `egrep -l '^\\*[a-g][#-]?:' *.krn` minor");

PERL allows shell commands by adding back-quotes around the command. Or "system()" can be used to run a shell command, which is needed in the cases when the shell command is using back-quotes itself.

Most common scale-degree pattern in each mode

Here is a demonstration of how to extract a list of the most common scale-degree sequences in the major and minor datasets:

   cd major
   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \ 
                | sortcount | head -n 10
   470	5 4 3 2 1
   361	5 6 5 4 3
   275	6 5 4 3 2
   225	3 5 4 3 2
   209	1 2 3 4 5
   198	2 1 7 6 5
   182	5 5 4 3 2
   180	3 3 3 3 3
   179	2 3 4 3 2
   174	3 3 2 2 1

   cd ../minor
   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
               | sortcount | head -n 10
   44	5 4 3 2 1
   31	1 2 3 4 5
   30	3 2 1 7 1
   29	1 2 3 2 1
   27	5 5 5 5 5
   25	3 2 1 2 3 
   22	4 5 3 2 1
   21	5 5 4 3 2
   21	4 3 2 1 2
   19	4 3 2 1 7

The command:

    cat *.krn | deg -a 

is used (for the major files) instead of :

    deg -a *.krn

because there are too many *.krn files for the AWK interpreter to handle all of them as commad-line arguments.

Uncommon patterns

Here is how to list all of the 5-note patterns which occur only twice in all of the major works:

   cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
             | sortcount | grep '^2\b' | less

Here are some of the patterns:

2       5 1 7 7 5
2       7 2 1 2 2
2       2 5 3 6 5
2       5 5 6 6 2
2       2 2 1+ 2 2
2       2 1 7 5 2
2       5 4 5 3 2-

The \b in the regular expressions "^2\b" means that there must be a word boundary at that position. This means that '^2\b' will match to 2 but not to 28, since 8 is part of the word "28".

To count the number of patterns that only occur twice:

  cat *.krn | deg -a | grep -v ^= | grep -v r | context -n 5 | ridx -H \
       | sortcount | grep '^2\b' | wc -l
  1098

There are 7064 unique 5-note patterns, so 15.5% only occur twice and 2094 or 29.6% only occur once.

• Exercise: Use the frequencies of 5-note scale patterns to calculate the likely key of melodies.

Locating patterns in music

The scale degree pattern occurs twice in 1591 songs, but where? Does the pattern occur twice in one song, or once in two songs?

To answer these questions, use the Humdrum Extra tools related to Themefinder searching: tindex, themax, and theloc.

"tindex" (thema indexer) creates a feature database (index) of melodic lines. "themax" searches the index, and "theloc" links the search results back to the original scores.

First, run "tindex" to create the feature database for the major songs:

   tindex *.krn > index.txt

Now, you can use it to search for pattens in the songs. Here is an example of how to search by scale degrees "7 2 1 2 2":

  themax -d "7 2 1 2 2" --count index.txt
   deut1920.krn::1	1
   deut2106.krn::1	1

The pattern occurs once in two separate files. The ::1 at the end of the filename means that the match occurred in the first spine of the file (which only has one spine since the music is monophonic). The 1 at the end of the lines indicates how many time the pattern occurs in the file.

You can add the --total option to count the occurrences in all of the data. The total count is given at the end of the output:

   themax -d "7 2 1 2 2" --count --total index.txt
   deut1920.krn::1	1
   deut2106.krn::1	1
   2

To determine where the match occurs within the music, use the --loc option:

   themax -d "7 2 1 2 2" --loc index.txt
   deut1920.krn::1	28-32
   deut2106.krn::1	24-28

In the first file the pattern occurs at notes 28 to 32, counting from the start of the music.

The theloc program can mark the notes in the original score by using this sort of command pipeline:

    themax -d "7 2 1 2 2" --loc index.txt | head -n 1 | theloc --mark

Try copying this output and pasting into VHV. In MacOS, you can use pbcopy:

    themax -d "7 2 1 2 2" --loc index.txt | head -n 1 | theloc --mark | pbcopy

Displaying the location in the second file:

   themax -d "7 2 1 2 2" --loc index.txt | head -n 2 | tail -n 1  | theloc --mark | pbcopy

The command pipeline:

    head -n 2 | tail -n 1 

is one way to extract the second line of content: head -n 2 extracts the first two lines of text and tail -n 1 extracts the last line of some text.

Showing matched music only

The myank command can be used with the --mark option to extract all measures containing marked (matched) notes:

    themax -d "7 2 1 2 2" --loc zindex | head -n 2 | tail -n 1  | theloc --mark | myank --mark | pbcopy

Aside: creating a musical figure with VHV

You can type alt-t to download a PDF of the currently viewed music. This can be drag-and-dropped into MS Word for example to become a figure:

Resulting PDF dragged and dropped into MS Word:

Note that the bounding box on the PDF will probably cause extra whitespace in MS Word, so you will have to format the picture so that text can be placed above or below it so that text can fill in the empty spaces of the figure.