Music Processing, from Stage1 to Final Edition
                      Created: 02/23/06
                      Revised: 04/28/09

The propose of this documention is to describe all of the steps
in the process of going from a set of primitive stage1 files to
a final edition of music.  What is laid out here is a continuous
work flow process; but the reality is that it is almost never
followed in its entirety.  Almost all works, in fact, spend a
good deal of time in a partial state of completion; many will
never reach the final state of a musical edition.

The real purpose of this documentation, then, is to gather in one
place all of the instructions for the various stages of data entry
and processing.

Note: Unless otherwise stated, all programs referred to in this
document are stored in the directory j:/release/progs.  On our
Linux systems, this directory is included in the searchable list
directories for zbex progs, which means they can be run directly
by name without typing zz first.

                    I. Initial Data Entry
                 (creating basic Stage2 files)

The final result of the Intial Data Entry process for a musical
movement (or piece) is a basic set of stage2 files.

At the moment, there are three ways to get there:

1. Create a set of Stage1 files, and then convert these to stage2.

   Stage1 files are single track only, and do not support such
   things as time changes or variable length measures.  Stage1
   files are the output of the program input.z (with midi enabled).

   Note: input.z is a combination of the old a program for midi
   data entry, the old b program for processing midi data, and the
   old look2 program for displaying the result.  At the moment
   (04/28/09) this program runs only in the old dmuse environment
   on a DOS machine.  Hardware is setup for this at the Middlefield

   Once created, Stage1 files may be converted in a generic way
   to the Stage2 format using the program begin.z.  Note: I recently
   did this with some stage1 Bach organ music, and the process worked

2. Create a set of Stage2 files, using the commercial program Sharp-eye.

   This process is currently used by Don and Ed at CCARH.  Basically,
   a peice (movement) is scanned by Sharp-eye and the Music-XML output
   is sent to the program Finale.  Finale has a plug-in which converts
   this to Stage2 files.  The scanning process involves some minimal
   editing.  The process works only with printed music.  The final
   Stage2 output requires some editing to bring the files into
   conformance with accepted practice.

3. Create Stage2 files by manually entering them in a DMUSE window.

   This process is slow, and is not recommended for large projects.
   For small projects, it does have the advantage of flexiblity.

             I(b). Setting up the file organization

This process is implied in creation process, but it is essential that
it be done at this point and that it be done right.  All Musedata files
are organized according to the following system:

      work/      -->  use any name you like
        stage2/        name must be stage2
          mvt1/    -->  use any name, but numbers are the default
            01      -->  use any name, but numbers are the default.
            02           The group structure determines the use and
            ...          meaning of a file.  See s2-specs.ten in
                         j:/release for details on group structures

        outputs/       name must be outputs
          score/   -->  The names at this level reflect the purpose and
          score18/      size of the music being compiled.  Letter portion
          short/        of the name must match one of the group names.
          skore16/      The number portion is the notesize.  Size 14 is
          parts/        the default and is not needed.

          score/        (example)
            formats      name must be formats
            i-files/     name must be i-files
              mvt1/  -->  these names match the names in the Stage2 directory
                01    -->  these file names are created by autoset, and are
                02         always numbers.  These are linear i-files for
                03         the various parts in the score.
            pages/       name must be pages
              mvt1/       these names match the names in the Stage2 directory
                01         these file names are created by mskpage, and are
                02         always numbers.  These are page specific i-files.

        editions/      any name is possible, but editions is preferred
          name1/        can be anything, but should be descriptive, e.g. 1st-2004/
            score/       any name, but descriptive.  This is where the .mpg edition
                         pages are stored.
            (some more examples of edition directories)
            scform  -->  this is the "make" file for the edition.  For no particular
                           reason, we have standardized on this name.  The scform
                           file can actually be stored anywhere on the system, since
                           all references to directories are by full path name.  The
                           scform file here is for building a score.  The scform files
                           for building parts are stored in the outputs/parts/pages/
                           partname/ directories
            score.cft    CFT compression of files in /score
            mes.srh      any name, but with srh extension.  This is the compressed
                           search file.
            postout.tmp/ this is a temporary directory used by the program
                           pspage.z to store .ps postscript file.
            ednum        this small file referenced by pspage.z to get the
                           CCARH edition number and production date.

            score/        (example)
              001.mpg     These are the pages in a score.  They should be numbered
              002.mpg     and should have the .mpg (music page) extension.

            parts/        (example)
              vln1         at the moment, there is no standardization on part names
                001.mpg     Notice that with the parts directory, there is an extra
                002.mpg     level of files.

        distrib/       name must be distrib
          kern/         must be kern:  kern files for each movement (by name)
          midi1/        must be midi1: midi1 files for each movement (by name)
          midiP/        must be midip: midip files for each movement (by name)
          mused/        must be mused: MUSEDATA files for each movement (by name)
          score/        must be score: score files for each movement (by name)
            mvt1/   -->  use any name, but numbers are the default

        doc            Optional file containing history/current-state of data
        notes          Optional file containing critical notes on data entry.

Sorry about this long list.  We are getting a bit ahead of ourselves here, but
it is important to show at the outset how the data structure should be set up
because all processes further down the line depend on this structure.

I haven't shown the structure above work/ because this will depend on what
piece is being entered.  This description belongs to another discussion on
the overall organization of CCARH data -- a very important subject, by the
way -- and one which has not, in my opinion, been worked out in satisfactory
detail.  The problem, in a nut shell, is that we do not yet have a clear idea
of what the public's view of our data should be -- although we have made
good progress here.

             II. Supplemental data entry on Stage2 files

This is actually an ongoing (and potentially never ending) process.  The
Stage2 file is fundamental data source for all other music files, whether
sound, display, print, or analysis.

But the Stage2 files that come out of Initial Data Entry often need
a number of things added, including slurs, articulations, ornaments,
transpositions, re-arrangement of stems and beams, clef changes,
track numbers, print suggestions to improve the visual layout, sound
suggestions for more accurate sound files.  In addition, the group
structure may need to be adjusted.  Cue-size notes may be added to
parts, etc., etc.

If Supplemental data entry is in an early stage, i.e., many basic
things need to be added, the program s2ed.z provides a graphical
means for doing this, part by part.

Otherwise, changes to Stage2 files must be done by hand.

For Stage2 files representing more than one instrument, or for
music on the grand staff, it may be necessary to merge two or
more Stage2 files.  There is a program called fmerge.z that purports
to do this, but it is flawed in many respects.  The directory
franfro inside release/progs contains some programs and a process
which is more careful about merging files.  The more careful the
program, the less work required to fix the result.

There have been some recent changes way multiple tracks are represented
in Stage2 files.

   (1) only use the backup command when a track reaches the end of a
       measure.  If the track does not reach, use the irest to complete
       the track.  This is necessary not only for score conversion,
       but also for making .srh files
   (2) When representing multiple instruments, do not use chords
       unless it is absolutely necessary.  autoset.z will construct
       chords when the music isorhythmic and the stems point the
       same direction.
   (3) the global print suggestion for rest placement can save a
       lot of time and space; try to use it.

                III. Create initial output displays

This is actually an interative process.  The initial objective is to
see the music in print and to use this to correct errors and to
enhance the music.

The final object is to get the stage2 files into the most advanced
state possible, before moving on to the Edition stage.

The basic sequence is autoset.z -> mskpage.z -> dskpage.z.  The first
object is to get these programs running without error.  This will
correct 99.9% of all format errors in the stage2 files.

              III(b). Determine the system layout

The system (and page) layout is influenced by the spacing of the
notes (compression factor), and by degree to which various empty
staff lines are dropped from the system (this also depends on
the system breaks.  Our software does not have a "top down" way
of arranging system breaks and page breaks.  There is a print
suggestion (on the measure record) for forcing a system break,
but the overall effect on system breaks is still left to chance.

The only way we have of deterining a system layout we like is to
run the autoset.z -> mskpage.z -> dskpage.z sequence several
times, with different compression values and possibly different
line spacings (e.g., contained in the files in outputs/formats).
This iterative process has, to date, always been successful in
producing a satisfactory system layout.

But it is essential that this process be completed to the user's
satisfaction at this point in time, because all future editing
will depend on a stable system layout.  The system layout can
be changed manually, of course, but this creates what we call
an "orphan" result -- that is, a result that is no longer tied
back to the original stage2 files.  It is of great stategic
importance that this connection be preserved, because we would
like to preserve the ability to make small changes in the Stage2
files and then run the autoset.z -> mskpage.z -> dskpage.z sequence
and other down stream processes again to get an "updated" Edition

       IV. Make adjustments to vertical and horizonal positions

Vertical adjustments and minor horizonal adjustments are made with
print suggestions in the Stage2 files.  These can be made at any
point in the process.

Major horizonal adjustments to note spacings can be made using
the xmskpage.z program, but because of bugs in this program, we
recommend that adjustments are made only once, and then not
changed.  The program displays each system as it is typeset, and
allows the user to move objects horizontally to improve the
layout.  The result of this editing is stored in the formats file
for this movement.  Thereafter, the basic typesetting sequence
should be autoset.z -> xmskpage.z -> dskpage.z.  Horizontal
editing results will be preserved, as long as you don't try
to make further changes.  At some point, I hope to make this
process more robust, so that further changes can be added.

            V. Make an Edition from the Page Files in OUTPUTS

There are several shortcomings with the page files in /outputs

 (1) Each movement is in a separate directory
 (2) Page bottoms are not uniform
 (3) Paging information is absent
 (4) The vertical arrangement of lines and systems is not optimal
 (5) Text underlay sometimes bumps into notes

In earlier times, it used to take several weeks to cobble together an
edition from the page files in /outputs/pages.  Each page had to be
separately edited and copied into an /editions directory.  Now there
is a faster way to do this work, and a means by which the process can
be automated, so that if (or when) changes are made up-stream (e.g.,
in the Stage2 files), the edition can be recompiled without doing all
of this work again.

The key to this process is an instruction file, the format of which
is documented in release/progs/edform.doc, and which for the time
being we are calling scform.  The file provides such information as:

   source lib:  pages library providing the source for the edition
   destin lib:  library into which output pages are to be stored
   blank page:  generic message to put on any "blank page"
   page id header:  location and font information for page id
   page id format:  page id, for every page.  Includes a page number
   top 7 "Z" records, required for .MPG file format

The file then provides, on a page by page basis, the page number, the
exact location of the source, and a set of (encoded) instructions on
how each musical staff line "L" is to be adjusted up or down, and also
how to adjust the vertical height of text underlay and a line of
figured harmonies, if either exist and need to be moved.

These instructions are optional; the page layout can be moved without
any adjustment.  But if the edition is to look "professional," the tops
and bottoms of the musical page should be uniform across the edition.
Simple instructions such as:

   VJ-1 = vertically justify proportionally to 2800
   VJ-2 = vertically justify proportionally to 2800; with x2 space between systems
   VJ-3 = vertically justify proportionally to 2800; use only inter-system space

can do this.  More complicated instructions address individual lines.
The scform file must be generated by hand, because the design of the
page layout and the individual files (with their separate movement
libraries) cannot be reliably constructed by computer.  The moving
instructions, however, can be generated by a program.  This program
is called vskpage.z, and is a variant of the page editing program
eskpage.z, about which more will be said in a moment.

vskpage.z is a full page graphics editor of a musical page.  The only
editing commands which work, however, are those which move the music
staff lines and music systems up and down, and those which move text
underlay and figured harmonies up and down.  The output of vskpage.z
is a line of instructions, displayed on the screen, which must be
transferred to the screen in which the scform file is being constructed.
This process may seem somewhat awkward, but it is flexible and provides
the user complete control over what goes in the scform file.

Once the scform file is built, the program that actually constructs
the edition is called vspace.z.  You can run vspave as many time as
you like, without changing anything in the source pages.  In this
way, by adjusting the scform file and its moving structions, you
get get the edition to look exactly the way you want.

Just to summarize an important point here; using the scform file, the
vskpage.z and vspace.z programs can build an edition from separate
pages libraries and can adjust vertical heights to avoid clashes
between musical lines.  But they cannot change the system breaks
or page breaks, and they cannot change the horizontal or vertical
layouts on the staff lines themselves.  These processes must be
completed in steps II, III, and IV.  However, it is possible to
return to earlier steps, modify things, and run the vspace.z program
again to get an updated version of the edition, page by page.
In other words, the makeup of the edition is determined entirely
by the content of the Stage2 files, the files in the /outputs/formats
directory, and the scform file used by the vspace.z program.

Why am I repeating all of this?  Because there is one more step
in the process.

                VI. Modify pages in the Edition

Once an edition has been constructed, it is still possible to edit
the individual pages to further improve the "presentation."  The
program that does this is the full screen graphical music editor
eskpage.z.  The input and output of this program is the same: a
page-specific i-file.  Note: It is also possible to edit the i-file
manually in a normal window, but this requires detailed knowledge
of the i-file format.

A musical page can always be improved (sometimes just a little,
sometimes a lot) using eskpage.z.  The problem is that eskpage.z
creates an "orphan" result; that is, the record of what has been
done is contained only in the results, and not memorialized in
the process (as in the case of the formats file or the scform file).
If a change is made upstream, say in one of the Stage2 files, it
cannot be made to propagate all the way to the final result.

For this reason, we make two recommendations:

 (1) try to make all improvements before Step VI.  Most position
     improvements can be accomplished with print suggestions in
     the stage2 files.  Most horizonal spacing improvements on
     staff lines can be made with the smskpage.z program and
     stored in the formats file.  Vertical arrangement of staff
     lines can be adjusted in the scform file.  What else is

 (2) Run eskpage.z only on editions that are to be "set in stone,"
     i.e., you don't expect to update them, but expect rather to
     make a new edition if further changes are warrented.  Two
     examples of this would be (1) a typesetting of music to be
     produced in hardcopy, e.g., a Dover Edition; (2) a musical
     score that is being produced for a specific purpose (e.g., a
     performance); (3) semi-permanent scores and parts to be
     published on the web.

               VII. Postlude 1: Create a .CFT compression

Once an edition of .mpg pages is made, it is possible to make a
single compressed file of the entire edition.  The Dmuse environment
has a utility to do this.

            VIII. Postlude 2: Create a .SRH search compression

I am currently working on a format for a compressed search file
(extension .srh).  The current program name is pagecomp.z.  It
operates on an entire edition.  The program that does the searches
is still work-in-progress.  (Old location: R:\NEWMAR\EXPER\P08.Z)

          IX. Postlude 3: Create a set of SCORE parameter files

The autoset.z -> mkspage.z -> dskpage.z sequence has been modified
for the purpose of creating SCORE parameter files corresponding to
the page-specific i-files in outputs/score.  The new sequence of
programs is autoscr.z -> paging.z -> scorecon.z.  The group name
is scrcon, so the output goes to outputs/scrcon.  If you look at
the i-files, you will see that they correspond exactly to the
i-files created by the normal sequence, but with extra information
appended to many of the file records.  This information is used by
scorecon.z to produce .pmx and .pmy score files in the directory

              X. Postlude 4: Create a set of MIDI files

MIDI files can be created for each movement from the Stage2 data
using the program midicon/mkmidi.z.  The process requires some
sound records and also a specification of channels and instruments,
which are stored in the Stage2 library and format under the name
name midicon.

             XI. Postlude 5: Create a set of TIFF files

The DMUSE environment provides a means of creating TIFF files on
a page-by-page basis.  These can then be converted to pdf files
using commercial software.

             XII. Postlude 6: Create a set of .ps files

We now have the ability to create .ps files directly from page
specific i-files.  The program pspage.z is a variant of dskpage.z.
Because these files relate directly to editions begin published
on the web, I have devised a file organizational structure that
deals with these files directly.  The top of the structure is
the directory dataout in c:/usr/local/apps.  The structure looks
as follows:

          contents (contains more documentation)
      pmx-form (proposed only)
      cft-form (proposed only)
    midi       (proposed only)
    humdrum    (proposed only)

The pspage.z program should be run from the edition library
that contains the source data.  An example might be:
This library must contain a temporary output library called
postout.tmp, and a small file called ednum, which contains
an assigned edition number for this conversion and a date.
The program will ask for the library containing the .mpg pages
you wish to convert.  Examples might be score, or parts/vln1.
The output .ps files go into postout.tmp, and you (the user) must
then move them to the proper location in the dataout directory.
At the moment, you (the user) are also responsible for clearing
the postout.tmp library before each use.  Otherwise, previous
files that are not over-written will still be there (i.e.,
page data from other parts).  I deal with this problem by renaming
postout.tmp directory to the name I want in the dataout directory
and then moving the whole thing there.  I then re-allocate a new
postout.tmp, which will by default be empty.