Music Processing, from Stage1 to Final Edition
=====================================================
Created: 02/23/06
Revised: 04/28/09
The propose of this documention is to describe all of the steps
in the process of going from a set of primitive stage1 files to
a final edition of music. What is laid out here is a continuous
work flow process; but the reality is that it is almost never
followed in its entirety. Almost all works, in fact, spend a
good deal of time in a partial state of completion; many will
never reach the final state of a musical edition.
The real purpose of this documentation, then, is to gather in one
place all of the instructions for the various stages of data entry
and processing.
Note: Unless otherwise stated, all programs referred to in this
document are stored in the directory j:/release/progs. On our
Linux systems, this directory is included in the searchable list
directories for zbex progs, which means they can be run directly
by name without typing zz first.
I. Initial Data Entry
──────────────────────
(creating basic Stage2 files)
The final result of the Intial Data Entry process for a musical
movement (or piece) is a basic set of stage2 files.
At the moment, there are three ways to get there:
1. Create a set of Stage1 files, and then convert these to stage2.
Stage1 files are single track only, and do not support such
things as time changes or variable length measures. Stage1
files are the output of the program input.z (with midi enabled).
Note: input.z is a combination of the old a program for midi
data entry, the old b program for processing midi data, and the
old look2 program for displaying the result. At the moment
(04/28/09) this program runs only in the old dmuse environment
on a DOS machine. Hardware is setup for this at the Middlefield
office.
Once created, Stage1 files may be converted in a generic way
to the Stage2 format using the program begin.z. Note: I recently
did this with some stage1 Bach organ music, and the process worked
perfectly.
2. Create a set of Stage2 files, using the commercial program Sharp-eye.
This process is currently used by Don and Ed at CCARH. Basically,
a peice (movement) is scanned by Sharp-eye and the Music-XML output
is sent to the program Finale. Finale has a plug-in which converts
this to Stage2 files. The scanning process involves some minimal
editing. The process works only with printed music. The final
Stage2 output requires some editing to bring the files into
conformance with accepted practice.
3. Create Stage2 files by manually entering them in a DMUSE window.
This process is slow, and is not recommended for large projects.
For small projects, it does have the advantage of flexiblity.
I(b). Setting up the file organization
──────────────────────────────────────────
This process is implied in creation process, but it is essential that
it be done at this point and that it be done right. All Musedata files
are organized according to the following system:
work/ --> use any name you like
stage2/ name must be stage2
mvt1/ --> use any name, but numbers are the default
01 --> use any name, but numbers are the default.
02 The group structure determines the use and
... meaning of a file. See s2-specs.ten in
j:/release for details on group structures
nvt2/
...
outputs/ name must be outputs
score/ --> The names at this level reflect the purpose and
score18/ size of the music being compiled. Letter portion
short/ of the name must match one of the group names.
skore16/ The number portion is the notesize. Size 14 is
parts/ the default and is not needed.
parts21/
score/ (example)
formats name must be formats
i-files/ name must be i-files
mvt1/ --> these names match the names in the Stage2 directory
01 --> these file names are created by autoset, and are
02 always numbers. These are linear i-files for
03 the various parts in the score.
...
mvt2/
...
pages/ name must be pages
mvt1/ these names match the names in the Stage2 directory
01 these file names are created by mskpage, and are
02 always numbers. These are page specific i-files.
...
editions/ any name is possible, but editions is preferred
name1/ can be anything, but should be descriptive, e.g. 1st-2004/
score/ any name, but descriptive. This is where the .mpg edition
pages are stored.
(some more examples of edition directories)
cont21/
xcont21/
parts/
scform --> this is the "make" file for the edition. For no particular
reason, we have standardized on this name. The scform
file can actually be stored anywhere on the system, since
all references to directories are by full path name. The
scform file here is for building a score. The scform files
for building parts are stored in the outputs/parts/pages/
partname/ directories
score.cft CFT compression of files in /score
mes.srh any name, but with srh extension. This is the compressed
search file.
postout.tmp/ this is a temporary directory used by the program
pspage.z to store .ps postscript file.
ednum this small file referenced by pspage.z to get the
CCARH edition number and production date.
score/ (example)
001.mpg These are the pages in a score. They should be numbered
002.mpg and should have the .mpg (music page) extension.
parts/ (example)
vln1 at the moment, there is no standardization on part names
001.mpg Notice that with the parts directory, there is an extra
002.mpg level of files.
...
vln2
...
distrib/ name must be distrib
kern/ must be kern: kern files for each movement (by name)
midi1/ must be midi1: midi1 files for each movement (by name)
midiP/ must be midip: midip files for each movement (by name)
mused/ must be mused: MUSEDATA files for each movement (by name)
score/ must be score: score files for each movement (by name)
mvt1/ --> use any name, but numbers are the default
mvt2/
...
mvtn/
doc Optional file containing history/current-state of data
notes Optional file containing critical notes on data entry.
Sorry about this long list. We are getting a bit ahead of ourselves here, but
it is important to show at the outset how the data structure should be set up
because all processes further down the line depend on this structure.
I haven't shown the structure above work/ because this will depend on what
piece is being entered. This description belongs to another discussion on
the overall organization of CCARH data -- a very important subject, by the
way -- and one which has not, in my opinion, been worked out in satisfactory
detail. The problem, in a nut shell, is that we do not yet have a clear idea
of what the public's view of our data should be -- although we have made
good progress here.
II. Supplemental data entry on Stage2 files
───────────────────────────────────────────────
This is actually an ongoing (and potentially never ending) process. The
Stage2 file is fundamental data source for all other music files, whether
sound, display, print, or analysis.
But the Stage2 files that come out of Initial Data Entry often need
a number of things added, including slurs, articulations, ornaments,
transpositions, re-arrangement of stems and beams, clef changes,
track numbers, print suggestions to improve the visual layout, sound
suggestions for more accurate sound files. In addition, the group
structure may need to be adjusted. Cue-size notes may be added to
parts, etc., etc.
If Supplemental data entry is in an early stage, i.e., many basic
things need to be added, the program s2ed.z provides a graphical
means for doing this, part by part.
Otherwise, changes to Stage2 files must be done by hand.
For Stage2 files representing more than one instrument, or for
music on the grand staff, it may be necessary to merge two or
more Stage2 files. There is a program called fmerge.z that purports
to do this, but it is flawed in many respects. The directory
franfro inside release/progs contains some programs and a process
which is more careful about merging files. The more careful the
program, the less work required to fix the result.
There have been some recent changes way multiple tracks are represented
in Stage2 files.
(1) only use the backup command when a track reaches the end of a
measure. If the track does not reach, use the irest to complete
the track. This is necessary not only for score conversion,
but also for making .srh files
(2) When representing multiple instruments, do not use chords
unless it is absolutely necessary. autoset.z will construct
chords when the music isorhythmic and the stems point the
same direction.
(3) the global print suggestion for rest placement can save a
lot of time and space; try to use it.
III. Create initial output displays
───────────────────────────────────────
This is actually an interative process. The initial objective is to
see the music in print and to use this to correct errors and to
enhance the music.
The final object is to get the stage2 files into the most advanced
state possible, before moving on to the Edition stage.
The basic sequence is autoset.z -> mskpage.z -> dskpage.z. The first
object is to get these programs running without error. This will
correct 99.9% of all format errors in the stage2 files.
III(b). Determine the system layout
───────────────────────────────────────
The system (and page) layout is influenced by the spacing of the
notes (compression factor), and by degree to which various empty
staff lines are dropped from the system (this also depends on
the system breaks. Our software does not have a "top down" way
of arranging system breaks and page breaks. There is a print
suggestion (on the measure record) for forcing a system break,
but the overall effect on system breaks is still left to chance.
The only way we have of deterining a system layout we like is to
run the autoset.z -> mskpage.z -> dskpage.z sequence several
times, with different compression values and possibly different
line spacings (e.g., contained in the files in outputs/formats).
This iterative process has, to date, always been successful in
producing a satisfactory system layout.
But it is essential that this process be completed to the user's
satisfaction at this point in time, because all future editing
will depend on a stable system layout. The system layout can
be changed manually, of course, but this creates what we call
an "orphan" result -- that is, a result that is no longer tied
back to the original stage2 files. It is of great stategic
importance that this connection be preserved, because we would
like to preserve the ability to make small changes in the Stage2
files and then run the autoset.z -> mskpage.z -> dskpage.z sequence
and other down stream processes again to get an "updated" Edition
automatically.
IV. Make adjustments to vertical and horizonal positions
──────────────────────────────────────────────────────────
Vertical adjustments and minor horizonal adjustments are made with
print suggestions in the Stage2 files. These can be made at any
point in the process.
Major horizonal adjustments to note spacings can be made using
the xmskpage.z program, but because of bugs in this program, we
recommend that adjustments are made only once, and then not
changed. The program displays each system as it is typeset, and
allows the user to move objects horizontally to improve the
layout. The result of this editing is stored in the formats file
for this movement. Thereafter, the basic typesetting sequence
should be autoset.z -> xmskpage.z -> dskpage.z. Horizontal
editing results will be preserved, as long as you don't try
to make further changes. At some point, I hope to make this
process more robust, so that further changes can be added.
V. Make an Edition from the Page Files in OUTPUTS
─────────────────────────────────────────────────────
There are several shortcomings with the page files in /outputs
(1) Each movement is in a separate directory
(2) Page bottoms are not uniform
(3) Paging information is absent
(4) The vertical arrangement of lines and systems is not optimal
(5) Text underlay sometimes bumps into notes
In earlier times, it used to take several weeks to cobble together an
edition from the page files in /outputs/pages. Each page had to be
separately edited and copied into an /editions directory. Now there
is a faster way to do this work, and a means by which the process can
be automated, so that if (or when) changes are made up-stream (e.g.,
in the Stage2 files), the edition can be recompiled without doing all
of this work again.
The key to this process is an instruction file, the format of which
is documented in release/progs/edform.doc, and which for the time
being we are calling scform. The file provides such information as:
source lib: pages library providing the source for the edition
destin lib: library into which output pages are to be stored
blank page: generic message to put on any "blank page"
page id header: location and font information for page id
page id format: page id, for every page. Includes a page number
top 7 "Z" records, required for .MPG file format
The file then provides, on a page by page basis, the page number, the
exact location of the source, and a set of (encoded) instructions on
how each musical staff line "L" is to be adjusted up or down, and also
how to adjust the vertical height of text underlay and a line of
figured harmonies, if either exist and need to be moved.
These instructions are optional; the page layout can be moved without
any adjustment. But if the edition is to look "professional," the tops
and bottoms of the musical page should be uniform across the edition.
Simple instructions such as:
VJ-1 = vertically justify proportionally to 2800
VJ-2 = vertically justify proportionally to 2800; with x2 space between systems
VJ-3 = vertically justify proportionally to 2800; use only inter-system space
can do this. More complicated instructions address individual lines.
The scform file must be generated by hand, because the design of the
page layout and the individual files (with their separate movement
libraries) cannot be reliably constructed by computer. The moving
instructions, however, can be generated by a program. This program
is called vskpage.z, and is a variant of the page editing program
eskpage.z, about which more will be said in a moment.
vskpage.z is a full page graphics editor of a musical page. The only
editing commands which work, however, are those which move the music
staff lines and music systems up and down, and those which move text
underlay and figured harmonies up and down. The output of vskpage.z
is a line of instructions, displayed on the screen, which must be
transferred to the screen in which the scform file is being constructed.
This process may seem somewhat awkward, but it is flexible and provides
the user complete control over what goes in the scform file.
Once the scform file is built, the program that actually constructs
the edition is called vspace.z. You can run vspave as many time as
you like, without changing anything in the source pages. In this
way, by adjusting the scform file and its moving structions, you
get get the edition to look exactly the way you want.
Just to summarize an important point here; using the scform file, the
vskpage.z and vspace.z programs can build an edition from separate
pages libraries and can adjust vertical heights to avoid clashes
between musical lines. But they cannot change the system breaks
or page breaks, and they cannot change the horizontal or vertical
layouts on the staff lines themselves. These processes must be
completed in steps II, III, and IV. However, it is possible to
return to earlier steps, modify things, and run the vspace.z program
again to get an updated version of the edition, page by page.
In other words, the makeup of the edition is determined entirely
by the content of the Stage2 files, the files in the /outputs/formats
directory, and the scform file used by the vspace.z program.
Why am I repeating all of this? Because there is one more step
in the process.
VI. Modify pages in the Edition
─────────────────────────────────
Once an edition has been constructed, it is still possible to edit
the individual pages to further improve the "presentation." The
program that does this is the full screen graphical music editor
eskpage.z. The input and output of this program is the same: a
page-specific i-file. Note: It is also possible to edit the i-file
manually in a normal window, but this requires detailed knowledge
of the i-file format.
A musical page can always be improved (sometimes just a little,
sometimes a lot) using eskpage.z. The problem is that eskpage.z
creates an "orphan" result; that is, the record of what has been
done is contained only in the results, and not memorialized in
the process (as in the case of the formats file or the scform file).
If a change is made upstream, say in one of the Stage2 files, it
cannot be made to propagate all the way to the final result.
For this reason, we make two recommendations:
(1) try to make all improvements before Step VI. Most position
improvements can be accomplished with print suggestions in
the stage2 files. Most horizonal spacing improvements on
staff lines can be made with the smskpage.z program and
stored in the formats file. Vertical arrangement of staff
lines can be adjusted in the scform file. What else is
there?
(2) Run eskpage.z only on editions that are to be "set in stone,"
i.e., you don't expect to update them, but expect rather to
make a new edition if further changes are warrented. Two
examples of this would be (1) a typesetting of music to be
produced in hardcopy, e.g., a Dover Edition; (2) a musical
score that is being produced for a specific purpose (e.g., a
performance); (3) semi-permanent scores and parts to be
published on the web.
VII. Postlude 1: Create a .CFT compression
──────────────────────────────────────────────
Once an edition of .mpg pages is made, it is possible to make a
single compressed file of the entire edition. The Dmuse environment
has a utility to do this.
VIII. Postlude 2: Create a .SRH search compression
────────────────────────────────────────────────────
I am currently working on a format for a compressed search file
(extension .srh). The current program name is pagecomp.z. It
operates on an entire edition. The program that does the searches
is still work-in-progress. (Old location: R:\NEWMAR\EXPER\P08.Z)
IX. Postlude 3: Create a set of SCORE parameter files
───────────────────────────────────────────────────────
The autoset.z -> mkspage.z -> dskpage.z sequence has been modified
for the purpose of creating SCORE parameter files corresponding to
the page-specific i-files in outputs/score. The new sequence of
programs is autoscr.z -> paging.z -> scorecon.z. The group name
is scrcon, so the output goes to outputs/scrcon. If you look at
the i-files, you will see that they correspond exactly to the
i-files created by the normal sequence, but with extra information
appended to many of the file records. This information is used by
scorecon.z to produce .pmx and .pmy score files in the directory
/outputs/scrcon/pages.
X. Postlude 4: Create a set of MIDI files
───────────────────────────────────────────
MIDI files can be created for each movement from the Stage2 data
using the program midicon/mkmidi.z. The process requires some
sound records and also a specification of channels and instruments,
which are stored in the Stage2 library and format under the name
name midicon.
XI. Postlude 5: Create a set of TIFF files
────────────────────────────────────────────
The DMUSE environment provides a means of creating TIFF files on
a page-by-page basis. These can then be converted to pdf files
using commercial software.
XII. Postlude 6: Create a set of .ps files
───────────────────────────────────────────
We now have the ability to create .ps files directly from page
specific i-files. The program pspage.z is a variant of dskpage.z.
Because these files relate directly to editions begin published
on the web, I have devised a file organizational structure that
deals with these files directly. The top of the structure is
the directory dataout in c:/usr/local/apps. The structure looks
as follows:
dataout
editions
postscpt
admin
progs
contents (contains more documentation)
beethoven
orch
sym1
score
parts
cb
...
sym2
...
chamber
...
haydn
mozart
bach
handel
...
pmx-form (proposed only)
cft-form (proposed only)
midi (proposed only)
humdrum (proposed only)
...
The pspage.z program should be run from the edition library
that contains the source data. An example might be:
j:/musedata/classical/beethoven/bhl/orch/sym7/editions/beta-2008.
This library must contain a temporary output library called
postout.tmp, and a small file called ednum, which contains
an assigned edition number for this conversion and a date.
The program will ask for the library containing the .mpg pages
you wish to convert. Examples might be score, or parts/vln1.
The output .ps files go into postout.tmp, and you (the user) must
then move them to the proper location in the dataout directory.
At the moment, you (the user) are also responsible for clearing
the postout.tmp library before each use. Otherwise, previous
files that are not over-written will still be there (i.e.,
page data from other parts). I deal with this problem by renaming
postout.tmp directory to the name I want in the dataout directory
and then moving the whole thing there. I then re-allocate a new
postout.tmp, which will by default be empty.